In a separate thread someone asked about deploying Newforma servers in a virtual environment.  We get questions on this topic all the time.  We also get a lot of support calls from customers that have performance related problems as a result of running in virtual environments.  I’m only aware of a couple of customers that have successfully deployed the Newforma Project Center server in a production environment.  So, I thought a poll on this topic might be appropriate.  If you have had success, please respond to this topic.  I’m sure many people would love to learn your secrets!

Note, the Newforma Info Exchange server works well in virtual environments.  Its performance demands are typically modest.

I’ve been production virtual since 7th edition. 7th ran very well virtually. Yes you must have adequate resources to support it. But it was generally well behaved. Disk I/O is important. we had 2 Project servers 4 gb ram 4 virtual cpus & 1 info xchng.

8th is another story. Resource demands for 8th seem way up, and I still have some perfromance issues. At the recommendation of support, already have gone from 4 to 12 GB per server, and I was told to run 8 cpus for each one!! Surprise, only the top most expensive Vsphere 4.x license supports more than 4. Fortuantely Vsphere 5 allows 8, when it gets here…, and I can upgrade…

More memory did help, but it gets consumed and not given back. When I went from 4 to 8, it looked ok for a while, but then it was back up against the top. Now I’ve gone to 12 and it may not be enough. And you can’t rely on VMware’s ballooning feature - you have to commit the RAM.

125 users, 90 Projects, 12 TB, split between 2 Project servers

So for me 8th has ben a major performance leap backwards.

So you might ask, why not dedicated H/W? Well, virtualization offers so many advantages in managing and recovering applications that I would never go back. Its the only way to manage any data center in order to make the most efficient use of resources - H/W, Power, & Human. There is however no free lunch - you can’t run 10 lbs of stuff in a 1 lb box.

Which brings me to my final point. Updates to 8th & subsequent editions need to make IMPROVED performance, as well as optimization for operating in a virtual environment, a major design requirement.

It is no longer acceptable in the marketplace for any vendor of server based applications to ignore, or only passively support virtualization. SQl, exchange, everything is being adapated and supported in a virtual environment.

There are plenty of features in the NPC now (more than I might use even)- I need the product to perform MUCH better, or else I won’t be able to continue to use it and recommend it to others.

You’re not alone in your characterization of Eighth Edition’s performance.  Our primary focus in Eighth Edition development was the delivery of a wide range of Contract Administration features for our customer base, and we didn’t give the day-to-day user performance experience as much attention as we would have liked.  The performance improvements we did make were principally oriented towards the back-end for very large NPCS servers, and the management of large numbers of NPCS servers in a Newforma domain.  We also made performance improvements when working with large numbers of projects. 

One of the major goals in the forthcoming Ninth Edition release will be performance – the everyday user experience within Project Center must be improved.  We are working to improve the speed in which project data is displayed, decrease the time for creation/editing windows to pop up, and reduce the memory footprint of the client.  We are also optimizing the communication between the Project Center client and server to reduce how much data is transferred between the two for all user operations.  As part of the Ninth Edition development cycle we have solicited feedback from some of our customers about which areas they find acutely painful.  The information we’ve collected includes areas that are slow, and metrics around how much data is being displayed/managed in those activity centers.  If you have specific use cases that you would like to share, we would be very happy to investigate those areas as well.

thx for the reply.

Do you plan to do anything to improve 8th?

How long will we be waiting for 9th?

Major new releases come out roughly once per year so I would expect that we’ll have Ninth Edition completed in the first part of 2012.

One more thought for your virtual environment.  Rather than adding hardware (CPU, memory) to your existing virtual machines, have you considered adding more VMs instead?  So, instead of an eight Core/12 GB machine, could your host server support additional four core/4-8 GB VMs instead with your active projects partitioned across these images for better load balancing?

Waiting almost a year for the next major release to get some relief is not an exciting option. It also asks me to take it on faith that it will be “better”. Of course I’d have to upgrade to find out, then its too late to go back. I request that making improvements in 8th edition performance be made a priority.

As to my environment, I have already distributed the load over 2 servers, as I will describe below. But I don’t have any metrics or rules to use to balance the load. Users are split about 50/50 but that did not have a big impact. Since moving a project kills the index and summaries for a few days, it’s not something I can justify doing using just trial and error.

VM Cluster: Vsphere 4.1, 5 hosts, each Dual quad core 3GHZ core2 xeon, each with 40GB of RAM. Host CPU is between 15 and 35% utlized, RAM is between 55 and 75% utilized.

Each NPC server is running 2008R2 with 4 cores, 12GB of RAM assigned. DB Disk is iSCSI attached to a NetApp storage array with 15K SAS drives. Project data on SATA accessed via 10GbE on the storage server. VM Hosts have mutliple 1 gbe.

I have 91 Projects, distributed 43 & 48. 129 Users 63, & 66 All project data is about 13TB.

If you have additional suggestions I’ll certainly consider them.

We will be virtualizing our rack (9 physical servers) but our dataset varies differently from what I gather is the norm on Newformant:  we have hundreds of smaller projects per-year, and thousands stored simultaneously, but a mere 1.2TB of data utilized across the past 6 years.

nreuben’s comments give me pause about going from 7 (which we’re on now) to 8.  I may pass on 8 altogether at this point and go to 9.

I’m sure it is difficult to blend new feature sets or improved functions with performance concerns.  Like the old Henry Ford quote, people just want a faster horse, so where does one draw the line?  As one of the presumably-smaller clients using Newforma, our druthers fall on performance and refinement of existing feature sets rather than new features. 

I assume the performance penalties nreuben speaks of will not impact us to the same degree, due to the massive difference in data size but IF it did, his hardware upgrades he mentions would basically skyrocket my virtualization project costs to an unaffordable level.  That’s an amount that would force me to make sweeping change to the proposed virtualization project, and thus is immediately keeping us on v7. 

All that said, we have enjoyed Newforma’s performance historically speaking and ultimately I have no business relationship I appreciate more than that with Newforma.

Well, can’t say for sure whether to bypass 8 based on your situation, but if you don’t need the features, or don’t have any issues, then I’d say wait. From what I’ve learned it sounds like indexing and rendering are big hogs, so with your smaller amount of data, you may not be so badly hit.

However I would still be in strongly favor of virtualization due to the operational flexibility. For example, I had a drive on one of the project servers suddenly fill up, and crashed the system. It had been poking along fine with 5gb or so free space, then all of sudden needed to dump 8gb. While not sure of the cause looks like a large chunk of new data for index might have created the need temporarily. In this case I just had to allocate more space for the volume. But if it was a fixed disk, a lot more painful to recover from that.

With Vsphere 5 the cpu limits are now increased so the previous limit of 4 is no longer a restriction. If you haven’t updated then do so. With 8 cpus assigned its a bit happier.

Finally making a little progress on analyzing this with the support team. There are a few settings to tweak which shift some of the loads outside the daily user window. Also at least one setting that’s set way to low only on virtual environments by default. But still not enough. I’m guessing that not much more is going to happen as they sound deep in on 9th instead. Not so happy about that….

Similar situation here.  I’m starting to debate which option is better. We have a single NPCS in our corporate office that has 1,280 projects and ~3.8 million files.  Total files on disk ~3.3TB.  Starting to show some performance drains and until we upgrade to vsphere 5 we’re limited to 4 CPUs on that vm and of the 10ghz available it hovers in the 6-8ghz range constantly.  So debating on just toughing it out and tossing 8cpus at it or going ahead and going back to the manual load balancing of the Newforma 4 days.

Is there going to be (should be!) some sort of load balancing built into future releases.  Manual load balancing now consists of building a 2nd NPCS server and then manually splitting the load. Version 4 we had to build multiple physical NPCS’s to get around the index size limitations and now that its on mysql we need to get around the cpu limitations now that we can have such huge indexes. If it ain’t one thing its another!

Thanks,
Creighton

I would recommed adding the CPUs just to lower the overall load on the system. it seems happier after adding the extra procs, with the peak loads being less severe.

However don’t negelect memory. Adding more ram helps mysql perform better. Unfortunately you cannot rely on memory ballooning as mysql wants to “know” how much ram is avaialble and sets that in the config file. So you have to give the vm dedicated ram. I guess you could take it back after mysql sets its’ config.

Load balancing is not free unfortunately, as the indexes must rebuild after moving a project. So you are w/o search for a day or more. Also there are not great metrics for deciding on what to move.

I think the architecture needs to be broken up into tiers, so that they can be scaled and distributed as needed, ie a DB tier, indexing tier, and client servicing tier. Then you could size the systems as needed by the function. The functions could be combined or run seperately base on needs as well.

Sorry I’m so late to the discussion, but I was intrigued by NREUBEN’s hardware and performance complaint. We have a project server virtualized on a very similar system and load, but only really noticed performance issues on the workstation (old computers ). Newer workstations seem to run Newforma 8.1 fine and searches are snappy.

I was wondering how the 15K SAS drives you have in the NetApp are configured?

Also, how many files in that 13 TBs of project data?

More RAM always helps with virtuals, and a good answer to most peoples since it’s cheap nowadays, but sometimes how the hard drives are setup can cause issues with databases as well.

Newforma is REALLY database intensive.

Our current configuration uses 10K sas drives for the DB volumes which are connected directly via 1gb iscsi running jumbo frames. Maybe worth testing 15K vs 10K to see if there is a difference?

Each of 2 Project servers has 8 cores assigned (4x2) and 12gb memory. The NIX server uses half that.

the project volume is only 12.5tb of actual files (6.4M) - the rest is snapshots

The cpu load associated with database is different on the two machines, even though the number of projects and users (and roughly files) is fairly well balanced between them.

Information on tuning has been limited.

Sorry, I was was wondering about the RAID setup yet neglected to toss out the acronym.

I have noticed RAID 5’s write penalty becoming a big issue with virtual guests and database servers, and thought maybe it was playing a part for you as well.

Its a Netapp storage system so its uses their RAID configuration - RAID DP. This quote from an article comparing RAID configurations:

“NetApp with its ONTAP OS and WAFL filesystem serializes random IO by journaling all writes in NVRAM, coalescing them, and writing the data on the first available block nearest to the head of the spinning disk. This way, the effective write IOPS of a NetApp storage system is some 3 times higher than a RAID 1 system “

http://www.brianmadden.com/blogs/rubenspruijt/archive/2010/11/27/vdi-and-storage-deep-impact.aspx

So while I couldn’t tell you how much faster, as we did not benchmark all these configurations, it should probably suck less than most other raid schemes

mbhame - 06 January 2012 04:54 PM

nreuben’s comments give me pause about going from 7 (which we’re on now) to 8.  I may pass on 8 altogether at this point and go to 9.

Given the imminent release of Ninth Edition I would recommend waiting till the next release as well.  Ninth Edition contains many performance improvements over Eighth Edition, and will also support a direct upgrade path from Seventh Edition (something we’ve never been able to do before - previously all upgrades where V -> V + 1, but never V -> V + 2).

Yes - defintely stay on 7. Even though I’ve mostly stabilized performance, I’m regularly finding bugs as well.