A few months ago, The Tolly Group released a report comparing Citrix and VMware VDI solutions.
They’re at it again. Today, The Tolly Group released another comparison. Today’s report compares Citrix XenServer 5 and VMware ESX 3.5.0 Update 3 with Citrix XenApp as the workload.
Citrix Systems commissioned Tolly to evaluate the performance of Citrix XenApp when running on Citrix XenServer 5 and compare that with XenApp running on VMware ESX 3.5u3.
Testing focused on system scalability and user quality-of-experience. This test report was approved for publication by VMware. The VMware End User License Agreement (EULA) requires such approval.
The testing was conducted in accordance with Tolly Common RFP #1101, Virtual Server Performance.
Summary of Results:
* Citrix XenServer 5 outperforms VMware ESX 3.5 by 41% in user scalability tests.
* XenApp, running on XenServer, retains a consistent user experience as load is increased to 164 users.
* Virtualizing 32-bit XenApp gives IT administrators a viable approach to increasing total user density on physical servers, without the need to re-certify their existing applications and drivers for a 64-bit platform.
* Consolidating XenApp farms on XenServer results in data center reliability benefits and cost savings.
Click here to download the report. You will need to register for the report download.
Strange how the VDI comparison commissioned by VMware favoured VMware and the XenApp Workload comparison commissioned by Citrix favoured Citrix.
Either the Tolly Group is fixing the results or the vendors are commsissioning reports they know they will win. Or I’m just a conspiracy theorist, where’s my tinfoil hat.
No tinfoil needed, wharlie — each report was talking about different things. The Tolly report on VMware was much more about usability and manageability.
Note that “scalability” in this report measured how many users could open Excel and Word on a hypervisor running 4 VMs.
You can imagine a different (and I’d suggest more realistic) scalability test that looked at how many total user sessions you can put on that box (eg, by running more VMs on ESX), and using more unpredictable, bursty workloads.
Hi John,
As you pointed out the tests don’t seem to be objective.
Also after going through a heap of Tolly reports, the Vendor who commissions the report always wins.
Most people just focus on the headlines and don’t bother looking at the details of the tests carried out.
It’s hard to be independent when you are being paid by the people you are testing.
Jason,
Thanks for posting this. I think it clearly shows a couple things:
– You are unbiased in your comments. You report things even when VMware isn’t the “top dog”
– VMware plays fair, at least with testing. This is a test that shows a competitor outperforming VMware, yet it is a VMware approved report. As long as the tester plays by the rules and conducts a fair test, VMware will authorize publication…even if the results are not what they would like for them to be.
Thanks again…keep up the great work!
KLC
Well Ken, I don’t mean to burst your bubble, but I am biased towards VMware, until a better product comes along that is good for me personally and the businesses that I support. That hasn’t happened (yet). I don’t mind highlighting the competition sometimes because competition is ultimately good for the customer.
While it may be true that you hold a bias towards VMware, you don’t let that bias jade you to the point of hiding “unfavorable” content. I think it is only through showing both sides of the fence that we can keep everyone honest (even VMware 🙂 )
Theres an interesting case study over at 360 the VMware and Xen guys which compares the 2 vendors for what they call “a difficult workload”:
http://360is.blogspot.com/2009/05/citrix-xenserver-case-studies.html
Disclaimer: Im a VMware and XenServer user, don’t ask me about Hyper-V 😉