I’m mildly excited for the upcoming week. If all goes well, I’ll be upgrading to AMD Opteron processors which support a virtualization assist technology called Rapid Virtualization Indexing (or RVI for short).
There is overhead introduced in VMware virtualization via the virtual machine monitor (VMM) and comes in three forms:
- Virtualization of the CPU (using software based binary translation or BT for short)
- Virtualization of the MMU (using software based shadow paging)
- Virtualization of the I/O devices (using software based device emulation)
RVI is found in AMD’s second generation of virtualization hardware support and it incorporates MMU (Memory Management Unit) virtualization. This new technology is designed to eliminate traditional software based shadow paging methods for MMU virtualization thereby reducing the overhead in bullet #2 above. VMware lab tests show that RVI provides performance gains of up to 42% for MMU-intensive benchmarks and up to 500% for MMU-intensive microbenchmarks.
How it works:
Software based shadow page tables store information about the guest VM’s physical memory location on the host. The VMM had to intercept guest VM page table updates to keep guest page tables and shadow page tables in sync. By now you can probably see where this is going: applications and VMs which had frequent guest page table updates were not as efficient as those with less frequent guest page table updates.
The above is similar to guest VM kernel mode calls/context switching to access CPU ring 0. Previously, the architecture wouldn’t allow it directly via the hardware so the VMKernel had to intercept these calls and hand-hold each and every ring 0 transaction. Throw 10,000+ ring 0 system calls at the VMKernel per second and the experience starts to become noticeably slower. Both Intel and AMD resolved this issue specifically for virtualized platforms by introducing a ring -1 (a pseudo ring 0) which guest VMs will be able to access directly.
VMware introduced support for RVI in ESX 3.5.0. RVI eliminates MMU related overhead in the VMM by relying on the technology built into the newer RVI capable processors to determine the physical location of guest memory by walking an extra level of page tables maintained by the VMM. RVI is AMD’s nested page table technology. The Intel version of the technology is called Extended Page Tables (EPT) and is expected sometime this year.
One of the applications of RVI that interests me directly is Citrix XenApp (Presentation Server). XenApp receives a direct performance benefit from RVI because it is an MMU-intensive workload. VMware’s conclusion in lab testing was that XenApp performance increased by approximately 29% using RVI. By way of the performance increase, we can increase the number of concurrent users on each virtualized XenApp box. There are two wins here: We increase our consolidation ratios on XenApp and we reduce the aggregate number of XenApp boxes we have to manage due to more densely populated XenApp servers. This is great stuff!
There is a caveat. VMware observed some memory access latency increases for a few workloads, however, they tell us there is a workaround. Use large pages in the guest and the hypervisor to reduce the stress on the Translation Lookaside Buffer (TLB). VMware recommends that TLB-intensive workloads make extensive use of large pages to mitigate the higher cost of a TLB miss. For optimal performance, the ESX VMM and VMKernel aggressively try to use large pages for their own memory when RVI is used.
For more information and deeper technical jibber jabber, please see VMware’s white paper Performance of Rapid Virtualization Indexing (RVI). Something to note is that all testing was performed on ESX 3.5.0 Update 2 with 64 bit guest VMs. I give credit to this document for the information provided in this blog post, including two directly quoted sentences.
For some more good reading, take a look at Duncan Epping’s experience with a customer last week involving MMU, RVI, and memory over commit.
Hi Jason, do you know if EPT is already supported on ESX 3.5 and 3i ?
BTW, nice post !