I’m currently reading chapters of the VMware Infrastructure 3 Advanced Technical Design Guide in my spare time and I came across their recommended ESX partitioning strategy. I’d like to think I’ve got a pretty good handle on ESX partitioning. I’m quite comfortable with it, and thus normally I would breeze quickly through partitioning documentation, verifying along the way that my partitioning was still on par.
My partitioning scheme was still looking good until I came across a new recommendation of creating a dedicated /opt partition. This is something I hadn’t done before. Under normal circumstances without a dedicated /opt partition, /opt is going to be a directory off / (root). Why is this not the greatest idea? I was enlightened by the fact that some VMware HA logging as well as some hardware agent logging is stored in /opt. There are enough posts on the VMTN forums describing situations where excessive logging on /opt chewed up all available partition space on / to warrant proactive measures. As we know, running out of disk space on / is less than ideal and that’s putting it mildly.
Solution: When building the ESX host, create a dedicated partition for /opt.
Now I’ll be honest, I’ve never run into a situation of excessive logging on /opt, but this is one of those strategies that falls into the category of “an ounce of prevention is worth a pound of cure”. Learn from the experiences of other VI administrators who I’m sure suffered some downtime as a result. Don’t wait for this happen to you when it doesn’t need to. The ESX host and its health is critical for datacenter and production operations. When the ESX host isn’t happy, the VMs running on it usually aren’t happy either.
That said, here is my updated partitioning scheme for ESX 3.5.0.
Create the following partitions in the following order:
| Mount Point | Type | Size | Primary? | Notes |
| /boot | ext3 | 250MB | Yes | The default from VMware is 97MB. When we migrated from ESX 2.x to ESX 3.x the partition size grew from 50MB to nearly 100MB. I came up with 250MB to leave breathing room for future versions of ESX which may need an even larger /boot partition. This is all a moot point because I don’t do in place upgrades. I rebuild with new versions. |
| <swap> | 1600MB | Yes | Twice the maximum amount of allocatable service console memory. My COS memory allocation is 500MB but if I ever increase COS memory to the 800MB max in the future, I’ve already got enough swap for it without having to rebuild the box to repartition. | |
| / | ext3 | 4096MB | Yes | The default from VMware is 3.7GB. We want plenty of space for this mount point so that we do not suffer the serious consequences of running out. |
| /home | ext3 | 4096MB | Not really needed anymore except as home directory storage for local and the default from VMware is that this partition no longer is created. For me this is just a carryover from the old ESX days. And disk space is fairly cheap (unless booting from SAN). I’ll put this and other custom partitioning out to pasture when I convert to ESXi where we are force fed VMware’s recommended partitioning. | |
| /tmp | ext3 | 4096MB | The default from VMware is that it doesn’t exist, rather it creates the /tmp folder under the / mount point. This is not a great idea. VMware uses a small portion of /tmp for the installation of the VirtualCenter agent but my philosophy is we should have plenty of sandbox space in /tmp for the unpacking/untarring of 3rd party utils such as HP Systems Insight Manager agents. | |
| /var | ext3 | 4096MB | The default from VMware is 1.1GB and additionally VMware makes the mount point /var/log isolating this partition strictly for VMware logs. We want plenty of space for this mount point so that we do not suffer the serious consequences of running out. In addition, we want this to be a separate mount point so as not to risk the / mount point by consuming its file system space. VMware logs and other goodies are stored on this mount point. | |
| /opt | ext3 | 4096 | The default from VMware is that it doesn’t exist, rather it creates the /opt folder under the /mount point. We want enough space for this mount point so that we do not suffer the consequences of running out. In addition, we want this to be a separate mount point so as not to risk the / mount point by consuming its file system space. VMware HA logging and sometimes hardware agent logging are stored on this mount point. This is a VI3 ATDG recommendation. | |
| <vmkcore> | 110MB | The default from VMware is 100MB. I got the 110MB recommendation from Ron Oglesby in his RapidApp ESX Server 3.0 Quick Start Guide (this book is a gem by the way; my copy is worn down to the nub). Although I asked Ron, he never explained to me where he came up with 110MB but let’s just assume the extra 10MB is cushion “just in case”. This is the VMKernel core dump partition. The best case scenario is you rarely if ever have a use for this partition although it is required by ESX whether it’s used by a purple screen of death (PSOD) or not. | ||
| Leave the remaining space unpartitioned. This can be partitioned as VMFS-3 later using VirtualCenter for maximum block alignment. |
Good post, and good book. Don’t you wish you had this 2-3 years ago? I did.
I follow these recommendations today with one modification. For environments with SAN or NAS for VM datastores, I am more liberal in creating / and /home partitions of at least 10GB in size. Will I need the space? Absolutely not. But if there is no need for local VMFS partitions, than why not.
Actually, can’t wait till ESXi takes over the planet and I can forget about the partitioning chapter of ESX administration. 🙂
Well I had the VI2 ATDG a few years ago. I still have it and it was/is an awesome resource. The book is pretty worn as I carried it everywhere for a few years. It has been replaced though by the VI3 ATDG.
10GB for /home? Let’s discuss that over a beer or coffee or cheesecake at the next VMworld 🙂
Jas
10GB for home is a relic of my lazy days to WinSCP DVD isos to my home directory, then mv-ing them to my VMTemplate datastore. Probably completely un-needed but why not if I have 73GB of local storage for ESX install only?
The next VMworld USA that is… 🙂 Can we find a Hooters in San Francisco? they have good cheesecake and I’ve heard they have beer.
Good post, Jason, it closely mirrors my own best practices.
In response to Sean, I do have to say that I always allow room for a local VMFS partition, even in a SAN/NAS environment. I’ve seen too many instances where a local VMFS partition would have come in handy, and even some cases where things didn’t work properly without a local VMFS partition.
Just my 2 cents…
Hi,
IIRC logging isn’t really the issue anymore as most of it was moved of opt nowadays.
Otherwise I agree with the comments and your design. Especially /var is needed as the core dumps go into there.
I have a bigger /home though and smaller sizer for /var /tmp and /opt but you can always discuss those.
http://www.mingle-mangle.org/2008/11/choosing-the-right-partition-layout-for-esx-servers/
@Sean @Scott Lowe I use /tmp for temporary storage space, unpacking of files for installation and configuration, etc. (I don’t use /home for this) .iso files would go on a VMFS-3 datastore whether that’s local left over unpartitioned space (usually not) or on the shared storage (SAN, and usually the case). My ESX hosts have mirrored 72GB drives internally so there is plenty of partitioning room, with some left over. We used to put .iso files on /vmimages but that was the old days of ESX prior to VI3. Now /vmimages is used as VMware’s storage location for VMware Tools and that’s about it.
Just to say that I’ve had VI3 hosts taken out by the VC agent core dumping every few minutes and placing core dumps in /var/core until / filled up.
We’ve been mounting at /var rather than /var/log for a while now.
Gary.
Jason – as far as Ron’s recommendation on the 110MB for the vmkcore partition – he’s maxing out the partition to the largest allowable size. If you try to make a vmkcore partition any larger, the installer will stop you with a warning message.
Jason,
Thanks for updating this.
Jason,
Any updates for this to work with Version 4.0 / vSphere?
Roger