I’ve been a fan of the Car Talk radio program since I was introduced to it in 1993. I hope the Tappet brothers don’t mind if I borrow the theme from one of their popular segments appropriately called Puzzler. It seemed fitting for this article which I’m going to call VMware Talk Puzzler. Not surprising, the goal of the Car Talk Puzzler is to listen to the problem (which is typically not simple), then provide the root cause. In this adaptation, I’ll present a real life vSphere problem. If you choose to take a stab, your job is three fold:
1) Identify the root cause of the problem.
2) Identify the solution.
3) Identify the unique tasks or chain of events which lead to the problem.
Here we go.
I was called in to help troubleshoot a problem. “Carl” had created a virtual machine in a VMware vSphere 4.1 Update 1 cluster. The problem Carl was experiencing was that the VM would not power on. Error messages in vCenter include but are not limited to:
-
“Failed to find a host for powering on the virtual machine.”
-
“The following faults explain why the registered host is not compatible.”
-
“The number of virtual CPUs may be limited by the guest OS selected for the virtual machine or by the licensing for the host.”
I asked if the ESXi cluster and vCenter were licensed. Carl confirmed by showing me that vCenter was licensed with Standard Edition and the hosts which wouldn’t power on the VM were still using 60 day Evaluation licensing as they were just recently built. I further verified the Evaluation licensing had not yet expired.
I asked Carl to show me details of the VM. He proceeded to show me the VM was created with the following shell specifications:
-
Guest OS: Microsoft Windows Server 2008 (64-bit)
-
VM Version: 4
-
CPU: 8
-
Memory: 4096MB
1) Identify the root cause of Carl’s vSphere problem.
2) Identify the solution for Carl.
3) Identify the tasks or chain of events which lead to Carl’s problem.
If you think you know the answer, write it on the top of a shrink wrapped pallet containing a Cisco UCS 5100 Series Blade Server Chassis, fully loaded with UCS B200 M2 Blade Servers each with 192GB RAM, UCS 6100 Series Fabric Interconnects, and a pair of Cisco Nexus 5548P next generation 10GbE switches and send to my mailing address.
Or… reply in the comments section below.
The first correct and complete answer (I hope there is just one) will receive internet recognition, real life respect, and if I can find one, a prize. No promises on that last one but I’ll see what I can do.
1) The root cause is the VM has 8 virtual CPUs but the Hardware version (4) only supports 4 virtual CPUs.
2) Solution is to use a command line tool or the GUI to reduce the # of CPUs to 1, 2, or 4
3) You can get into this situation by using command line tools on a VM object. If the VM is a v4 hardware VM, you can change it to 8 CPUs by using PowerCLI (as an example):
Set-VM -VM $vm -NumCpu 8
This command will not error out on a v4 VM (I have tested it on VC 4.1u1) and will cause the VM to no longer power on.
Chaching!
1) Carl’s problem is that the VM hardware version is version 4 instead of version 7 and version 4 hardware won’t support 8 vCPUs
2) The solution is to upgrade the virtual hardware to version 7
3) I’m not sure this is exactly how Carl did it, but this is how I made it happen on 4.1 without U1:
1. Create a new VM, select Custom and pick v4 for Virtual Machine Version in the wizard.
2. Clone the VM you just created. At the end of the wizard be sure to select the “Edit Virtual Hardware” checkbox.
3. While the Virtual Machine Properties dialog is displayed for the new VM, change the vCPU count to 8.
The result is a v4 VM with 8 vCPUs. Also, the error message you included was not the complete list because VMware actually did a good job of telling you precisely what was wrong 🙂
VM Version 4 only supports 4-way Virtual SMP?
The problem is that VM Version 4 does not support more than 4 vCPUs. He should upgrade the VM Version to version 7, then he should be fine.
@Justin
Superior effort. However, the VM was right sized at 8 vCPUs. The solution of shifting down to 4 vCPUs wouldn’t fit the customer requirement.
Carl doesn’t know scripting or .vmx editing by hand with vi or nano. He said that he used the vSphere Client to create the 8 vCPU virtual machine.
@Brad Clarke
You nailed it. Congratulations! You are correct, using the verbage “errors include but are not limited to..” I cut one sentence of the error message out because that would have completely given away the answer to number 1 and number 2. Frankly number 1 and number 2 were easy enough. The real challenge was number three in knowing how to create an 8 vCPU on hardware version 4 using only the vSphere Client. I had one extra bit of information I witheld which would have either added confusion or gave away number 3. If nobody solved it, I would have added it in a few days. It was going to be “The guest VM has the Windows 2008 64-bit OS installed on it”.
Just a stab…
ESXi Standard only allows 4 way vSMP. He needs to license his ESX hosts to Ent Plus to leverage 8 way vSMP. Perhaps he applied the vcenter licenses, but not the ESX…
I faced the same issue on Monday.
What i did was.
1. Upgrade the Virtual hardware for that VM 2. HW Version changed from 4 to 7
3. Power off the VM
4. Change the vCPU to 8
5. Power on the VM
It worked.
I know i’m a bit late
But there is a reason it says (Exerimental) after the Edit virtual hardware checkbox 🙂 the reason is it’s not working IMAO
I had problems with customers unable to complete a deploy from template, with the error: Number of virtual devices exceeds the maximum for a given controller.
They tryed to change the portgroup befor deploying, it ran to 80% and then gave that error.
//C
Tell your graphics guy Haywood Jabuzzoff that I like his VMware Talk logo!
Hi I’m “Carl”.
At the time, the ESXi host was just recently added and was still using the evaluation license … which does allow for 8 vCPU.
I wanted to test a couple of 8 vCPU boxes on the host and deployed from an existing template that I had. (Not realizing that it was 4 hardware)
To save a step, I selected the “Edit Virtual Hardware” at the end of the wizard and changed from 2 vPCU to 8.
Clone completes fine … but can’t power on …
Changed the hardware to 7 and all is well.
Good thing to keep in mind, all those existing hardware 4 templates that you’ve been using for years … should be upgraded to 7 while you are thinking about it.
Again, Nice job Brad!
That “Edit Virtual Hardware” option was ‘Experimental’ forever and still gives me the creeps because it seems to operate outside the normal vCenter sanity/boundary checks.
I avoid it at all costs and wish I could remove it from the UI.