I ran into an disturbing problem this evening in the lab. While in the Virtual Infrastructure Client (VIC), I attempted to perform a graceful shut down on a VM by right clicking on it and choosing Shut Down Guest. Unfortunately the graceful shutdown and restart options were grayed out which is a good indicator that the VMware Tools are not installed or not running. I logged into the VM and strangely enough, the VMware Tools were installed and the VMware Tools service was running. Even stranger, when I went back to the VIC, the VMware Tools status now showed “Tools OK”.
It was then that I noticed VMware Tools status was showing “Not Running” for a whole slew of other VMs which I knew had tools installed.
A quick search uncovered a recently updated VMware KB article 1008709 “VMware Tools status shows as not running after running VMware Consolidated Backup“. Mind you, I’m not running VCB in the lab (thank God and Veeam), however, the description in the KB article mostly matched my situation.
During the normal VMware Consolidated Backup (VCB) operation, the VMware Tools status changes from OK to Not Running for some time during the initial snapshot operation, but it returns to OK after the VCB operation completes.
However, on hosts installed with the patch bundle ESX350-200901401-SG, the VMware Tools status on the virtual machines may stay as Not Running even after the VCB operation completes.
Although the KB article specifically ties the problem to VCB, the problem is not limited to VCB in my experience. Other applications that perform snapshots can cause the behavior, such as the product I’m using: Veeam Backup 3.0. The root cause stems from a January 2009 VMware patch: ESX350-200901401-SG.
There are a few work arounds, the second of which I discovered on my own:
- Restart the mgmt-vmware service immediately after the backup job is done. This changes the Tools status to OK. You can write a cron job to do it periodically.OR
- Log in and log out, or log out if you are already logged in, from the virtual machine. This changes the Tools status to OK if it was showing as Not running.OR
- Use VCBMounter to look for virtual machine name or UUID rather than virtual machine IP. Virtual machine IP only works when the status of tools is OK, but virtual machine name and UUID works even if the Tools status shows as Not running.
After reading the KB article, I ran a service mgmt-vmware restart and after about a minute, the VMware Tools status for all my VMs changed in status from “Not Running” to “Tools OK”. The host and all of its VMs briefly disconnected as well but don’t worry, they’ll come back on their own shortly.
Until VMware releases a permanent fix, it sounds like I can expect this behavior daily after each Veeam backup completes.
By the way, if you’re running VCB, this condition will cause future VCB backups to fail if the VCBMounter is set to look for the virtual machine IP rather than virtual machine name or UUID. Nobody likes failed backups so please make sure you get this sorted out in your environment if the problem exists.
I normally use VM Name instead of IP 🙂
Weird problem, haven’t encountered it yet.
I wonder if ESX 3.5 Update 4 released yesterday fixes this issue? According to the release notes, “this version of ESX Server requires a VMware Tools upgrade”. This gives me hope!
@Vigo I’m going to apply Update 4 this week and I’ll respond back here with the results.
Jason, I am experiencing the EXACT problem as you describled here. Thank you for sheding light on this.
http://communities.vmware.com/message/1214911 I believe it is VCB that triggers this problem too. Lets see if U4 will get this fix permanently. ~Cheers!
I have been seeing similar issues in my VI environment, only in my case it is constantly showing an “out of date” warning. I’ve tried removing tools, reinstalling, and both seem to work for a time. A few weeks later the tools are reporting they’re out of date again. I’ll have to take a closer look at the snapshots and backups as a the potential culprit.
As for U4. After the last couple of releases, I take my sweet time adopting new releases. Even for test environments.
Yeah, I’ve blogged on this a couple of weeks back (http://www.virtualvcp.com/content/view/82/1/).
In the mean time, I’ve updated some test ESX hosts to ESX 3.5 Update 4. They’ve been running for two days now with VCB jobs running on some VMs as well, and the problem now seems to have gone away with Update 4.
This however still leaves me with a somewhat bitter taste in my mouth as I won’t be able to update my production environment to the same ESX build for Update 4 for a couple of weeks as I first need to test the hell out of update 4 before we’ll trust it after what happened with update 2.
Regards
Rynardt Spies
http://www.virtualvcp.com
VCP/vExpert
Rynardt, thanks for the updates. I would NOT do U4 until it’s being widely taken. Lucky you have a lab for testing. 🙂
I applied u4 on 2 hosts last night (1 ESX, 1 ESXi). I also applied all ESX/ESXi patches up to date as well. This morning I updated all VMware Tools on the VMs. I also kicked off a new round of Veeam backups this morning to see if the problem still exists. So far, of the 6 VMs that have completed their backups and snapshot closures, I haven’t see VMware Tools go offline. Looking good.
The issue has been resolved with Update 4. I ran full backups and all of the VMware Tools are still showing the status of “Tools OK”