vmware-configcheck

June 11th, 2011 by jason No comments »

Detect corruption/integrity problems with your .vmx files with the following Service Console command (ESX only):

/usr/bin/vmware-configcheck | grep -v PASS

The command walks through the inventory and checks whether each virtual machine in the inventory matches the set of rules on configuration that is indicated in the rules file on the host: /etc/vmware/configrules

Why would you do this?  As stated above, the utility will detect corruption in the construct of a .vmx file.  Integrity issues in the .vmx file can cause a failed vMotion and the VM will crash.  I’ve seen it happen twice (out of tens of thousands of vMotion operations).

The command appears to have been deprecated and is not included in ESXi.

IBM x3850 NICs Lose Network Connectivity With ESXi 4.0 Update 1

June 11th, 2011 by jason No comments »

This is a heads up on an issue I ran into some time ago upgrading to VMware ESXi 4.0 Update 1 on an IBM System x3850.  Granted, it’s an aging hardware platform and fast becoming a dated issue, nonetheless this information may help someone out of a late night or weekend fiasco.

Shortly after the upgrade, VMs began experiencing intermittent losses in network connectivity.  Tied to the problem, the following error was revealed in the ESXi log files:

WARNING: LinNet: netdev_watchdog: NETDEV WATCHDOG: vmnic7: transmit timed out

The root cause turned out to be a known issue with the e1000e driver on ESXi 4.0u1 and the IBM x3850.  The issue is documented well in VMware KB Article 1010313 (Intel 82571 NICs intermittently lose connectivity with ESX 4.x).  The KB article was updated last April and appears to still be giving VMware fits as it has spread to vSphere 4.1.  According to the KB article:

This issue may occur if the Message Signaled Interrupt (MSI) mode is enabled for the e1000e driver and this mode is not supported in a server platform. This driver supports these three interrupt modes:

  • 0.Legacy
  • 1.MSI
  • 2.MSI-X

ESX 4.0 added support for Message Signaled Interrupts in network and storage drivers. The default interrupt mode for the e1000e driver under ESX 4.x is MSI (1).

The workaround according to the KB article is to configure the e1000e driver to use Legacy (0) Interrupt mode (thus disabling MSI mode) by performing the following:

  1. Open a console to the ESX or ESXi host.
  2. To configure the e1000e module option IntMode and use Legacy (0) interrupts for a 4-port NIC, run the command:
    • esxcfg-module -s IntMode=0,0,0,0 e1000e
    • Note: A mode number must be specified for each NIC port. In case of 2 quad port NICs, specify the mode 0 for all 8 ports with the command:esxcfg-module -s IntMode=0,0,0,0,0,0,0,0 e1000e
  3. On ESX host, run this command to rebuild initrd:
    • esxcfg-boot -b
    • Note: This step is not applicable to ESXi hosts.
  4. Reboot the ESX/ESXi host for the changes to take effect.

Scripted Removal Of Non-present Hardware After A P2V

June 11th, 2011 by jason No comments »

After converting a physical machine to a virtual machine, it is considered a best practice to remove unneeded applications, software, services, and device drivers which were tied to the physical machine but no longer applicable to the present day virtual machine.  Performing this task from time to time manually isn’t too bad but at large scale, a manual process becomes inefficient.  There are tools available which will automate the process of removing unneeded device drivers (sometimes referred to as ghost hardware).  A former colleage put together a scripted solution for Windows VMs which I’m sharing here. 

Copy the .zip file to the virtual machine local hard drive, extract it, and follow the instructions in the readme.txt file.  I have not thoroughly tested the tool.  No warranties – use at your own risk.  I would suggest using it on a test machine first to become comfortable with the process before using it on production machines or using on a large scale basis.

Download: remnonpresent.zip (719KB)

VMunderground BPaaS (Beantown Party as a Service)

June 3rd, 2011 by jason No comments »

Event:  VMunderground BPaaS (Beantown Party as a Service): Tech Field Day 6 Edition

Calling all New England VMUG members, guests, and potential sponsors, as of Friday evening, there are still plenty of tickets available for the upcoming VMunderground BPaaS (Beantown Party as a Service).  I’ll be there and I hope to see you there as well.  Following are the announcement details from VMunderground:

VMunderground is happy to announce that we’ll be helping throw a virtualizaiton community bash with the Tech Field Day 6 crew!!  If you are in the Boston area or local enough to Fenway, mark your calendars for Thursday night, June 9.  It will be the mother of all FCoTR user group meetings!!  Planning a dance fight for FCoTR supporters versus the FCoAC (Avian Carrier) supporters.  It will be spectacular!

Seriously, it’s on. It’s happening. And their will be a big giant green monster involved.  Tech Field Day has the EMC Club at historic Fenway Park in Boston and TFD is opening up the event attendence to VMunderground! We have around 100 seats we’re looking to fill.  If you’re in the Boston area this night clear your calendar and sign up for a ticket and get ready to bring your “A” game to the VMunderground BPaaS (Beantown Party as a Service).   

If you’ve had the fortune of attending a VMunderground WuPaaS (Warm-up Party as a Service) at the last two VMworlds, you know that VMunderground is a blast.  Some of the smartest virtualization minds in the world, best sponsors, and always some good food, brews and conversations.  We’re currently rounding up some great sponsors to help support this community focused event. Stay tuned to learn more about this awesome opportunity to rub elbows with Boston’s finest vPeople and the Tech Field Day crew.  

If you’re interested in sponsoring the first installment of BPaaS, please contact Theron Conrey and Sean Clark at beantown@vkaboom.com (parent company of VMunderground).  We have a few slots open for sponsorship, but speak up soon to reserve your spot.

ATTENDEES:  Please note, this is NOT the VMworld party.  This is a first-time joint collaboration with Tech Field Day and you should probably live within Boston area before committing to this event.  VMunderground rocks, but not maybe not enough to fly into Boston for.  😉

TICKETS: Tickets go on sale in two 50 ticket batches: Friday, May 27 at 4 PM Eastern time.  And the evening of Monday, June 6.  There will be 100 tickets.  When they’re gone, they’re gone. Watch twitter or this site for more info.  @vSeanClark and @TheronConrey are a good place to start.

Co-scheduling Visualized

May 21st, 2011 by jason No comments »

I stumbled onto this time lapse video of 51 airplanes taking off (and others taxiing) at Boston’s Logan International Airport.  One thought immediately popped into my mind: co-scheduling, which is a function of The VMware vSphere CPU Scheduler.  The accelerated speed of the video really pronounced the importance of precision the scheduler is responsible for, which in this case is the air traffic controller (or controllers).

httpv://www.youtube.com/watch?v=3k-xG8XX1EM

How does this video relate to co-scheduling?

  • Imagine the planes represent CPU execution (or more accurately CPU execution requests).
  • Imagine the various runways & taxiways represent the number of vCPUs in a VM.

The scheduler is responsible for managing the traffic, making sure there’s a clear path for each plane to move forward and to be on time. 

  • With less runways and taxiways (vCPUs in a VM), scheduling complexity is reduced.
  • Adding runways and taxiways (vCPUs in a VM) increases scheduling complexity but with a limited number of planes (guest OS CPU execution requests), scheduling will still be manageable and planes will arrive on time.
  • Now add a significant number of planes (4 vCPU, 8 vCPU) to our multitude of criss/crossing runways and taxiways.  The precision required to avoid accidents and maintain fairness becomes extremely complex.  The result is high %RDY time for VMs on the host.

How do we deal with scheduling complexity?

  1. Right size VMs whether they are new builds or P2V.  A minimalist approach to resource guarantees is the best place to start when we’re working with consolidated infrastructure and shared resources.
  2. If you’ve already right sized VMs and you’re running into high %RDY times:
    • Balance workloads by mixing VMs having both lower and higher number of vCPUs on the same host/cluster
    • Add cores to the host/cluster by:
      • Scaling up (increasing the core count in the host)
      • Scaling out (increasing the number of hosts in the cluster)

(Video source: @GuyKawasaki‘s Holy Kaw!)

He is serious, and don’t call him Scott

May 20th, 2011 by jason No comments »

5-20-2011 10-47-54 AMHappy Friday!  Today’s treat is the announcement of a new tech blog by my friend in VMware virtualization, Microsoft SQL,  and the occassional fine cigar, Todd Scalzott (@tscalzott).  I love the title of his blog: Don’t Call Me Scott.  Content focus will be Tech ramblings from a guy named Todd, too often called Scott.  I’m looking forward to what you have to share Todd!

Howdy Partner

May 17th, 2011 by jason No comments »

I started my IT career working as a contractor in both short and long term engagements at medium to large customer sites.  Since then, and for the past 13+ years, I’ve grown my career in a customer role.  Along the way, I’ve picked up a tremendous amount of experience and expertise across several technologies.  VMware virtualization came onto the scene and I was drawn to specialize in… well, you know the story there. 

At present, I work for a great company and on a daily basis I’m at the helm of the largest vSphere implementation I’ve ever seen and possibly one of the largest in the region.  I’ve networked, made a lot of friends, maybe a few enemies, and I’ve been the recipient of an unmeasurable amount of opportunity, kindness, and generosity available only to customers in the VMware community.  However, from a role and operational aspect, I feel I’ve reached the peak of the mountain and I’ve seen and experienced all of the challenges that this mountain has to offer.  It’s time to try another mountain.

I’m hanging up my customer hat.  On Monday of next week, I begin a new role with Dell Compellent, a VMware Technology Alliance Partner.  I’ll have two titles:  Tactical Marketing Senior Advisor and Virtualization Product Specialist.  Each speaks to a degree of what my various responsibilities will entail.  My VMware experience will be leveraged continuously as I provide SME technical expertise to Storage Architects, Business Partners, and Customers on design, planning, and integration.  In addition, I’ll be involved with consulting, product demos, solution certification, white papers, and reference architectures.  In summary, I’ll be splitting my time between colleagues, customers, and more lab infrastructure than I might know what to do with, and at the same time exercising more of my design muscles.

So what does all this mean and how is it going to change Jason?  Let’s go through the list of things which come to my mind:

  • The VMware Virtualization Evangelist stays, though independent of this news I have been thinking about shortening the title to VMware vEvangelist (thoughts?).  That said, I’ll need to provide extra thought in what and how I write.  It is my underlying intent to deliver this news not from the standpoint of “hey, I got a new job”, but more importantly to instantiate the necessary transparency and disclosure from this point on.  This blog (and my twitter account @jasonboche) has always been and will continue to be mine.  I’ve made it quite clear in the past that my writing is my own and not the opinion or view of my employer.  This carries forward and I will continue to be an independent voice as much as possible but the fact that I work for a VMware Partner in the future will be inescapable.  Which brings me to the next point…
  • VMware’s policy is that, other than a few people which were grandfathered in, VMware Partners cannot be VMware User Group (VMUG) leaders.  I’ve been the Minneapolis VMUG leader for close to 5 years.  I’ve been involved with the group since the beginning when it was founded by @tbecchetti.  Although Dell Compellent was allowing me to continue carrying the VMUG torch, VMware forbids it.  It’s a fair policy and I agree 100% with it.  The Minneapolis VMUG members own and operate the group and this is clearly what’s best for the charter and its members.  A few weeks ago, I began the transition plan with the help of VMware and have talked with several potential candidates for taking over the VMUG leader role.  If I haven’t talked to you yet and you’re interested in leading or co-leading the group, please contact me via email expressing your interest.  Be sure to leave your name and contact information.  Our group has a quarterly meeting coming up this Friday which I’ll be conducting business as usual.  Our Q3 meeting in September is where I’ll likely be stepping down and introducing the new leader(s).
  • I’m still attending Gestalt Tech Field Day 6 evening activities in Boston 6/8 – 6/11, but I will not formally be a delegate nor will I be a delegate going forward as I’m no longer considered independent.  Again, Gestalt IT guidelines and I completely get it, it’s what is best for the group.  I’m looking forward to seeing some old friends as well as new faces from **I can’t let the cat out of the bag just yet, area locals will find out soon**.
  • I’m going to get my hands on kit which I’ve not had the chance to work with in the past.  Don’t be completely surprised if future discussion involves Dell Compellent.  At the same time, don’t automatically jump to a conclusion that I’ve transformed into a puppet.  Cool technology motivates me and is ultimately responsible for where I am at today.  I enjoy sharing the knowledge with peers when and where I can.  I believe that by sharing, everyone wins.
  • VMworld – you’ll probably see me at the booth.
  • Partner Exchange – I may be there as well.
  • VMworld Europe – I hope but not counting on it.  I didn’t ask.

I think that covers everything.  Compellent is a local (to me) storage company which I like.  I think Dell will add a lot of strength, opportunity, and growth.  I’m excited to say the least!

Jas