vSphere, Hardware Version 7 and Hot Plug

VMware’s vSphere added hot plug features in hardware version 7 (first introduced in VMware Workstation 6.5) that were not available in the earlier version 4 virtual hardware. Virtual hardware version 7 adds the following new features to VMware virtual machines:

• LSI SAS virtual device – provides support for Windows Server 2008 fail-over cluster configurations
• Paravirtual SCSI devices – recently updated to allow booting, can allow higher-performance (greater throughput and lower CPU utilization) than the standard virtual SCSI adapter – especially in SAN environments where I/O-intensive applications are used. Currently supported in Windows Server 2003/2008 and Red Hat Linux 5 – although any version of Linux could be modified to support PVSCSI.
• IDE virtual device – useful for older OSes that don’t support SCSI drivers
• VMXNET 3 – next generation Vmxnet device with enhanced performance and enhanced networking features.
• Hot plug virtual devices, memory and CPU – supports hot add/remove of virtual devices, memory and CPU for supported OSes.

While the “upgrade” process from version 4 to version 7 is well-known, some of the side effects are not well publicised. The most obvious change after the migration from version 4 to version 7 is the affect hot plug has on the PCI bus adapters – some are now hot plug by default, including the network adapters!

Safe to remove network adapters. Really?

Note that the above example demonstrates also that the updated hardware re-enumerates the network adapters (see #3 and #4) because they have moved to a new PCI bus – one that supports hot plug. Removing the “missing” devices requires a trip to device manager (set devmgr_show_nonpresent_devices=1 in your shell environment first.) This hot plug PCI bus also allows for an administrator to mistakenly remove the device from service – potentially disconnecting tier 1 services from operations (totally by accident, of course.

Devices that can be added while the VM runs with hardware version 4

In virtual hardware version 4, only SCSI devices and hard disks were allowed to be added to a running virtual machine. Now with hardware version 7,

Devices that can be added while the VM runs with hardware version 7

additional devices (USB and Ethernet) are available for hot add. You could change memory and CPU on the fly too, if the OS supports that feature and they are enabled in the virtual machine properties prior to running the VM:

CPU and Memory Hot Plug Properties

However, the hot plug NIC issue isn’t discussed in the documentation, but Carlo Costanzo at VMwareInfo.com passes on Chris Hahn’s great tip to disable hot plug behaviour in his blog post complete with visual aids. The key is to add a new “Advanced Configuration Parameter” to the virtual machine configuration: this new parameter is called “devices.hotplug” and its value should be set to “false.” However, adding this parameter requires the virtual machine to be turned-off, so it is currently an off-line fix.

Quick Take: Red Hat and Microsoft Virtual Inter-Op

This week Red Hat and Microsoft announced support of certain of their OSes as guests in their respective hypervisor implementations: Kernel Virtual Machine (KVM) and Hyper-V, respectively. This comes on the heels of Red Hat’s Enterprise Server 5.4 announcement last month.

KVM is Red Hat’s new hypervisor that leverages the Linux kernel to accelerate support for hardware and capabilities. It was Red Hat and AMD that first demonstrated live migration between AMD and Intel-based hypervisors using KVM late last year – then somewhat of a “Holy Grail” of hypervisor feats. With nearly a year of improvements and integration into their Red Hat Enterprise Server and Fedora “free and open source” offerings, Red Hat is almost ready to strike-out in a commercially viable way.

Microsoft now officially supports the following Red Hat guest operating systems in Hyper-V:

Red Hat Enterprise Linux 5.2, 5.3 and 5.4

Red Hat likewise officially supports the following Microsoft quest operating systems in KVM:

Windows Server 2003, 2008 and 2008 R2

The goal of the announcement and associated agreements between Red Hat and Microsoft was to enable a fully supported virtualization infrastructure for enterprises with Red Hat and Microsoft assets. As such, Microsoft and Red Hat are committed to supporting their respective products whether the hypervisor environment is all Red Hat, all Hyper-V or totally heterogeneous – mixing Red Hat KVM and Microsoft Hyper-V as necessary.

“With this announcement, Red Hat and Microsoft are ensuring their customers can resolve any issues related to Microsoft Windows on Red Hat Enterprise Virtualization, and Red Hat Enterprise Linux operating on Microsoft Hyper-V, regardless of whether the problem is related to the operating system or the virtualization implementation.”

– Red Hat press release, October 7, 2009

Many in the industry cite Red Hat’s adoption of KVM as a step backwards [from Xen] requiring the re-development of significant amount of support code. However, Red Hat’s use of libvirt as a common management API has allowed the change to happen much more rapidly that critics assumptions had allowed. At Red Hat Summit 2009, key Red Hat officials were keen to point out just how tasty their “dog food” is:

Tim Burke, Red Hat’s vice president of engineering, said that Red Hat already runs much of its own infrastructure, including mail servers and file servers, on KVM, and is working hard to promote KVM with key original equipment manufacturer partners and vendors.

And Red Hat CTO Brian Stevens pointed out in his Summit keynote that with KVM inside the Linux kernel, Red Hat customers will no longer have to choose which applications to virtualize; virtualization will be everywhere and the tools to manage applications will be the same as those used to manage virtualized guests.

– Xen vs. KVM, by Pam Derringer, SearchDataCenter.com

For system integrators and virtual infrastructure practices, Red Hat’s play is creating opportunities for differentiation. With a focus on light-weight, high-performance, I/O-driven virtualization applications and no need to support years-old established processes that are dragging on Xen and VMware, KVM stands to leap-frog the competition in the short term.

SOLORI’s Take: This news is good for all Red Hat and Microsoft customers alike. Indeed, it shows that Microsoft realizes that its licenses are being sold into the enterprise whether or not they run on physical hardware. With 20+:1 consolidation ratios now common, that represents a 5:1 license to hardware sale for Microsoft, regardless of the hypervisor. With KVM’s demonstrated CPU agnostic migration capabilities, this opens the door to an even more diverse virtualization infrastructure than ever before.

On the Red Hat side, it demonstrates how rapidly Red Hat has matured its offering following the shift to KVM earlier this year. While KVM is new to Red Hat, it is not new to Linux or aggressive early adopters since being added to the Linux kernel as of 2.6.20 back in September of 2007. With support already in active projects like ConVirt (VM life cycle management), OpenNebula (cloud administration tools), Ganeti, and Enomaly’s Elastic Computing Platform, the game of catch-up for Red Hat and KVM is very likely to be a short one.