Would you run VMware ESX on SATA? You might in a lab environment! Here's what to look for.

The VMware ESX hardware compatibility list is awesome but it’s kind of hard to wade through. It’s super-detailed, but difficult to navigate if one is browsing for compatible hardware. Although SATA and especially PATA aren’t exactly mainstream in enterprise datacenters, they’re the most-likely storage attachment for labs and tinkerers like me.

So I decided to put together a “cheat sheet” listing the compatible SATA and PATA chipsets. In the spirit of openness, I’m presenting this data here for all to see, and I welcome corrections and updates. Indeed, I’ll try to keep this page up to date as new hardware is added!

PATA Drivers for ESX

Parallel ATA isn’t widely available anymore, but those desiring to run ESX on older hardware will want to make sure it uses one of the following controllers. Sadly, there aren’t many of them, and not many “hobbyist” motherboards use these specific chipsets. But that’s the fact of it.

• AMD SB700/SP5100 and AMD SB800 series
• Intel ICH7

That’s it. Three chipsets from two vendors. Both are server-oriented, too, so they’re harder to find in cheaper desktop motherboards.

You’re likely safe if you use an Intel ICH7 server board, but most use other-brand controllers that won’t work without some hacking. And you might not want to hack on your storage drivers…

SATA Drivers for ESX

The situation is a little brighter for the new SATA standard. Although lots of serial ATA controllers remain unsupported, there are enough here that an average shopper ought to be able to spot one of them on a motherboard.

Again, buying an Intel board is preferred, though the latest Sandy Bridge chipsets (P67/H67) are notably absent. I’ve heard that the controllers may function fine, however.

• AMD SB700/SP5100
• Broadcom BCM HT1000, HT1100 (aka ServerWorks)
• Intel ESB2 (ICH6), ICH7, ICH9, ICH10 (X58)
• nVidia MCP55

Stephen’s Stance

No enterprise will probably use non-RAID SATA for a production VMware ESX server, but it’s interesting to know what’s supported. Although most of these are fairly dated (the latest platforms are notably absent), they may be backwards-compatible with the items on this list. I’d love to hear from folks who have successfully run ESX 4 on Intel’s new PCH controller found in the P67, or H67 boards specifically!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware ESX SATA and PATA Compatibility Cheat Sheet
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Converged Networking Adapters like this QLogic 8242 are all the rage, but which are supported in VMware ESX and which have the broadest coverage of DCB features?

VMware has one awesome hardware compatibility list. It’s not the breadth of it (in fact, ESX’s range of hardware support is astonishingly small) but the thorough, public way in which VMware shares this information. It’s really worth a look, and I regularly visit the list to see what’s changed.

But the thoroughness and detail of the VMware HCL can be daunting. It’s fairly easy to search for a specific piece of hardware, but it’s difficult to tell what’s supported in a general sense. This causes trouble for people who are shopping for hardware, since it’s hard to know what’s supported without an exhaustive search.

As part of the preparation for my Storage for Virtual Servers seminar series this year, I decided to do some research. I’ve boiled down certain key hardware categories into a general plain-english list of what’s in and out of the ESX HCL. In the spirit of openness, I’m presenting this data here for all to see, and I welcome corrections and updates. Indeed, I’ll try to keep this page up to date as new hardware is added!

Minimum Requirements

No one should run unsupported hardware in a production environment. So the very minimum requirement for every buyer should be a list in the VMware ESX HCL with the proper firmware, driver, and software.

FCoE also requires a flow control mechanism, preferably 802.1Qbb, and the DCBX protocol to enable communication between ports. I strongly suggest running 802.1Qaz bandwidth management as well, since it will allow you to get more performance from your CNAs than plain PFC.

Congestion management (802.1Qau) isn’t required, and I’m not sure about the state of affairs for interoperability. So that’s a “nice to have” at this point, though it will certainly become more desirable in the future.

You might also look for a CNA that supports other protocols over DCB, since iSCSI is an attractive alternative to FCoE in the data center, as is NFS. But that’s just my opinion!

FCoE CNAs for VMware ESX

Let’s kick things off with FCoE CNAs. I’m not a huge fan of FCoE in general, but I do see it as an increasingly-viable protocol for large-scale enterprise virtualization products.

It’s critical to have supported hardware, and I suggest researching specific models on the HCL before buying. But here’s the general state of affairs with regard to FCoE CNAs:

I was surprised to see that Brocade and QLogic list 802.1Qau Congestion Management (QCN) as supported. I didn’t realize anyone supported this spec yet, or indeed that there was such a spec!

It’s also puzzling to see that QLogic does not list 802.1Qaz Bandwidth Management (ETS) as supported for the (admittedly older) QLE8042 card. Perhaps that’s a error? But then again, Emulex doesn’t list 802.1Qbb Priority Flow Control (PFC) for the LP21000, and although FCoE doesn’t require Qbb, it’s a very good idea…

Update: Emulex tells me the LP21000 does indeed support PFC (Qbb).

Stephen’s Stance

If you’re shopping for networking gear and want to move to FCoE eventually, I hope this list will help you get started. Most OEMs supply the latest Emulex, QLogic, and Brocade CNAs with their own part number, and the long ESX HCL is testament to this fact. But, generally speaking, if you’re using a latest-generation CNA like the QLogic 8200 or Emulex OneConnect series, you ought to be fine. The Brocade CNAs look good, too, but are a little harder to find.

Emulex tells me they will add QCN support once it’s ratified, and I look forward to learning more about the Brocade and QLogic implementations. I’d also love some feedback on how well these features work, and if they’re all they’re cracked up to be. Is one vendor’s PFC or ETS as good as another’s? And does DCBX really work in practice? I believe most folks aren’t mixing HBAs widely and are instead sticking to one manufacturer. That’s probably a good idea out here on the cutting edge!

I was surprised that no Intel or Broadcom cards are listed as supporting FCoE in the HCL. The Intel X520 is impressive, and I’d taken their “open FCoE” claims to include VMware. But I guess they’re not there yet, and this seems to be a glaring omission for a major player like Intel! Similarly, Broadcom is only talking about iSCSI with VMware. Where is their CNA with VMware support?

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware ESX FCoE CNA Compatibility in Plain English
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3

Expanded Support for Enhanced vmxnet Adapter

Not specifically a storage change, but the enhanced vmxnet adapter introduced back in the original release of ESX 3.5 now works with most versions of Windows Server 2003 and XP Pro. Look for improved performance when using guest-side SMB and NFS as well as the guest iSCSI initiator. Note that you cannot select this driver when configuring non-Enterprise Edition machines; you have to select Windows Server 2003 Enterprise Edition (64-bit) regardless of which version of Server 2003 you are using.

Expanded SAS and SATA Controller Support

If you’d like to install ESX on a server equipped with a PMC 8011, Intel ICH9 or ICH10, CERC 6/I SATA/SAS Integrated RAID Controller, or HP Smart Array P700m Controller, you’ll find happiness in Update 4.

The Intel controllers are especially important, as we’re seeing them used more and more and this driver is more full-featured than the earlier Broadcom HT 1000 and Intel ICH7 drivers. The Intel ICH9/ICH10 is a dual-mode (IDE/ATA and AHCI/SATA) driver, supports SATA hard drives, SSDs, and optical drives, and now enables VMFS support when in AHCI/SATA mode. It’s not clear whether VMware actually supports VMFS datastores on ICH9/10 SATA, but it says it works. Anyone want to try it out? One thing is certain: You can’t use SATA drives in a shared/clustered environment because SATA does not include reservations. See this tech note and especially this question:

Earlier, it was mentioned that we can create VMFS if we use AHCI/SATA mode. If so, why did VMware not claim VMFS support when using SATA controller running in AHCI/SATA mode?

VMware might decide to add support in the near future. There is no strong need to have VMFS support on a SATA drive, because native SATA protocol does not support reserve/release. Reserve/release is needed if VMFS is used as clustered file system in a shared disk environment.

PXE Boot Support

Rich at VM/ETC points out that Update 4 includes experimental PXE boot support for ESX and ESXi. As he notes, this has major implications for cloud computing platforms, since it means that ESX servers can boot guests without local storage at all. Very interesting! Let’s bet that Update 5 (expected in June or July) will include this as a supported option.

Updated QLogic, Emulex, and LSI Drivers

Like most ESX updates, this one included updated Fibre Channel drivers.

• The QLogic Fibre Channel Adapter driver and firmware (versions 7.08-vm66 and 4.04.06, respectively) include bug fixes and enhanced NPIV support.
• On the Emulex side, driver version 7.4.0.40 supports the company’s HBAnyware 4.0 management software.
• Users of SAS and SCSI LSI MegaRAIDs will find driver version 3.19vmw (megaraid_sas) and 2.6.48.18 vmw (mptscsi) which improves performance and enhances event handling capabilities.

Expanded Sun Storage Array Support

All you StorageTek loyalists out there will be happy to see support for Sun’s low-end StorageTek 2530 SAS array as well as the modular 6580 and 6780 Fibre Channel arrays. It looks like just about every model in Sun’s current storage lineup is now supported in ESX.

Expanded Network Card Support

Support for Gigabit cards is greatly expanded, including HP’s quad-port NC375i and dual-port NC362i and NC360m, Intel’s Gigabit CT and 82574L, and NetXtreme’s BCM5722, BCM5755, BCM5755M, and BCM5756. Intel’s widely-used 10-gig 82598EB cards are now supported as well.

Tweaks and Fixes

Looking through the release notes, a few storage-related tweaks and fixes stand out:

1. WMware can optionally automatically throttle back the queue depth when congestion is encountered. See Controlling LUN queue depth throttling in VMware ESX for 3PAR Storage Arrays for more information.
2. VMklinux module heap size can now be adjusted as LUN queue-depth values are increased. Since tuning LUN queue depths is one common trick of the storage trade to improve performance, especially in queue-stingy systems like ESX, this is welcome news. But call VMware support before you monkey with it!
3. An RDM-related issue where SCSI inquiry data over 36 bytes was truncated or corrupted (for example when using Microsoft VSS and NetApp SnapDrive) has been resolved.

Well, that’s all folks. I suggest you all read the release notes for yourself, and please leave a comment if you see an error in what I wrote here or have something to add!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Changes in VMware ESX 3.5 Update 4
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on Update 2, see my article, Storage Fixes in VMware ESX Server 3.5 Update 2.

Update 3 is not as important from a storage perspective, but there are some goodies in there:

• New VMDK recovery tool (nifty!)
• Storage tweaks like fixed multipathing on IBM SVC, interrupt coalescing on QLogic 4 Gb FC HBAs, and some bug fixes
• Expanded support for SATA (but not for VMFS), SAS, and some Broadcom NICs (but still no TOE)

Read on for more details!

VMDK Recovery Tool

ESX users with Update 3 get a nifty new script for deeper inspection of disk image files on VMFS. Although I haven’t had a chance to try it out yet, it appears to be related to VCB, in that it identifies which disk blocks make up a VMDK. But the VMDK Recovery script (which runs in Service Console on ESX – sorry, ESXi!) allows you to copy these blocks to a new file.

What good is this? Well, it could certainly be used to recover VMDK files from corrupted VMFS datastores, and could serve as an undelete feature as well. And as Rich Brambley points out on VM/ETC, it could be the basis for some nifty scripted backup and restore operations as well. I’m looking forward to seeing what happens with this, but it’s definitely a minor addition.

Storage Tweaks

Update 3 includes some minor tweaks and bug fixes in storage:

• ESX used to only support most-recently used (MRU) multipathing with IBM’s SAN Volume Controller storage virtualization platform, but Update 3 allows fixed policies to be used as well.
• The VMkernel iSCSI driver gets two new advanced configuration options (Disk.UseLunReset and Disk.UseDeviceReset) to control whether multipath failover will issue a LUN reset or target reset.
• SCSI sense codes are now included in /var/log/vmkernel by default.  This should assist in deep troubleshooting of storage problems.
• The lengthy iSCSI target discovery process that used to happen whenever a discovery address was added has been removed, speeding up iSCSI storage configuration.
• Users of QLogic 4 Gb Fibre Channel HBAs will now get updated firmware and a new performance feature called interrupt coalescing. As it sounds, IC will combine interrupts from multiple I/Os, improving overall performance by reducing the number of interrupts the CPU has to handle.

Note that VMware ESX 3.5 Update 3 still doesn’t really make maximal use of 10 Gb Ethernet. Besides the lack of TCP offload engine (TOE) support, the company still doesn’t make any speed claims when people step up to faster NICs. The Known Issues list specifically says it’s for connectivity, not performance. At least the QLogic iSCSI HBA support is pretty good.

[ad#banner]

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Storage Changes in VMware ESX 3.5 Update 3
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.