Converged Networking Adapters like this QLogic 8242 are all the rage, but which are supported in VMware ESX and which have the broadest coverage of DCB features?

VMware has one awesome hardware compatibility list. It’s not the breadth of it (in fact, ESX’s range of hardware support is astonishingly small) but the thorough, public way in which VMware shares this information. It’s really worth a look, and I regularly visit the list to see what’s changed.

But the thoroughness and detail of the VMware HCL can be daunting. It’s fairly easy to search for a specific piece of hardware, but it’s difficult to tell what’s supported in a general sense. This causes trouble for people who are shopping for hardware, since it’s hard to know what’s supported without an exhaustive search.

As part of the preparation for my Storage for Virtual Servers seminar series this year, I decided to do some research. I’ve boiled down certain key hardware categories into a general plain-english list of what’s in and out of the ESX HCL. In the spirit of openness, I’m presenting this data here for all to see, and I welcome corrections and updates. Indeed, I’ll try to keep this page up to date as new hardware is added!

Minimum Requirements

No one should run unsupported hardware in a production environment. So the very minimum requirement for every buyer should be a list in the VMware ESX HCL with the proper firmware, driver, and software.

FCoE also requires a flow control mechanism, preferably 802.1Qbb, and the DCBX protocol to enable communication between ports. I strongly suggest running 802.1Qaz bandwidth management as well, since it will allow you to get more performance from your CNAs than plain PFC.

Congestion management (802.1Qau) isn’t required, and I’m not sure about the state of affairs for interoperability. So that’s a “nice to have” at this point, though it will certainly become more desirable in the future.

You might also look for a CNA that supports other protocols over DCB, since iSCSI is an attractive alternative to FCoE in the data center, as is NFS. But that’s just my opinion!

FCoE CNAs for VMware ESX

Let’s kick things off with FCoE CNAs. I’m not a huge fan of FCoE in general, but I do see it as an increasingly-viable protocol for large-scale enterprise virtualization products.

It’s critical to have supported hardware, and I suggest researching specific models on the HCL before buying. But here’s the general state of affairs with regard to FCoE CNAs:

I was surprised to see that Brocade and QLogic list 802.1Qau Congestion Management (QCN) as supported. I didn’t realize anyone supported this spec yet, or indeed that there was such a spec!

It’s also puzzling to see that QLogic does not list 802.1Qaz Bandwidth Management (ETS) as supported for the (admittedly older) QLE8042 card. Perhaps that’s a error? But then again, Emulex doesn’t list 802.1Qbb Priority Flow Control (PFC) for the LP21000, and although FCoE doesn’t require Qbb, it’s a very good idea…

Update: Emulex tells me the LP21000 does indeed support PFC (Qbb).

Stephen’s Stance

If you’re shopping for networking gear and want to move to FCoE eventually, I hope this list will help you get started. Most OEMs supply the latest Emulex, QLogic, and Brocade CNAs with their own part number, and the long ESX HCL is testament to this fact. But, generally speaking, if you’re using a latest-generation CNA like the QLogic 8200 or Emulex OneConnect series, you ought to be fine. The Brocade CNAs look good, too, but are a little harder to find.

Emulex tells me they will add QCN support once it’s ratified, and I look forward to learning more about the Brocade and QLogic implementations. I’d also love some feedback on how well these features work, and if they’re all they’re cracked up to be. Is one vendor’s PFC or ETS as good as another’s? And does DCBX really work in practice? I believe most folks aren’t mixing HBAs widely and are instead sticking to one manufacturer. That’s probably a good idea out here on the cutting edge!

I was surprised that no Intel or Broadcom cards are listed as supporting FCoE in the HCL. The Intel X520 is impressive, and I’d taken their “open FCoE” claims to include VMware. But I guess they’re not there yet, and this seems to be a glaring omission for a major player like Intel! Similarly, Broadcom is only talking about iSCSI with VMware. Where is their CNA with VMware support?

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware ESX FCoE CNA Compatibility in Plain English
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Symposium will be held August 4th through 6th and attendance is free for qualified IT end-user folks. I’m very much looking forward to presentations by Nth President, Mark Gonzalez, Raghu Raghuram of VMware, Scott McClellan of HP, Chuck Brown of Intel, AJ Casamento of Brocade, Dave Crespi of Emulex, Kumar Malavalli of InMage, and my friend, Shannon Smith and Chris VanWagoner, both of CommVault. There will also be some excellent hands-on lab opportunities with the latest storage gear. Victoria Halsey’s Hamster Revolution talk looks great, too!

I’ll have a Birds of a Feather Lunch table on Tuesday focused on cloud storage. On Wednesday I’ll be moderating a panel on cloud computing, featuring McClellan, Raghuram, Brown, Van Wagoner, and Geoff Tudor, co-founder of Nirvanix. Thursday I get my own keynote, where I’ll be presenting Three Enterprise Storage Problems You Can Solve Today.

What: Nth Generation Symposium

Where: Disneyland, Anaheim, CA

When: August 4-6, 2009

How: Register on-line

© sfoskett for Stephen Foskett, Pack Rat, 2009. | LA Folks: Come to the Nth Generation Symposium!
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3

Expanded Support for Enhanced vmxnet Adapter

Not specifically a storage change, but the enhanced vmxnet adapter introduced back in the original release of ESX 3.5 now works with most versions of Windows Server 2003 and XP Pro. Look for improved performance when using guest-side SMB and NFS as well as the guest iSCSI initiator. Note that you cannot select this driver when configuring non-Enterprise Edition machines; you have to select Windows Server 2003 Enterprise Edition (64-bit) regardless of which version of Server 2003 you are using.

Expanded SAS and SATA Controller Support

If you’d like to install ESX on a server equipped with a PMC 8011, Intel ICH9 or ICH10, CERC 6/I SATA/SAS Integrated RAID Controller, or HP Smart Array P700m Controller, you’ll find happiness in Update 4.

The Intel controllers are especially important, as we’re seeing them used more and more and this driver is more full-featured than the earlier Broadcom HT 1000 and Intel ICH7 drivers. The Intel ICH9/ICH10 is a dual-mode (IDE/ATA and AHCI/SATA) driver, supports SATA hard drives, SSDs, and optical drives, and now enables VMFS support when in AHCI/SATA mode. It’s not clear whether VMware actually supports VMFS datastores on ICH9/10 SATA, but it says it works. Anyone want to try it out? One thing is certain: You can’t use SATA drives in a shared/clustered environment because SATA does not include reservations. See this tech note and especially this question:

Earlier, it was mentioned that we can create VMFS if we use AHCI/SATA mode. If so, why did VMware not claim VMFS support when using SATA controller running in AHCI/SATA mode?

VMware might decide to add support in the near future. There is no strong need to have VMFS support on a SATA drive, because native SATA protocol does not support reserve/release. Reserve/release is needed if VMFS is used as clustered file system in a shared disk environment.

PXE Boot Support

Rich at VM/ETC points out that Update 4 includes experimental PXE boot support for ESX and ESXi. As he notes, this has major implications for cloud computing platforms, since it means that ESX servers can boot guests without local storage at all. Very interesting! Let’s bet that Update 5 (expected in June or July) will include this as a supported option.

Updated QLogic, Emulex, and LSI Drivers

Like most ESX updates, this one included updated Fibre Channel drivers.

• The QLogic Fibre Channel Adapter driver and firmware (versions 7.08-vm66 and 4.04.06, respectively) include bug fixes and enhanced NPIV support.
• On the Emulex side, driver version 7.4.0.40 supports the company’s HBAnyware 4.0 management software.
• Users of SAS and SCSI LSI MegaRAIDs will find driver version 3.19vmw (megaraid_sas) and 2.6.48.18 vmw (mptscsi) which improves performance and enhances event handling capabilities.

Expanded Sun Storage Array Support

All you StorageTek loyalists out there will be happy to see support for Sun’s low-end StorageTek 2530 SAS array as well as the modular 6580 and 6780 Fibre Channel arrays. It looks like just about every model in Sun’s current storage lineup is now supported in ESX.

Expanded Network Card Support

Support for Gigabit cards is greatly expanded, including HP’s quad-port NC375i and dual-port NC362i and NC360m, Intel’s Gigabit CT and 82574L, and NetXtreme’s BCM5722, BCM5755, BCM5755M, and BCM5756. Intel’s widely-used 10-gig 82598EB cards are now supported as well.

Tweaks and Fixes

Looking through the release notes, a few storage-related tweaks and fixes stand out:

1. WMware can optionally automatically throttle back the queue depth when congestion is encountered. See Controlling LUN queue depth throttling in VMware ESX for 3PAR Storage Arrays for more information.
2. VMklinux module heap size can now be adjusted as LUN queue-depth values are increased. Since tuning LUN queue depths is one common trick of the storage trade to improve performance, especially in queue-stingy systems like ESX, this is welcome news. But call VMware support before you monkey with it!
3. An RDM-related issue where SCSI inquiry data over 36 bytes was truncated or corrupted (for example when using Microsoft VSS and NetApp SnapDrive) has been resolved.

Well, that’s all folks. I suggest you all read the release notes for yourself, and please leave a comment if you see an error in what I wrote here or have something to add!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Changes in VMware ESX 3.5 Update 4
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The pistol shot heard this week was the starting gun for FCoE, not the execution of iSCSI

To hear this week’s storage industry news reports, one might think that Wagner’s fat lady came to Storage Networking World (SNW), singing her song as the iSCSI world collapses. Storagebod wonders what iSCSI’s death will look like. Chris Mellor at The Register says “Game Over” as NetApp, QLogic, Emulex and VMware join EMC and Cisco in singing the praises of Fibre Channel over Ethernet (FCoE). Mellor suggests that the protocol will devalue Dell’s EqualLogic investment, as if HP’s acquisition of LeftHand wasn’t enough, even as fellow Register-ite, Bryan Betts disagrees.

But The Register didn’t invent the “FCoE kills iSCSI” meme – it’s just natural to imagine that these two protocols would be in a fight to the death. And if it’s a duel, then this year’s SNW conference would seem to be the first volley, as EMC introduced a FCoE Connectrix switch (based on Cisco), NetApp announced the first native FCoE array, and everyone qualified Emulex and QLogic adapters. However, despite these announcements, it’s way too early to bury iSCSI!

FCoE and iSCSI are similar in concept:

• Both rely on Ethernet physical connectivity
• Both transmit SCSI packets
• Both are aimed at date center users

But there are major differences as well:

• iSCSI is routable in an IP network
• iSCSI can use IP services like IPsec
• Software initiators can give iSCSI connectivity to any server, regardless of hardware
• FCoE will require converged network adapters (CNAs), while iSCSI can run on any Ethernet adapter
• FCoE will start at 10 Gb, while iSCSI can operate at just about any speed

Looking at this list, one might realize that FCoE is really a competitor for faster-than-4 Gb Fibre Channel. It’s not just a data center product, it’s an enterprise (read high-end and expensive) product, and that’s exactly where it will flourish. I have no doubt that Cisco and Brocade will successfully transition their Fibre Channel product lines to FCoE, and that QLogic and Emulex will sell a gazillion CNAs. But what about the rest of the market?

VMware’s adoption has shown that there is a taste for shared, networked storage outside the rarified budgets of the enterprise world. So far, no storage protocol has won the midrange and virtual server market, with Fibre Channel, iSCSI, and NFS duking it out along side internal SAS and SATA and the odd InfiniBand and external SAS solution. Although CNAs and FCoE ought to work fine in the virtual data center, not everyone will have a taste for them. There will always be plenty of folks who just want inexpensive external networked storage arrays, and iSCSI is the best thing they’re likely to see any time soon!

And iSCSI isn’t done evolving. Folks like Mellor, Chuck Hollis, and Storagebod are lauding FCoE at 10 gigabit speeds, but seem to forget that iSCSI can run at that speed, too. It can also run on the same CNAs and enterprise switches. And although wide(r)-area SANs aren’t common, I’m beginning to see some interest in leveraging the routability and other advanced features of IP in the storage world. iSCSI still has some cards to play! And the non-enterprise space isn’t nearly as awful as some make it sound – it is and will remain a bigger, more diverse market than the high end, and there are some serious buyers that will never get into FCoE.

Right now, the SAN world is expanding like it hasn’t done for years. iSCSI deployments are accelerating, growing the whole market. Sure, FCoE will probably completely replace old-school Fibre Channel over the next five years. But it will have to share the market with the now well-established iSCSI. It looks to me like Dell and HP made smart investments.

Update: More coverage on the topic:

• Doug Rainbolt from Alacritech is skeptical of the drivers for FCoE
• David Dale from NetApp feels that FCoE is unlikely to intrude on the iSCSI “sweet spot”

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Is the FCoE Starting Pistol Aimed at iSCSI?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This was pretty puzzling, but Rich from VM /ETC and Duncan from Yellow Bricks were right – they just hadn’t updated the guide yet (even though the revision date was July 25).

So without further ado, the announcement:  You can now use Emulex and QLogic 8 Gb Fibre Channel HBAs with VMware ESX to give blazing I/O performance right where it’s needed.  Of course, 8 Gb storage arrays remain rare, but this will change soon.

An end-to-end 8 Gb FC SAN will likely provide all the performance of 10 Gb FCoE, and it’s available now instead of 2009 or 2010.  10 Gb iSCSI and NFS are also supported as of Update 2 if you’re more of a TCP/IP person…

© sfoskett for Stephen Foskett, Pack Rat, 2008. | QLogic and Emulex Deliver 8 Gb Fibre Channel For VMware ESX
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

1. Support for Volume Shadow Copy Service (VSS) for filesystems Windows Server 2003 and Windows Server 2008 guests.  This means VMware services like VCB and SRM can now signal Windows Server to quiesce filesystems before creating a clone or snapshot and is a major addition!
2. Windows Server 2003 guests also get application quiescing, where supported.
3. You can now extend a live, running VMFS volume as storage is added, just like Windows Vista and 2008 guests could already do with raw device mode (RDM).  Note that this only works for flat disks with no persistent snapshots open.

A few items are now officially supported, too:

1. Support for Storage VMotion between and among Fibre Channel and iSCSI datastores is here, and in Update 1.
2. iSCSI over 10 gigabit Ethernet (10 GbE) is now supported allegedly, though the I/O Compatibility Guide still says “Gigabit only”.  Of course, it already worked, but it’s nice to have official support just the same…  Bring on the datacenter Ethernet!  They’ll support this in Update 1 too.
3. And NFS over 10 GbE is supported, too!

1. They supposedly added support for 8 gigabit Fibre Channel (8 Gb FC) HBAs, but I don’t see it in the I/O Compatibility Guide. The Emulex LP12000/LP12002 are noted as 4 Gb only, the QLogic QLE2560/QLE2562 aren’t listed, the HP 81/82 series isn’t there either.  Update:  QLogic and Emulex HBAs are supported after all!
2. Support for more Serial-Attached SCSI (SAS), including Dell SAS 5/E, LSI 1078, Sun SG‐XPCIESAS‐R‐EXT‐Z, and HP SC08GE is listed in the Stotage Compatibility Guide.
3. CIM system health monitoring of some QLogic and Emulex HBAs is new too.

[ad#banner]

See also:

• Scott Lowe’s (as always) excellent overview
• VM /ETC’s colorful post
• Excellent coverage at Yellow Bricks

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Storage Fixes in VMware ESX Server 3.5 Update 2
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Overall, I was impressed with many of these products this last year, but certainly CommVault, HDS, ProStor, Xsigo, and Akorri (winners all) stood out with important products. I too was surprised to see the absence of some heavyweights: Symantec’s strength in archiving and backup didn’t net them an award, and both Emulex and QLogic were skipped for the 8 Gb FC market. EMC was overlooked, too, but I’m certain that will be remedied for 2008 after the flurry of excellent new products announced these last few months.

Backup Software

• Winners: CommVault’s Simpana 7.0 – an impressive product that brings a very wide range of data protection features in an integrated package. They’d be in my top three! Next was the intriguing RecoverGuard from Continuity Software, a product which intrigues me. Finally, Yosemite’s amazingly priced FileKeeper Professional took bronze – I definitely would have put this up there, too!
• Strong contenders: EVault’s solution would have been my pick since it effectively tackles a real need – mobile and SMB data protection! Also, Symantec’s strong NetBackup 6.5 upgrade deserved more attention.

Backup Hardware

• Winners: NEC’s HydraStor HS8 isn’t a platform I’m familiar with – I’ll have to take the judges at their word! But I can’t argue with silver and bronze winners, Copan and ProStor’s cool RDX – both are truly revolutionary products and deserve the spotlight.
• Strong contenders: I would definitely have added Gresham’s Clareti VTL to the list – it’s much more than “just a VTL” and I hope it gets some more attention!

Disk and Disk Subsystems

• Winners: Seagate’s Momentus 5400 FDE.2 deserves considerable attention, bringing built-in hardware encryption to the mobile data market – every mobile disk should have this technology! NetApp’s FAS2000 (silver) is nice enough, but I was much more impressed with the bronze-winning USP V from HDS this year.
• Strong contenders: I would have given the USP V the top prize for 2007, but I can’t argue with the little Seagate disk. The AMCC 3ware 9600 RAID adapter deserves some attention, too.

Storage Management Software

• Winners: Akorri’s BalancePoint (gold) impresses me much like bronze-winning Onaro’s Application Insight does. Both make the big leap from “storage” to “application data”, and both are worthy winners. I’ve never used the silver-winning Finisar NetWisdom product, but it looks like others are impressed with it.
• Strong contenders: This was a crowded field, and Symantec, WysDM, Quantum StorNext, and the rest probably came close. I agree with the Akorri placing, but would have picked one of these instead of the potentially mis-categorized Finisar product.

Networking Equipment

• Winners: Xsigo took the top honors with their InfiniBand-based VP780 platform. This is truly a next-generation product, and it is getting serious attention and traction, and deserved a spot on the list! Riverbed’s excellent Optimization System (silver) also deserved its ranking, but I’m not familiar enough with the Storwize product to know if it’s truly bronze-worthy.
• Strong contenders: I would have tipped either QLogic or Emulex‘s 8 Gb offerings for a spot. Despite my jokes, 8 Gb FC is an important element of the modern SAN and both companies have carved out a compelling product, but apparently neither shipped in volume until this month…

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Reacting to TechTarget’s Storage Products of the Year
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

So what’s IBM doing this time? Well, they’ve taken LSI’s proven Engenio 1333 array technology (which they’ve used for a while) and turned on iSCSI functionality. IBM veers off course from the startups by not bundling snapshot and replication technology with their new array, however. This reduces the cost of entry but diminishes the impact of this new technology, since adding that software can easily double the price of this Ford Escort storage system.

IBM has also certified a complete Fibre Channel SAN solution with Microsoft, if you’re into that sort of thing. Their DS3400 can chat with an Emulex HBA and Brocade switch with quick setup and guaranteed compatability or your money back! (I made up that last part…) I’ve been pleased by Microsoft’s Simple SAN push in the past and think this is an excellent alternative to iSCSI for sites that aren’t ready to take the storage-over-Ethernet plunge yet.

Why care? Well, simply because this “compact” market is where the big action is in storage right now. Thanks to the VMware explosion, just about every smaller-than-10-TB shop is currently buying and deploying SANs right now, a fact that has warmed the hearts (and fed the sales people) at companies like EqualLogic, LeftHand, HP, Dell, and the rest for a year or so now. IBM was too early to market with an iSCSI array back in 2001, but has had nothing to sell since they axed the Adaptec-powered DS300/400 back in January. Welcome back!

© sfoskett for Stephen Foskett, Pack Rat, 2007. | IBM Warms to the Compact Storage Market
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I guess the egg comes before the chicken after all…

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Emulex and QLogic Learn to Speak Klingon!
This post was categorized as Enterprise storage, Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.