For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3
• Storage Changes in VMware ESX 3.5 Update 4
  
• Storage Changes in the VMware vSphere 4 Family

One first step is VMware’s whitepaper, “What’s New in VMware vSphere 5.0 – Storage“.

Licensing and Availability of Features

VMware has once again changed the licensing and pricing model, throwing the Internet into a tizzy:

• “Advanced” has been eliminated, moving up to “Enterprise”
• Pooled vRAM entitlements work across the entire vCenter environment
• New features like Policy-Driven Storage and Storage DRS (along with SIOC) are exclusive to “Enterprise Plus” licenses
• VAAI, PSP multipathing, and Storage vMotion are only found in “Enterprise”
• Thin Provisioning and VADP are available regardless of edition

VMware mucked with vSphere licensing again...

Major New vSphere 5 Storage Features

Storage DRS

Storage DRS is the world’s worst-kept secret, with everyone and his brother talking about it for over a year. Like the existing VM DRS capability, Storage DRS creates resource clusters and automatically moves VMs between them. Storage DRS uses utilization and performance metrics to make the call, and has three modes of operation. It sounds awesome, but it’s an Enterprise Plus-only feature.

Storage APIs – Storage Awareness (VASA)

There’s not much information presently, but a VAAI companion is introduced in vSphere 5: The vSphere Storage APIs for Storage Awareness (VASA) is a communication mechanism for vCenter to detect array capabilities like RAID Level, Thin Provisioning State, Replication State, etc. This will come in handy for all the other features in vSphere 5, especially policy-driven storage!

Policy-Driven Storage

Another new Enterprise Plus feature is Policy-Driven Storage. This allows storage tiers to be defined in vCenter based on SLA, performance, and other metrics which are used during provisioning, cloning, Storage vMotion, and Storage DRS. It leverages VASA for metrics and characterization and supports all arrays in the HCL, regardless of whether they’re NFS, iSCSI, or FC. It includes easy compliance status reporting in vCenter as well.

FCoE Software Initiator

Those of us “in the know” about storage expected VMware to add software FCoE support, so it’s no surprise that they did. This dramatically expands the potential FCoE footprint from just the few CNAs already supported in vSphere 4. It appears to be based on Intel’s OpenFCoE, since it shows up as “Intel Corporation FCoE Adapter” in the GUI!

vSphere Storage Appliance (VSA)

VMware enters the virtual storage appliance (VSA) market with their own offering, the vSphere Storage Appliance (also called VSA). Aimed primarily at the SMB market, it’s actually fairly clever, replicating storage between two or three nodes in a cluster for high availability and using NFS for access rather than iSCSI. And unlike the Celerra UBER that so many techies loved, the VMware VSA is ready for production use!

Existing Storage Features Enhanced in vSphere 5

VMFS 5

VMFS has been improved for scalability and efficiency, but the 2 TB limit on VMDKs remains (except for physical RDM). Only storage geeks like me need to worry about the specifics, but suffice to say that VMFS 5 requires less tuning and worrying and ought to scale and perform better thanks to increased maximums and leveraging the Atomic Test and Set (ATS) technology also used in VAAI. For newly-created volumes, there’s no more block size tuning, and alignment issues are addressed.

Storage APIs – Array Integration (VAAI 2)

See also VMware VAAI Storage Array Support in Plain English

VAAI has been revved, bringing back Thin Provisioning Stun (the AWOL “fourth primitive”) and adding NFS support.

There are now five block primitives for VAAI, depending on if you count thin space reclaim. This is really more of a bug fix than anything, since most folks assumed that the existing thin support already reclaimed deleted VMs and vMotioned VMDKs. I’m more interested in the addition of SCSI UNMAP in addition to WRITE_SAME! There’s also some additional T10 support, though I’m not clear on what it is or where it works.

We’ve also got VAAI for NFS environments now. NAS had sweet thin provisioning support even before block datastores, but the new Reserve Space command adds thick provisioning if that’s what you like. We’ve also got Full File Clone, which is like Full Copy for NFS but doesn’t work with Storage vMotion. And there’s some Extended Stats API to bring in more detail on file status. I also hear there’s an API for Native Snapshot Support, but it’s not widely discussed. Finally, note that NFS plugins come from vendors, not VMware as is the case for block VAAI.

Storage I/O Control

SIOC is enhanced for use in Storage DRS environments, becoming aware of the new datastore clusters. It also gets NFS support, and presumably uses VASA for metrics. But it’s still only available with Enterprise Plus licenses.

iSCSI Initiator GUI

The solid vSphere iSCSI initiator gets a friendly GUI for configuration. I like friendly iSCSI GUIs – just ask Microsoft!

Storage vMotion

The mechanism behind Storage vMotion has changed for a third time in as many releases, this time using “Mirror Mode” to mirror writes to in-progress vMotions. It also now supports migration of vSphere snapshots and Linked Clones. This can be offloaded for VAAI block, but not NFS.

vSphere Replication

New in vCenter Site Recovery Manager 5.0 is software-based replication. Although not technically a vSphere 5 feature, this is a major new storage feature in the VMware world. It allows any-to-any software-based storage replication for disaster recovery.

Stephen’s Stance

Once again, VMware added a ton of new storage enhancements to vSphere. With storage rapidly becoming the limiting factor in scalability and performance of virtual machine environments, this is no surprise. Also not surprising is the fact that major features like Policy-Driven Storage and Storage DRS (along with SIOC) are exclusive to “Enterprise Plus” licenses. I can’t blame VMware for making a buck, but it would be nice if more capabilities were available to the small shops!

I’ll be writing on all these features in detail shortly. Watch this space!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Storage Changes in VMware vSphere 5
This post was categorized as Enterprise storage, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

VMware's PSA is awash in abbreviations and options

I am often questioned during my Storage for Virtual Environments seminar presentations about VMware’s Pluggable Storage Architecture (PSA). This system is fairly straightforward and concept: VMware provides native multipathing support for a variety of storage arrays, and allows third parties to substitute their own plug-ins at various points in the stack. But the profusion of acronyms and third-party options makes it difficult for end-users to figure out what is going on. In an effort to help, I present here another entry in my “VMware storage features in plain English” series.

Note: I am more of a storage guy than a virtualization expert. I consider myself one of those end-users who have had trouble figuring out what’s going on with PSA specifically, in VMware storage features in general. I welcome comments and suggestions for corrections or improvements to this and all of my articles. Thanks for your help!

Introducing Pluggable Storage Architecture (PSA)

Pluggable storage architecture was one of the major enhancements introduced in vSphere 4. Functionally similar to Microsoft’s MPIO stack for Windows, PSA includes native multipathing support and allows vendors to plug in their own advanced features.

I find the VMware diagram confusing. Is mine more or less accurate and readable?

The ESX kernel (VMkernel) walks down through three layers when communicating with storage:

1. In the top layer, VMware native NMP or third-party MPP software decides which SATP to use, or whether to use the native interface. MASK_PATH also operates at this layer.
2. The SATP layer includes native generic path selection (active/active, active/passive), standard ALUA, as well as allowing third-party plugins (SATP) to override its behavior. The SATP monitors these paths, reports changes, and initiates fail-over on the array as needed.
3. At the PSP layer, software decides which physical channel to use for I/O requests.

There are three types of PSA plugins for vSphere 4:

1. Storage Array Type Plug-In (SATP)
2. Path Selection Plug-in (PSP)
3. A complete third-party multipathing software stack (MPP)

As is the case with VAAI, VMware includes a number of third-party plug-ins in the ESXi install. Users can simply activate many of these according to their needs, though some require additional fees and licensing.

Storage Array Type Plug-in (SATP) List

Storage Array Type Plug-Ins (SATPs) to the VMware Pluggable Storage Architecture multipathing solution for the specific characteristics of the storage array. This is very important, since each storage array design differs substantially in detail and support, especially when it comes to load-balancing and failover between controllers, ports, and paths. So it is critical for VMware to have developed a standard interface to communicate with arrays.

SATPs allow load balancing across multiple paths, intelligent path selection, and over troubled conditions such as “chatter”, when passed rapidly fail back and forth between controllers.

The SATP has critical tasks to perform in the PSA stack:

1. Decide which method of communication to use with the storage (PSA or native)
2. Monitor the health of the physical I/O channels or paths
3. Report any changes in the state of the paths up the stack
4. Perform actions required to fail over storage between controllers on the array

VMware vSphere includes a variety of generic plugins for storage arrays. I’ve identified the following:

• VMW_SATP_LOCAL – Local SATP for direct-attached devices
• VMW_SATP_DEFAULT_AA – Generic for active/active arrays
• VMW_SATP_DEFAULT_AP – Generic for active/passive arrays
• VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arrays

Although I have sometimes seen other SATP plug-ins mentioned, the following plug-ins are all that are listed in the VMware ESX Hardware Compatibility List.

• VMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGI
• VMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)
• VMW_SATP_CX – EMC/Dell CLARiiON  and Celerra (also VMW_SATP_ALUA_CX)
• VMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, Invista
• VMW_SATP_INV – EMC Invista and VPLEX
• VMW_SATP_EQL – Dell EqualLogic systems

EMC PowerPath and HDS HDLM also support a variety of storage arrays, but I would classify these as full MPP replacements for PSA, rather than SATP plug-ins.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp satp list

Path selection plug-in (PSP) List

In contrast to the diversity of VAAI and SATP plug-ins, the universe of path selection plug-ins is fairly small. Most storage arrays are supported with either Most Recently Used (MRU) or Fixed path selection approaches. Many also support Round Robin (RR) path selection. The only vendor with a specific PSP that is not also part of a full MPP (like EMC PowerPath or HDS HDLM) is Dell, which offers a special routed path selection plug-in for the EqualLogic iSCSI arrays.

• VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arrays
• VMW_PSP_FIXED – Fixed – Supports hundreds of storage arrays
• VMW_PSP_RR – Round-Robin – Supports dozens of storage arrays
• DELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arrays

As mentioned, EMC PowerPath also offers path selection as a plug-in in addition to the full MPP stack. Many other vendors offer unique path selection plug-ins, over 100 in total, but these are not specifically called out in the VMware HCL apart from their existence. I would love to learn more about them, however.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp psp list

Tell Me More About PSA!

As mentioned above, I am by no means an expert in VMware Pluggable Storage Architecture. Rather, I am interested in learning more and passing on this knowledge to others. I welcome your comments and feedback, and especially your corrections to the information presented here. I will try to keep this page updated as new versions of vSphere are introduced and as I learn more about this technology. Thank you for your help and understanding!

Further Reading:

• What’s New in VMware vSphere 4: Storage
• Next-Generation ESX Storage: A Pluggable Core Storage Architecture
• Storage Changes in the VMware vSphere 4 Family
• Pluggable Storage Architecture, exploring the next version of ESX/vCenter
• Vmware vSphere 4.1 PSA (Pluggable Storage Architecture) Understanding
• Explain the Pluggable Storage Architecture (PSA) Layout

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware PSP and SATP in Plain English
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

IBM today announced VAAI support for SVC, and promised it for the DS8000

VMware’s vStorage API for Array Integration (VAAI) is one of the most-important storage technology advances of the decade, allowing the VMware ESX to integrate and coordinate operations with supported enterprise storage arrays. After an explosion of initial support, however, VAAI implementation slowed. IBM was notably absent from the party, listing only partial support for a single product line for the better part of the year. But they’ve turned on the VAAI heat, releasing full support for the XIV and SVC and promising DS8000 in the near future.

VAAIing for Attention

For the complete updated VMware VAAI support matrix, see my post, VMware VAAI Storage Array Support in Plain English

Under vSphere 4.1, VAAI includes three “primitives”, or supported features:

1. Full copy enables the hypervisor to direct the storage array to copy data without sending it all the way “up the stack”
2. Block zeroing supports enhanced thin provisioning communication
3. Hardware-assisted locking allows the hypervisor to more-efficiently share a SCSI LUN

IBM’s original VAAI support amounted only to the basic T10 zeroing support using the generic “vmw_vaaip_t10″ plugin on the SVC and related Storwize V7000. This fell well short of the specialized three-primitive plugins offered by rivals like EMC and HDS.

IBM VAAI Support is Spreading

Far from being asleep at the wheel, IBM was busy creating a complete VAAI implementation that would be portable across a variety of storage platforms. XIV was the first beneficiary of this work, gaining full VAAI support and a listing in the VMware ESX HCL. This came in version 10.2.4 of the XIV software, released and approved in April of 2011. But insiders hinted something more was on the way.

Today, IBM announced that their “IBM_VAAIP_MODULE” plugin would support the SAN Volume Controller (SVC) and related Storwize V7000 as well. The array-side functionality will be added in version 6.2 of the software, which IBM promises to deliver in June of this year.

Now, at the IBM Storage Summit, I hear that this same code will enable the DS8000 to support all three VAAI primitives shortly. Once this is delivered, IBM’s key platforms (XIV, SVC, and DS8k) will all support VAAI, putting IBM on par with their competitors.

Stephen’s Stance

I am pleased to see IBM planning full VAAI support, since I feel this is absolutely key to future storage developments. I am further impressed that they developed a cross-platform VAAI codebase that could be deployed across such diverse architectures as XIV, SVC, and DS8k. Once this is released, IBM will finally be up to speed with their key competition.

I won’t recommend anything that’s not on the official VMware hardware compatibility list (HCL), and I’m glad to see that V7000 and XIV is now listed. It will take months DS8k to appear, though. By that time, vSphere 5 (with its rumored VAAI enhancements) will be here. Hopefully IBM will be out day-and-date with support.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | IBM Adds VAAI Support to XIV and SVC
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Now I have been asked to write an article for TechTarget on the very same subject: How to select a virtual server backup product. I’d like to request input for this piece, and hope we can work together to produce a useful list of recommendations. Note that this isn’t a buyer’s guide like the DCIG effort: There will be no exhaustive lists of functions and features here. Instead, I’m writing about the options available in a more general sense.

I came away from the Tech Field Day presentations by Veeam, Symantec, and others impressed by the creativity of virtual server backup approaches. Clearly, these aren’t the same old backup products! W. Curtis Preston quipped that a hypervisor sometimes seems like a little datacenter in a box, and that changes everything. Every action taken impacts the operation of the whole system, but the virtualization layer also adds new flexibility.

Virtual Server Backup Features

What would you look for in a virtual server backup product? Here’s my initial concept:

• Supported environments - What dependencies do the products have?
  • Do they require integration features like VCP or VADP? We should definitely talk more about the integration features offered by the hypervisor!
  • Some products support only virtual environments, while others work on traditional machines as well. And what about non-VMware support?
• Advantages and disadvantages – Applications designed for virtual environments certainly offer many advantages, but what about the drawbacks?
  • How important is it to use a single unified product across both virtual and non-virtual environments, as many traditional backup vendors suggest?
  • Is the whole “full-machine vs. file-level” argument worth having?
• Application integration – To what extent to they extend above the OS and hypervisor?
  • I’d like to talk about integration with common virtualized applications, but which are most prevalent?
  • We should talk about crash-consistent backup, too, especially when it comes to virtual machines.
  • Then there’s management integration: What options are there for automated deployment and setup? And what metrics and reports are important?
• Advanced features – These new products have some pretty impressive features!
  • Is deduplication a critical need in the backup app?
  • We should also talk about non-scheduled approaches like continuous data protection (CDP)
  • There are also alternative backup targets, like snapshots, disk/VTL, and the cloud
  • I’m also curious about throttling/QoS features
• Other areas
  • What about supportability of virtual machine backups – are there issues with companies not allowing advanced backup features to be used?
  • Do the “new guard” of virtual server backup companies offer any unique support and integration options?

I’d like to identify a few key features to look for in particular. What are the most-important elements for a virtual server backup product, as opposed to a traditional offering?

Finally, what about non-VMware environments. I know a bit about Microsoft DPM, having been at the introduction of that product, but what offerings are there for Hyper-V or Xen environments? How do they differ from VMware-focused products?

Stephen’s Stance

I welcome the input of end-users, analysts and consultants, and the product vendors themselves. I want to come up with a good guide to selecting these products, and I think an open discussion is a good way to make that happen. Who knows, it might even be a worthwhile companion to the information in the DCIG guide!

I’m not sure how the final article will be released, but I believe it will be offered as a free download on BitPipe.com. Regardless, the discussion and commentary here will remain free, open, and available. Thank you!

Updates

I’ve already had some excellent feedback from Twitter. Thanks!

• @Stu suggested I read What’s Next in VMware Backup
• @ChrisDearden says “what about talking about how to get the data out of the backup system – ie to offsite / archive etc” and “I’m not sure if trap is the word I would use. no more than all the tape based providers use the same format (“
• @WGoderis says “what about scalability , what if you need to back up hundreds or thousands of vms”

© sfoskett for Stephen Foskett, Pack Rat, 2011. | How To Select a Virtual Server Backup Product?
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

My Storage for Virtual Environments seminar is really packed with content!

Next Thursday will be the first time I present my new-for-2011 Storage for Virtual Environments seminar. I’ve worked hard on the content, and appreciate all the input I’ve had from my readers and industry contacts since I first asked for help here on my blog.

The Virtual Opportunity

So what am I going to cover in this seminar? I hit on an interesting twist while working on the content: Virtual environments pose a challenge but also create an opportunity for storage people. Virtual environments are very demanding of storage features and performance, but we have the necessary tools in our arsenal. We just have to use them!

Server virtualization is the first step to a new IT paradigm

The first of three seminar hours is spent talking about the “big picture” issues for server virtualization. We talk about the trend for IT to move in a new direction, and how server virtualization is just the first step. It’s a bridge to the past, enabling legacy applications and operating systems to me used in a more-dynamic and flexible fashion. But the real change comes next, as virtualization and containerization hits the applications and they move away from legacy infrastructure.

This InformationWeek Analytics survey shows a slowdown in the trend towards virtual servers (from their 2010 Virtualization Management Survey)

Bur the reality check for server virtualization is starting to hit us. Look at the chart above, from an InformationWeek Analytics study on Virtualization Management. Server virtualization pros are scaling back their projections for virtualization penetration this year. Although most still believe many servers will be virtualized before the end of the year, it’s not the clear majority who thought that in 2009.

The VIrtualgeek survey demonstrates VMware's amazing market share

Another great source of information is the virtualization survey conducted by Chad Sakac. Although any web poll is not as scientific as a professional survey, it does reveal some interesting facts. One stand-out fact is the overwhelming: VMware ESX really dominate market for server virtualization. Although Microsoft Hyper-V and Citrix XenServer do have users, VMware controls more than half of the market.

Next, I talk about some of the realities of server virtualization:  I contrast the simple virtual hardware environment that the guest believes it is running on with the complicated reality of the modern virtual data center. I also talk little bit about the cloud, and what this future IT infrastructure will look like.

Transitioning to Storage

Storage and network I/O have a massive impact on virtual machine performance

But this is all a prelude to the meat of the seminar content: The impact that these virtual environments will have on storage infrastructure. As one of the three pillars of server performance, storage I/O is critical. But we must also consider the integration opportunities for advanced storage technologies.

This leads me to the roadmap for the rest of the seminar. Storage folks hold the keys to the future, but they must unlock the doors. It’s time to start thinking about storage in a new way:

1. Storage presentation must adapt to the demands of the hypervisor
2. Converged I/O technologies (“Everything over Ethernet”) bring exciting new opportunities
3. Storage virtualization technology (volume management, thin provisioning, automated tearing, and so on) finally have a use in virtualized environments
4. New storage architectures, including solid-state disks, are also required

This is the real thrust of my presentation. Storage folks need to change both their assumptions and architectures to respond to the challenges of this new virtual environment.

Join In!

Want to get involved? Here’s what you can do:

1. If you’re not already, subscribe to my rss feed – I’ll use the “Virtual Storage” category for this topic
2. Check back over the next month or so as I post slides, commentary, and data regarding storage for virtualization environments
3. Consider joining me at the seminars – I have 9 dates, and I’m sure most readers would qualify to attend free of charge!
   1. Thurs. Mar. 10 – Philadelphia, PA
   2. Thurs. Jun. 16 – Phoenix, AZ
   3. Tue. Jun. 28 – London, UK
   4. Tue. Jul. 19 – Toronto, ON
   5. Thurs. Jul. 21 – Cleveland, OH
   6. Tue. Aug. 9 – Denver, CO
   7. Tue. Oct. 18 – Dallas, TX
   8. Thurs. Oct. 20 – St. Louis, MO
4. As the year progresses, I’ll be asking for clarification and posting updates to the material

I look forward to hearing from you, meeting you at the seminars, and putting some great content together!

A Note on Copyright

As promised, the seminar slides will be released using the Creative Commons license, specifically CC-BY-NC-SA 3.0. This means that anyone out there can get a copy, remix them, or reuse them as long as they attribute me, share their work with the world under the same license, and don’t try to make money off of it. As producers of this seminar series, TechTarget will have a special license allowing them to use the content.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Introducing Storage for Virtual Environments (From My Seminar)
This post was categorized as Enterprise storage, Everything, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Note: This information was based on the “VMware Storage/SAN Compatibility Guide” and is being regularly updated. Please add comments here and I will add products and change and update listings as soon as they appear in the guide!

Where, Why, and What is VAAI?

I’ve previously discussed the fact that VMware’s excellent ESX hardware compatibility list (HCL) is so comprehensive than obscures basic facts about supported products. This is especially true for VAAI, since compatibility is only noted as a footnote in individual storage array listings. It does not help matters that not all VAAI plugins support all three capabilities.

Like my previous posts regarding FCoE CNA’s, SATA and PATA chipsets, and home/lab network cards, I’ll attempt to boil down the VMware ESX HCL into plain English. This data will also be part of my Storage for Virtual Servers seminar presentation, the first of which will be given on March 10 in Philadelphia.

I’m not going to spend too much time on “what is VAAI” in this post. Instead, I suggest you read the following blog posts and VMware’s excellent guide, “What’s New in VMware vSphereTM 4.1 — Storage“?

• What is VAAI, and how does it add spice to my life as a VMware admin?
• vStorage APIs for Array Integration aka VAAI
• If You Ever Needed Convincing About VAAI…
• VMware VAAI pros and cons and the hidden fourth primitive

The Three VAAI Primitives

You ought to read the updated Complete List of VMware VAAI Primitives since it’s much more thorough and informative!

In ESX 4.1, that vStorage API for Array Integration includes three basic capabilities or primitives:

1. Blocks zeroing is a communication method for thin provisioning capable storage arrays, allowing them to quickly zero out storage capacity for later reclamation.
2. Full copy commands the storage array to make a mirror or snapshot of data without any I/O on the server hardware.
3. Hardware assisted locking enables more granular control of shared storage resources in ESX clusters

In order to support VAAI, a storage array requires two things:

1. Hardware capable of supporting one or more of the three primitives listed above
2. A software plug in for ESX enabling communication and integration

VAAI Plug In Support

Creating a VAAI plug in is not a trivial task, and not all storage arrays are yet supported. I have heard grumbling from storage vendors that EMC (the storage vendor that owns VMware) has been given early access to VAAI information, allowing them to support this feature set before their competitors. However, this has not stopped a diverse set of other unrelated storage vendors from quickly producing and releasing effective and complete VAAI plugins.

As of this writing, there are 11 array-specific plugins and one general-purpose plug in available for ESX 4.1. EMC, NetApp, 3PAR (HP), HDS, FalconStor, Fujitsu, IBM, Dell (EqualLogic), and HP (LeftHand, P9000, P2000) have produced VAAI plugins supporting all three primitives. Additionally, a cloud in supporting the T10 blocks zeroing methods is available, enabling other arrays to support this one primitive. Note that the T10 primitive should support nearly any capable array, but not all have been tested and qualified for use with it.

VAAI Support Matrix

Note that similar OEM versions (for example, Fujitsu’s FibreCAT CLARiiONs, and the Gateway/Lenovo/Acer AMS line) are also supported the same as the manufacturer’s offerings. I’ve simplified and eliminated similar models (the Dell EqualLogic PS6000E, PS6000S, PS6000X, PS6000XV, and PS6000XVS are all listed simply as PS6000).

Updates:

• IBM recently added full VAAI for the XIV, SVC, and similar Storwize V7000. I’m sure they’re also working on complete VAAI plugins for the big DS8000 systems!
• EMC certified the new VNX line for VAAI (FC only for now) and the V-Max just gained iSCSI VAAI support.
• FalconStor added VAAI for NSS, enabling any storage array to be used.
• HP created VAAI plugins for the P9500 and MSA P2000 lines as well.

Stephen’s Stance

VAAI is an exciting new capability for VMware ESX, and demonstrates the enterprise readiness of vSphere 4.1. Although not all storage arrays are yet supported, the diverse assortment listed above should cover the majority of enterprise storage environments. I fully expect that the obvious holes will be filled in soon, and I look forward to updating this list when I hear news of those product releases. I also look forward to learning of additional capabilities added as VAAI primitives in the future!

The Exhaustive List

I am attempting to keep this list up to date. My authoritative source of information is the VMware Storage Compatibility Guide. This is the only source of information I will use, since only official and supported implementations belong in production. But I welcome pointers, suggestions, and referrals for updates!

This list is complete as of February 21, 2011

Full VAAI (All 3 primitives)

• Dell
  • EqualLogic
    • iSCSI (vmw_vaaip_eql)
      • Dell EqualLogic PS4000E (Dell EqualLogic PS4000X; Dell EqualLogic PS4000XV)
      • Dell EqualLogic PS5000E (Dell EqualLogic PS5000X; Dell EqualLogic PS5000XV)
      • Dell EqualLogic PS5500E
      • Dell EqualLogic PS6000E (Dell EqualLogic PS6000S; Dell EqualLogic PS6000V; Dell EqualLogic PS6000XV; Dell EqualLogic PS6000XVS)
      • Dell EqualLogic PS6010E (Dell EqualLogic PS6010S; Dell EqualLogic PS6010X; Dell EqualLogic PS6010XV; Dell EqualLogic PS6010XVS)
      • Dell EqualLogic PS6500E (Dell EqualLogic PS6500X)
      • Dell EqualLogic PS6510E (Dell EqualLogic PS6510X)
      • EqualLogic PS100E
      • EqualLogic PS200E
      • EqualLogic PS300E
      • EqualLogic PS3600X
      • EqualLogic PS3700X
      • EqualLogic PS3800XV
        • EqualLogic PS3900XV
      • EqualLogic PS400E
      • EqualLogic PS50E
      • EqualLogic PS70E
• EMC
  • FC (vmw_vaaip_cx)
    • EMC CLARiiON CX4-120 (EMC CLARiiON CX4-120 C8/C8X; Dell/EMC CLARiiON CX4-120; Fujitsu FibreCAT CX4-120)
    • EMC CLARiiON CX4-240 C8/C8X
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8/C8X; Dell/EMC CLARiiON CX4-480)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8/C8X; Dell/EMC CLARiiON CX4-960)
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC VNX5100
    • EMC VNX5300
    • EMC VNX5500
    • EMC VNX5700
    • EMC VNX7500
  • FC/iSCSI/FCoE (VMW_VAAI_SYMM)
    • EMC Symmetrix VMAX
  • iSCSI (vmw_vaaip_cx)
    • EMC Celerra CNS
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC CLARiiON CX4-120 C8
    • EMC CLARiiON CX4-240 C8
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8)
• FalconStor
• FC (unknown)
• CDP Gateway
• CDP SA
• CDP VS
• CDPx Gateway
• IPStor Enterprise
• NSS Gateway
• NSS SA
• NSS VS
• Fujitsu
  • FC (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
  • iSCSI (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
• Hitachi
  • FC (vmw_vaaip_hds)
    • Hitachi AMS 2100 (Acer AMS2100; Gateway AMS2100; HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (Acer AMS2300; Gateway AMS2300; HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Acer AS2040; Gateway GS2040
    • Hitachi AMS 2010
    • Hitachi BR1600/BR1600E/BR1600S
    • Hitachi Virtual Storage Platform (Hitachi VP9500; HP StorageWorks P9500)
    • Nihon-Unisys Sanarena 1910
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
  • iSCSI (vmw_vaaip_hds)
    • Hitachi AMS 2010
    • Hitachi AMS 2100 (HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Hitachi BR1600E
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
    • Hitachi Virtual Storage Platform
• HP
  • 3PAR
    • FC (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
    • iSCSI (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
  • P9500
  • FC (hp-vaaip-p9000)
  • HP P9500
  • P2000
  • FC/iSCSI (hp-vaaip-p2000)
  • HP MSA P2000
  • LeftHand
    • iSCSI (vmw_vaaip_lhn)
      • HP LeftHand P4500
      • HP LeftHand P4000 VSA
      • HP LeftHand P4000sb
      • HP LeftHand P4300 (HP LeftHand P4300 G2)
      • HP LeftHand P4500 (HP LeftHand P4500 G2)
      • HP LeftHand P4800
      • HP ProLiant DL380
      • Dell 2950
      • IBM x3650
      • LeftHand NSM 160
      • LeftHand NSM 185
      • LeftHand NSM 2060 (LeftHand NSM 2060 G2)
      • LeftHand NSM 2120 (LeftHand NSM 2120 G2)
      • LeftHand NSM 260
      • LeftHand NSM 320
      • LeftHand NSM 326
      • LeftHand NSM 3650
      • LeftHand NSM 380
      • LeftHand NSM 4150
      • LeftHand VSA
• IBM
  • FC/iSCSI (IBM_VAAIP_MODULE)
    • IBM XIV
    • IBM SVC
    • IBM V7000
    • Fujitsu VS850
    • Actifio
• NetApp
  • FC (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • FCoE (VMW_VAAIP_NETAPP)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • iSCSI (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series (Fujitsu Eternus NR1000F Series Model F2040)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series (Fujitsu Eternus NR1000F Series Model F3160)
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series

T10 Block Zero Only

• Bull
  • FC (vmw_vaaip_t10)
    • Bull Optima2000
  • iSCSI (vmw_vaaip_t10)
    • Bull Optima2000c
    • Bull Optima2000i
• NEC
  • FC (vmw_vaaip_t10)
    • NEC iStorage D3-30
    • NEC iStorage D4-30
  • iSCSI (vmw_vaaip_t10)
    • NEC iStorage D3-30/D3-30i
    • NEC iStorage D4-30/D4-30i
• Fujitsu
  • iSCSI (vmw_vaaip_t10)
    • Fujitsu Eternus VS850
• IBM
  • iSCSI (vmw_vaaip_t10)
    • IBM Storwize V7000
    • IBM SVC

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware VAAI Storage Array Support in Plain English
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing
  • First up, a battery charger? La Crosse BC-700 Review: A Battery Charger That Does Not Suck
  • I urge you to See W. Curtis Preston’s Backup Central Live!
  • I was looking for The Best Network Card For VMware ESX Home Lab Machines
  • From my Network Computing blog: Unified Storage: What Is It Good For?
  • I was amazed to see the result of my HP Photosmart printer series: The Power of Negative Publicity
• Other great links
  • Chris Evans wrote a solid piece for Datamation: Virtualization and Storage: Overview, Vendor Solutions
  • Matt Simmons talks about dealing with storage: I’m here to shard data and chew bubblegum…
  • Alex Popescu’s 5 Approaches to Scalable Storage Solutions led to Jeff Darcy’s Introduction to Distributed Filesystems
  • Xangati talks Field Day: Running Comprehensive and Deep at Tech Field Day #5
  • And then there’s Wireless Field Day, courtesy of Andrew von Nagy: Shamrockin’ the Wireless Industry
  • Greg Ferro continues spreading the word on FCoTR: Presenting Silvano Gai With FCoTR Button
  • Howard Marks takes up the topic of specialized hard disk drives: Yes, Virginia, RAID Drives Are Different
  • Howard was spurred by Simon Gallagher, among others: Home Labbers beware of using Western Digital SATA HDDs with a RAID Controller
  • The continuing saga of Google Chrome and H.264 takes a weird turn: Microsoft adds H.264 support to Google Chrome
  • Simon Long fights the good fight, presenting solid technical info: VMware View Desktops: IDE or SCSI? BusLogic, LSI Logic or PVSCSI?
  • A key topic in storage is the balance between performance and capacity, as noted by Xiotech: Performance and Capacity Tradeoffs and the Rise of a New Class of Storage
  • An amusing infographic: The United States of Shame (CHART)
  • Kevin Houston takes on Cisco UCS: What Cisco Has to Do to Win the Blade Server Market
  • Finally, Jeremy Gaddis is back with Bypassing the Internet Kill Switch

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, February 4, 2011
This post was categorized as Enterprise storage, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Desktop NICs Won’t Work

Most of the network interfaces you will find at retail simply won’t work. Realtek is the dominant provider of Gigabit Ethernet controllers for motherboards and add-in cards right now, but none of their chips are natively supported by VMware ESX. The number-two slot seems to be Marvell’s Yukon chips, with Intel’s desktop controllers close behind. None of these will work, either.

Although it is possible to get a non-supported NIC to work in VMware ESX, it’s not a good idea. First, ESX won’t install unless it finds a supported NIC in the box. Then there’s quite a bit of fiddling to get the driver up and running. And you’re left with a potentially-weird configuration that might not support advanced features. It’s a much-better idea to locate and purchase a supported NIC.

Here’s what not to buy

Many inexpensive Ethernet cards and motherboards have a chip with the Realtek “digital crab” logo. None of these will work for VMware ESX.	The big “psychedelic M” identifies a Marvell controller. Skip these, too.

Selecting a Functional Home/Lab NIC

My “home/lab” network card criteria are simple:

1. They are specifically listed on the VMware ESX HCL for version 4.1 with no hacks or trickery involved
2. They cost less than $100 US
3. You can easily purchase them at retail from major online vendors (Newegg and Amazon)
4. They use PCI or PCI Express bus and have 1 or more RJ45 Gigabit Ethernet ports

It’s really amazing how few cards meet these criteria: There are really just a few cards to consider in this range.

Here’s what to buy

PCI Adapters
	The Intel Pro/1000 MT server adapter should work, and the dual-port is cheaper on Amazon (Newegg)
PCI Express (PCIe) adapters
The Intel Pro/1000 CT desktop adapter is a cheap and functional PCIe NIC (Newegg)	The Intel Pro/1000 PT server adapter is a little more expensive but potentially better-supported (Newegg)	The HP NC112T also appears to be well-supported and affordable

Although some have reported success with the very-cheap Intel Pro/1000 GT desktop adapter, I can’t recommend it. I’ve heard many negative reviews of folks trying and failing to get this adapter to work in the latest versions of VMware ESX. I think it’s worth the money to step up to the CT or PT instead!

Note also that I have not personally tried the specific adapters listed and linked here. I intend to purchase one or more over the next few months and will update this post when I do, but I welcome feedback on your experiences with them!

Stephen’s Stance

VMware ESX seems especially picky about network adapters, and the fact that it will not install without a supported NIC onboard is a real stumbling block for users. I definitely recommend picking up a well-supported NIC like the Intel Pro/1000 MT (PCI) or CT/PT (PCIe) or the HP NC112T.

My home/lab machine has two PCIe slots and two PCI slots. I had intended to use a PCI NIC, but will probably buy a Pro/1000 PT card instead. It’s affordable and called out specifically as supported in the VMware ESX HCL. Sounds good to me!

If you have a suggestion for a NIC that fits the criteria above, please do let me know. I’d love to have more choices in this list!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Best Network Card For VMware ESX Home Lab Machines
This post was categorized as Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Would you run VMware ESX on SATA? You might in a lab environment! Here's what to look for.

The VMware ESX hardware compatibility list is awesome but it’s kind of hard to wade through. It’s super-detailed, but difficult to navigate if one is browsing for compatible hardware. Although SATA and especially PATA aren’t exactly mainstream in enterprise datacenters, they’re the most-likely storage attachment for labs and tinkerers like me.

So I decided to put together a “cheat sheet” listing the compatible SATA and PATA chipsets. In the spirit of openness, I’m presenting this data here for all to see, and I welcome corrections and updates. Indeed, I’ll try to keep this page up to date as new hardware is added!

PATA Drivers for ESX

Parallel ATA isn’t widely available anymore, but those desiring to run ESX on older hardware will want to make sure it uses one of the following controllers. Sadly, there aren’t many of them, and not many “hobbyist” motherboards use these specific chipsets. But that’s the fact of it.

• AMD SB700/SP5100 and AMD SB800 series
• Intel ICH7

That’s it. Three chipsets from two vendors. Both are server-oriented, too, so they’re harder to find in cheaper desktop motherboards.

You’re likely safe if you use an Intel ICH7 server board, but most use other-brand controllers that won’t work without some hacking. And you might not want to hack on your storage drivers…

SATA Drivers for ESX

The situation is a little brighter for the new SATA standard. Although lots of serial ATA controllers remain unsupported, there are enough here that an average shopper ought to be able to spot one of them on a motherboard.

Again, buying an Intel board is preferred, though the latest Sandy Bridge chipsets (P67/H67) are notably absent. I’ve heard that the controllers may function fine, however.

• AMD SB700/SP5100
• Broadcom BCM HT1000, HT1100 (aka ServerWorks)
• Intel ESB2 (ICH6), ICH7, ICH9, ICH10 (X58)
• nVidia MCP55

Stephen’s Stance

No enterprise will probably use non-RAID SATA for a production VMware ESX server, but it’s interesting to know what’s supported. Although most of these are fairly dated (the latest platforms are notably absent), they may be backwards-compatible with the items on this list. I’d love to hear from folks who have successfully run ESX 4 on Intel’s new PCH controller found in the P67, or H67 boards specifically!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware ESX SATA and PATA Compatibility Cheat Sheet
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Converged Networking Adapters like this QLogic 8242 are all the rage, but which are supported in VMware ESX and which have the broadest coverage of DCB features?

VMware has one awesome hardware compatibility list. It’s not the breadth of it (in fact, ESX’s range of hardware support is astonishingly small) but the thorough, public way in which VMware shares this information. It’s really worth a look, and I regularly visit the list to see what’s changed.

But the thoroughness and detail of the VMware HCL can be daunting. It’s fairly easy to search for a specific piece of hardware, but it’s difficult to tell what’s supported in a general sense. This causes trouble for people who are shopping for hardware, since it’s hard to know what’s supported without an exhaustive search.

As part of the preparation for my Storage for Virtual Servers seminar series this year, I decided to do some research. I’ve boiled down certain key hardware categories into a general plain-english list of what’s in and out of the ESX HCL. In the spirit of openness, I’m presenting this data here for all to see, and I welcome corrections and updates. Indeed, I’ll try to keep this page up to date as new hardware is added!

Minimum Requirements

No one should run unsupported hardware in a production environment. So the very minimum requirement for every buyer should be a list in the VMware ESX HCL with the proper firmware, driver, and software.

FCoE also requires a flow control mechanism, preferably 802.1Qbb, and the DCBX protocol to enable communication between ports. I strongly suggest running 802.1Qaz bandwidth management as well, since it will allow you to get more performance from your CNAs than plain PFC.

Congestion management (802.1Qau) isn’t required, and I’m not sure about the state of affairs for interoperability. So that’s a “nice to have” at this point, though it will certainly become more desirable in the future.

You might also look for a CNA that supports other protocols over DCB, since iSCSI is an attractive alternative to FCoE in the data center, as is NFS. But that’s just my opinion!

FCoE CNAs for VMware ESX

Let’s kick things off with FCoE CNAs. I’m not a huge fan of FCoE in general, but I do see it as an increasingly-viable protocol for large-scale enterprise virtualization products.

It’s critical to have supported hardware, and I suggest researching specific models on the HCL before buying. But here’s the general state of affairs with regard to FCoE CNAs:

I was surprised to see that Brocade and QLogic list 802.1Qau Congestion Management (QCN) as supported. I didn’t realize anyone supported this spec yet, or indeed that there was such a spec!

It’s also puzzling to see that QLogic does not list 802.1Qaz Bandwidth Management (ETS) as supported for the (admittedly older) QLE8042 card. Perhaps that’s a error? But then again, Emulex doesn’t list 802.1Qbb Priority Flow Control (PFC) for the LP21000, and although FCoE doesn’t require Qbb, it’s a very good idea…

Update: Emulex tells me the LP21000 does indeed support PFC (Qbb).

Stephen’s Stance

If you’re shopping for networking gear and want to move to FCoE eventually, I hope this list will help you get started. Most OEMs supply the latest Emulex, QLogic, and Brocade CNAs with their own part number, and the long ESX HCL is testament to this fact. But, generally speaking, if you’re using a latest-generation CNA like the QLogic 8200 or Emulex OneConnect series, you ought to be fine. The Brocade CNAs look good, too, but are a little harder to find.

Emulex tells me they will add QCN support once it’s ratified, and I look forward to learning more about the Brocade and QLogic implementations. I’d also love some feedback on how well these features work, and if they’re all they’re cracked up to be. Is one vendor’s PFC or ETS as good as another’s? And does DCBX really work in practice? I believe most folks aren’t mixing HBAs widely and are instead sticking to one manufacturer. That’s probably a good idea out here on the cutting edge!

I was surprised that no Intel or Broadcom cards are listed as supporting FCoE in the HCL. The Intel X520 is impressive, and I’d taken their “open FCoE” claims to include VMware. But I guess they’re not there yet, and this seems to be a glaring omission for a major player like Intel! Similarly, Broadcom is only talking about iSCSI with VMware. Where is their CNA with VMware support?

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware ESX FCoE CNA Compatibility in Plain English
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.