VMware's PSA is awash in abbreviations and options

I am often questioned during my Storage for Virtual Environments seminar presentations about VMware’s Pluggable Storage Architecture (PSA). This system is fairly straightforward and concept: VMware provides native multipathing support for a variety of storage arrays, and allows third parties to substitute their own plug-ins at various points in the stack. But the profusion of acronyms and third-party options makes it difficult for end-users to figure out what is going on. In an effort to help, I present here another entry in my “VMware storage features in plain English” series.

Note: I am more of a storage guy than a virtualization expert. I consider myself one of those end-users who have had trouble figuring out what’s going on with PSA specifically, in VMware storage features in general. I welcome comments and suggestions for corrections or improvements to this and all of my articles. Thanks for your help!

Introducing Pluggable Storage Architecture (PSA)

Pluggable storage architecture was one of the major enhancements introduced in vSphere 4. Functionally similar to Microsoft’s MPIO stack for Windows, PSA includes native multipathing support and allows vendors to plug in their own advanced features.

I find the VMware diagram confusing. Is mine more or less accurate and readable?

The ESX kernel (VMkernel) walks down through three layers when communicating with storage:

1. In the top layer, VMware native NMP or third-party MPP software decides which SATP to use, or whether to use the native interface. MASK_PATH also operates at this layer.
2. The SATP layer includes native generic path selection (active/active, active/passive), standard ALUA, as well as allowing third-party plugins (SATP) to override its behavior. The SATP monitors these paths, reports changes, and initiates fail-over on the array as needed.
3. At the PSP layer, software decides which physical channel to use for I/O requests.

There are three types of PSA plugins for vSphere 4:

1. Storage Array Type Plug-In (SATP)
2. Path Selection Plug-in (PSP)
3. A complete third-party multipathing software stack (MPP)

As is the case with VAAI, VMware includes a number of third-party plug-ins in the ESXi install. Users can simply activate many of these according to their needs, though some require additional fees and licensing.

Storage Array Type Plug-in (SATP) List

Storage Array Type Plug-Ins (SATPs) to the VMware Pluggable Storage Architecture multipathing solution for the specific characteristics of the storage array. This is very important, since each storage array design differs substantially in detail and support, especially when it comes to load-balancing and failover between controllers, ports, and paths. So it is critical for VMware to have developed a standard interface to communicate with arrays.

SATPs allow load balancing across multiple paths, intelligent path selection, and over troubled conditions such as “chatter”, when passed rapidly fail back and forth between controllers.

The SATP has critical tasks to perform in the PSA stack:

1. Decide which method of communication to use with the storage (PSA or native)
2. Monitor the health of the physical I/O channels or paths
3. Report any changes in the state of the paths up the stack
4. Perform actions required to fail over storage between controllers on the array

VMware vSphere includes a variety of generic plugins for storage arrays. I’ve identified the following:

• VMW_SATP_LOCAL – Local SATP for direct-attached devices
• VMW_SATP_DEFAULT_AA – Generic for active/active arrays
• VMW_SATP_DEFAULT_AP – Generic for active/passive arrays
• VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arrays

Although I have sometimes seen other SATP plug-ins mentioned, the following plug-ins are all that are listed in the VMware ESX Hardware Compatibility List.

• VMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGI
• VMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)
• VMW_SATP_CX – EMC/Dell CLARiiON  and Celerra (also VMW_SATP_ALUA_CX)
• VMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, Invista
• VMW_SATP_INV – EMC Invista and VPLEX
• VMW_SATP_EQL – Dell EqualLogic systems

EMC PowerPath and HDS HDLM also support a variety of storage arrays, but I would classify these as full MPP replacements for PSA, rather than SATP plug-ins.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp satp list

Path selection plug-in (PSP) List

In contrast to the diversity of VAAI and SATP plug-ins, the universe of path selection plug-ins is fairly small. Most storage arrays are supported with either Most Recently Used (MRU) or Fixed path selection approaches. Many also support Round Robin (RR) path selection. The only vendor with a specific PSP that is not also part of a full MPP (like EMC PowerPath or HDS HDLM) is Dell, which offers a special routed path selection plug-in for the EqualLogic iSCSI arrays.

• VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arrays
• VMW_PSP_FIXED – Fixed – Supports hundreds of storage arrays
• VMW_PSP_RR – Round-Robin – Supports dozens of storage arrays
• DELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arrays

As mentioned, EMC PowerPath also offers path selection as a plug-in in addition to the full MPP stack. Many other vendors offer unique path selection plug-ins, over 100 in total, but these are not specifically called out in the VMware HCL apart from their existence. I would love to learn more about them, however.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp psp list

Tell Me More About PSA!

As mentioned above, I am by no means an expert in VMware Pluggable Storage Architecture. Rather, I am interested in learning more and passing on this knowledge to others. I welcome your comments and feedback, and especially your corrections to the information presented here. I will try to keep this page updated as new versions of vSphere are introduced and as I learn more about this technology. Thank you for your help and understanding!

Further Reading:

• What’s New in VMware vSphere 4: Storage
• Next-Generation ESX Storage: A Pluggable Core Storage Architecture
• Storage Changes in the VMware vSphere 4 Family
• Pluggable Storage Architecture, exploring the next version of ESX/vCenter
• Vmware vSphere 4.1 PSA (Pluggable Storage Architecture) Understanding
• Explain the Pluggable Storage Architecture (PSA) Layout

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware PSP and SATP in Plain English
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Note: This information was based on the “VMware Storage/SAN Compatibility Guide” and is being regularly updated. Please add comments here and I will add products and change and update listings as soon as they appear in the guide!

Where, Why, and What is VAAI?

I’ve previously discussed the fact that VMware’s excellent ESX hardware compatibility list (HCL) is so comprehensive than obscures basic facts about supported products. This is especially true for VAAI, since compatibility is only noted as a footnote in individual storage array listings. It does not help matters that not all VAAI plugins support all three capabilities.

Like my previous posts regarding FCoE CNA’s, SATA and PATA chipsets, and home/lab network cards, I’ll attempt to boil down the VMware ESX HCL into plain English. This data will also be part of my Storage for Virtual Servers seminar presentation, the first of which will be given on March 10 in Philadelphia.

I’m not going to spend too much time on “what is VAAI” in this post. Instead, I suggest you read the following blog posts and VMware’s excellent guide, “What’s New in VMware vSphereTM 4.1 — Storage“?

• What is VAAI, and how does it add spice to my life as a VMware admin?
• vStorage APIs for Array Integration aka VAAI
• If You Ever Needed Convincing About VAAI…
• VMware VAAI pros and cons and the hidden fourth primitive

The Three VAAI Primitives

You ought to read the updated Complete List of VMware VAAI Primitives since it’s much more thorough and informative!

In ESX 4.1, that vStorage API for Array Integration includes three basic capabilities or primitives:

1. Blocks zeroing is a communication method for thin provisioning capable storage arrays, allowing them to quickly zero out storage capacity for later reclamation.
2. Full copy commands the storage array to make a mirror or snapshot of data without any I/O on the server hardware.
3. Hardware assisted locking enables more granular control of shared storage resources in ESX clusters

In order to support VAAI, a storage array requires two things:

1. Hardware capable of supporting one or more of the three primitives listed above
2. A software plug in for ESX enabling communication and integration

VAAI Plug In Support

Creating a VAAI plug in is not a trivial task, and not all storage arrays are yet supported. I have heard grumbling from storage vendors that EMC (the storage vendor that owns VMware) has been given early access to VAAI information, allowing them to support this feature set before their competitors. However, this has not stopped a diverse set of other unrelated storage vendors from quickly producing and releasing effective and complete VAAI plugins.

As of this writing, there are 11 array-specific plugins and one general-purpose plug in available for ESX 4.1. EMC, NetApp, 3PAR (HP), HDS, FalconStor, Fujitsu, IBM, Dell (EqualLogic), and HP (LeftHand, P9000, P2000) have produced VAAI plugins supporting all three primitives. Additionally, a cloud in supporting the T10 blocks zeroing methods is available, enabling other arrays to support this one primitive. Note that the T10 primitive should support nearly any capable array, but not all have been tested and qualified for use with it.

VAAI Support Matrix

Note that similar OEM versions (for example, Fujitsu’s FibreCAT CLARiiONs, and the Gateway/Lenovo/Acer AMS line) are also supported the same as the manufacturer’s offerings. I’ve simplified and eliminated similar models (the Dell EqualLogic PS6000E, PS6000S, PS6000X, PS6000XV, and PS6000XVS are all listed simply as PS6000).

Updates:

• IBM recently added full VAAI for the XIV, SVC, and similar Storwize V7000. I’m sure they’re also working on complete VAAI plugins for the big DS8000 systems!
• EMC certified the new VNX line for VAAI (FC only for now) and the V-Max just gained iSCSI VAAI support.
• FalconStor added VAAI for NSS, enabling any storage array to be used.
• HP created VAAI plugins for the P9500 and MSA P2000 lines as well.

Stephen’s Stance

VAAI is an exciting new capability for VMware ESX, and demonstrates the enterprise readiness of vSphere 4.1. Although not all storage arrays are yet supported, the diverse assortment listed above should cover the majority of enterprise storage environments. I fully expect that the obvious holes will be filled in soon, and I look forward to updating this list when I hear news of those product releases. I also look forward to learning of additional capabilities added as VAAI primitives in the future!

The Exhaustive List

I am attempting to keep this list up to date. My authoritative source of information is the VMware Storage Compatibility Guide. This is the only source of information I will use, since only official and supported implementations belong in production. But I welcome pointers, suggestions, and referrals for updates!

This list is complete as of February 21, 2011

Full VAAI (All 3 primitives)

• Dell
  • EqualLogic
    • iSCSI (vmw_vaaip_eql)
      • Dell EqualLogic PS4000E (Dell EqualLogic PS4000X; Dell EqualLogic PS4000XV)
      • Dell EqualLogic PS5000E (Dell EqualLogic PS5000X; Dell EqualLogic PS5000XV)
      • Dell EqualLogic PS5500E
      • Dell EqualLogic PS6000E (Dell EqualLogic PS6000S; Dell EqualLogic PS6000V; Dell EqualLogic PS6000XV; Dell EqualLogic PS6000XVS)
      • Dell EqualLogic PS6010E (Dell EqualLogic PS6010S; Dell EqualLogic PS6010X; Dell EqualLogic PS6010XV; Dell EqualLogic PS6010XVS)
      • Dell EqualLogic PS6500E (Dell EqualLogic PS6500X)
      • Dell EqualLogic PS6510E (Dell EqualLogic PS6510X)
      • EqualLogic PS100E
      • EqualLogic PS200E
      • EqualLogic PS300E
      • EqualLogic PS3600X
      • EqualLogic PS3700X
      • EqualLogic PS3800XV
        • EqualLogic PS3900XV
      • EqualLogic PS400E
      • EqualLogic PS50E
      • EqualLogic PS70E
• EMC
  • FC (vmw_vaaip_cx)
    • EMC CLARiiON CX4-120 (EMC CLARiiON CX4-120 C8/C8X; Dell/EMC CLARiiON CX4-120; Fujitsu FibreCAT CX4-120)
    • EMC CLARiiON CX4-240 C8/C8X
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8/C8X; Dell/EMC CLARiiON CX4-480)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8/C8X; Dell/EMC CLARiiON CX4-960)
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC VNX5100
    • EMC VNX5300
    • EMC VNX5500
    • EMC VNX5700
    • EMC VNX7500
  • FC/iSCSI/FCoE (VMW_VAAI_SYMM)
    • EMC Symmetrix VMAX
  • iSCSI (vmw_vaaip_cx)
    • EMC Celerra CNS
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC CLARiiON CX4-120 C8
    • EMC CLARiiON CX4-240 C8
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8)
• FalconStor
• FC (unknown)
• CDP Gateway
• CDP SA
• CDP VS
• CDPx Gateway
• IPStor Enterprise
• NSS Gateway
• NSS SA
• NSS VS
• Fujitsu
  • FC (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
  • iSCSI (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
• Hitachi
  • FC (vmw_vaaip_hds)
    • Hitachi AMS 2100 (Acer AMS2100; Gateway AMS2100; HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (Acer AMS2300; Gateway AMS2300; HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Acer AS2040; Gateway GS2040
    • Hitachi AMS 2010
    • Hitachi BR1600/BR1600E/BR1600S
    • Hitachi Virtual Storage Platform (Hitachi VP9500; HP StorageWorks P9500)
    • Nihon-Unisys Sanarena 1910
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
  • iSCSI (vmw_vaaip_hds)
    • Hitachi AMS 2010
    • Hitachi AMS 2100 (HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Hitachi BR1600E
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
    • Hitachi Virtual Storage Platform
• HP
  • 3PAR
    • FC (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
    • iSCSI (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
  • P9500
  • FC (hp-vaaip-p9000)
  • HP P9500
  • P2000
  • FC/iSCSI (hp-vaaip-p2000)
  • HP MSA P2000
  • LeftHand
    • iSCSI (vmw_vaaip_lhn)
      • HP LeftHand P4500
      • HP LeftHand P4000 VSA
      • HP LeftHand P4000sb
      • HP LeftHand P4300 (HP LeftHand P4300 G2)
      • HP LeftHand P4500 (HP LeftHand P4500 G2)
      • HP LeftHand P4800
      • HP ProLiant DL380
      • Dell 2950
      • IBM x3650
      • LeftHand NSM 160
      • LeftHand NSM 185
      • LeftHand NSM 2060 (LeftHand NSM 2060 G2)
      • LeftHand NSM 2120 (LeftHand NSM 2120 G2)
      • LeftHand NSM 260
      • LeftHand NSM 320
      • LeftHand NSM 326
      • LeftHand NSM 3650
      • LeftHand NSM 380
      • LeftHand NSM 4150
      • LeftHand VSA
• IBM
  • FC/iSCSI (IBM_VAAIP_MODULE)
    • IBM XIV
    • IBM SVC
    • IBM V7000
    • Fujitsu VS850
    • Actifio
• NetApp
  • FC (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • FCoE (VMW_VAAIP_NETAPP)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • iSCSI (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series (Fujitsu Eternus NR1000F Series Model F2040)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series (Fujitsu Eternus NR1000F Series Model F3160)
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series

T10 Block Zero Only

• Bull
  • FC (vmw_vaaip_t10)
    • Bull Optima2000
  • iSCSI (vmw_vaaip_t10)
    • Bull Optima2000c
    • Bull Optima2000i
• NEC
  • FC (vmw_vaaip_t10)
    • NEC iStorage D3-30
    • NEC iStorage D4-30
  • iSCSI (vmw_vaaip_t10)
    • NEC iStorage D3-30/D3-30i
    • NEC iStorage D4-30/D4-30i
• Fujitsu
  • iSCSI (vmw_vaaip_t10)
    • Fujitsu Eternus VS850
• IBM
  • iSCSI (vmw_vaaip_t10)
    • IBM Storwize V7000
    • IBM SVC

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware VAAI Storage Array Support in Plain English
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Hard disk drive makers are adding flash storage to their conventional spinning-platter drives to improve performance and are targeting the performance PC market. Wait a second, haven’t we seen this before? As Rocky eventually said to Bullwinkle, “but that trick never works!”

Flash as a Cache

Using flash memory as a disk cache is a pretty good idea. Flash has awesome random read performance and fairly good write speed (compared to a hard disk drive, at least). That’s why more and more enterprise storage vendors are adding flash as a disk cache, not just a plain tier of storage.

EMC is the latest to make the move, announcing “FAST Cache” for their midrange Clariion and Celerra enterprise storage systems last week. They join NetApp, Sun, and others already offering similar capability. Fusion-IO has been the champion PCIe flash provider, but STEC is expected to join them soon.

See my post, Is Flash A Disk Or A Cache?

Flash-as-a-cache hasn’t been as easy to roll out as flash-as-a-disk, but it promises to be more effective. An array that completely integrates flash can take advantage of its positives (fast random read, fast-ish write, low power) without stumbling over its shortcomings (big write blocks, shorter lifespan).

Flash in a Disk

Although EMC is doing the right thing by adding FAST Cache, their implementation uses disk drive form factor flash rather than the PCI cards selected by others. It may prove more-difficult to optimize the system for the characteristics of flash when one is writing through a conventional disk drive interface like Fibre Channel or SAS. Would-be flash-and-platter drives face the same issue: How do you use flash effectively when it’s abstracted from the server and presented as a conventional disk?

The hybrid hard disk drive (H-HDD) method, rolled out back in 2007, added ATA commands allowing a compatible operating system to specify whether data sent to a hybrid drive should be written to flash or disk. These products were paired with Windows Vista’s ReadyBoost and SuperFetch to produce performance gains that never materialized in practice. The so-called “ReadyDrive” has become a footnote in history, along with Intel’s “Robeson” effort to add a flash cache to the motherboard.

It is unclear what the new generation of hybrid hard drives allegedly on the drawing boards at Toshiba and Seagate would look like. It is unlikely that they would use the H-HDD interface, but they will likely be aimed at the same performance laptop and desktop market. Servers have continued migrating towards advanced enterprise storage systems that pack their own cache, reducing the impact of bare hybrid drives.

Ingredients for Success

Rather than repeat the mistakes of the past, these companies could integrate real smarts into the disk controller, allowing it to autonomously move data to the flash cache to improve everyday performance without any special operating system support. This “tiered storage in a can” approach might deliver the goods that old-fashioned H-HDDs never could.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Hybrid SSD/Hard Disk Drives: This Time For Sure!
This post was categorized as Enterprise storage, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Fear

Back when I used to teach the “Storage 101″ session at Storage Decisions, I was consistently amazed to find little awareness of enterprise NAS systems. People complained about LUNs and Fibre Channel but when I suggested using NFS or SMB their heads almost exploded. “We would never use that for application storage,” they shouted. “File servers are for home directories, not data center stuff!” Clearly, NAS faces an uphill battle.

In a recent piece I wrote, I referred to what I consider to be the prime best practice: Use the right tool for the job. It’s a simple statement, and one that resonates beyond IT and the technology world. But it can be devilishly difficult to see what the right tool is sometimes. Why not use NAS for virtual machine storage? NetApp has been beating that drum for years, yet NAS has a very small footprint in VMware. How about databases on NFS? Exchange over SMB? Block storage has a massive lead over NAS in all of these areas.

Rear this “best practices” piece, Use Process Solutions For Process Problems, Technical Solutions For Technical Ones

IT folks seem downright fearful of file-level storage protocols. Has NFS really burned them that badly over the decades? Can SMB/CIFS really be as bad as they think?

Loathing

I wonder if this terror has more to do with the products people have used than the fundamental concept of file services. Many NAS servers (and clients) are barely functional. Sadly, NFS and SMB are easy to get 80% right, but the 20% corner case interaction takes decades to overcome. My daily storage consulting work exposes me to a myriad of NAS configurations, and few of the multi-platform combinations end well.

Note: Although it has long been known by a variety of names, the Windows NAS protocol is currently called Server Message Block or SMB. Common Internet File System (CIFS) was a failed mid-1990′s attempt by Microsoft to make this protocol standard on the Internet.

Consider the Mac. Apple added an SMB client to OS X in 2001 but, despite many updates, it is far from reliable. Mac users in general loathe connecting to Windows file servers, and business users have located numerous bugs in the handling of Mac-specific file types. It’s bad enough that one company, GroupLogic, created an entire AFP server for Windows just to solve these tricky issues.

This situation often happens in reverse, too. Windows admins are justifiably cautious when deploying non-Windows SMB servers, whether software (Samba, Novell, etc) or system (NetApp, Celerra, BlueArc, etc). As a very early NetApp user, I watched their CIFS/SMB server evolve over a decade and a half into a fairly robust solution, but the early years were downright painful.

Lest you throw rocks at Redmond, know that SMB is not alone with functionality problems. The interoperability of NFS servers and clients is a bit better thanks to open(ish) standards and open source implementations, but its reputation is just as bad. And Apple’s proprietary AFP protocol is downright notorious.

I’ve been there myself many times. I tried to set up a home server based on open source software (Linux, FreeBSD, Samba, Netatalk, etc) but rejected it outright after many frustrating years. Today I use a Mac Mini for file sharing in OS X and serving iTunes music and movies (goodbye, Firefly!) And years of fighting with Samba in enterprise environments taught me two things: It’s possible to get it running well with Windows clients but it’s astonishingly easy to get it wrong.

Enterprise NAS?

We all know that interoperability is devilishly difficult. I don’t envy the NetApp and EMC engineers that have to tweak and tune their server for every possible client, bugs and all. And I am impressed that, after probably millions of man-hours of work, they were able to come up with something stable for a subset of use cases. But this just makes me even more cautious about third-party NAS servers.

I talk to storage vendors all the time, and many of their new products support NFS and SMB. But my internal alarms start going off when I hear about these products. There are two simple reasons for this:

1. As mentioned above, NAS is rare in primary data center applications. It may be common for user files (euphemistically called “unstructured data”) and certain distributed applications (simulation, rendering, etc), but most use cases still call for block SCSI (FC/iSCSI) storage.
2. As further mentioned, getting NAS right takes a massive amount of effort. New and small vendors tend to slap Samba on their (Linux-based) box and call it a day. This is very, very far from being sufficient for enterprise use.

This is why I usually ignore NAS functionality in storage systems except for long-tenured and deep-pocketed vendors. Although the world is turning to “Unified Storage” and multi-protocol support, I’m focusing primarily on block (SCSI) and cloud (REST) capability because the former has proven somewhat easier than NAS to get working and the latter is both simple and “green field” with no legacy concerns.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Why Do I Ignore NAS?
This post was categorized as Apple, Computer History, Enterprise storage, Gestalt IT, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Frankenweenie saves young Victor in Tim Burton's macabre short film

When is a solution integrated and when is it a Frankenstein-like mashup of tangled tech? Apparently, that line is crossed when it’s your competitor’s offering…

In my time in the storage industry, I’ve seen enough franken-storage come and go to make me skeptical whenever a new “integrated” solution is announced. But a lot of this stuff works just fine, so I also know that integrated solutions aren’t always bad!

The latest industry blog flame war centers around NetApp’s recently-announced solid state storage solution, which pairs a V-Series NAS head and a Texas Memory Systems RamSan-500 flash storage system. Perhaps NetApp’s Val Bercovici did get a bit over-excited in his post on the topic, but he wasn’t just talking about the RamSan: He was laying out how NetApp’s WAFL technology can work in an SSD world, and using some recent performance test numbers on that solution as well as their PAM cache cards as an illustration of this.

The next thing you know, we have EMC’s Storagezilla and IBM’s Barry Whyte calling the company out for what they (and others. like Storagebod) see as an underwhelming product offering. That’s all well and good, and I’ll let the reader decide if NetApp’s moves warranted a press release, but now things have gotten uglier…

EMC’s Chuck Hollis called the whole RamSan idea to account, saying it was “Frankenstorage“, causing NetApp’s Alex MacDonald to engage in a little “I know you are but what am I” in reference to EMC’s CLARiiON/Celerra “unified storage” solutions.

It’s time to bring some sanity to this whole integrated solution concept. Every product in the storage world is an amalgamation of OEM parts to one extent or another, and there are always integration issues. Certainly many of EMC’s offerings could be the subject of name-calling: They use STEC SSD drives in the DMX, they use Quantum deduplication engines in their CDLs, and their Celerra NS platform does include a complete Fibre Channel SAN behind the curtain. But they’re not alone, and not even wrong in doing this: Every vendor relies on OEMs, and as a wise man said, “working with an OEM gives you the flexibility to pick best of breed technologies” and that’s exactly what customers want. Any objective person would welcome qualification and integration of TMS’ RamSan with a solid platform like the NetApp V-Series – it’s a certifiable win for the customer. Just like they would be happy to see EMC leveraging great technology from Quantum and STEC.

Chuck goes on to point out some downsides to these OEM combinations, and they’re certainly fair criticisms:

• When you’re buying this from this guy and that from that guy, it’s bound to cost more because everyone needs their cut.
• Since all attempts at unified heterogeneous device management have failed, a combo is certainly harder to manage than a single device.
• With multiple vendors in the mix, fingerpointing is common once support is needed.

But these criticisms can be mitigated by the vendors themselves. They can give up some margin in order to gain market share. They can create unified management interfaces for the combinations they sell and support. And they can really support what they sell, refusing to give in to the temptation to say “not my problem” when the going gets rough. And companies deal with these problems all the time! Frankenstorage doesn’t have to be so scary…

This post can also be found on Gestalt IT: The Difference Between “Integration” and “Frankenstein”

© sfoskett for Stephen Foskett, Pack Rat, 2009. | The Difference Between “Integration” and “Frankenstein”
This post was categorized as Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What was he getting at? Read the comments in Chuck’s post and you’ll understand his implication: Chuck suggests that NetApp “emulates” Fibre Channel in their FAS/OnTap devices on top of the WAFL “file system”, while EMC’s AX storage (behind the NX4) uses “real” Fibre Channel, so it’s better. He goes on to say that EMC is doing a brisk business replacing misfit NetApp FC arrays with real FC kit from EMC. But, as is so often the case, the truth is a little more complex than this: All enterprise storage arrays “emulate” Fibre Channel drives to one extent or another, and using the wrong tool for the job will always lead to trouble.

Is It Real Or Is It Virtual?

Let’s knock one thing out right away: Chuck is right, NetApp is emulating Fibre Channel drives with their FAS arrays. They really do lay out chunks of storage on something kind of like a file system with a bunch of logic mixed in and then pretend that this space is a plain-jane SCSI drive connected over Fibre Channel. And I’ll add to the “scandal” by pointing out that NetApp does exactly the same thing with their iSCSI drives!

Now let’s move on to an even more important point: All modern storage arrays emulate SCSI drives! That’s right, every enterprise storage array is lying, pretending to serve up basic drives but really slicing and dicing them in the background for their own nefarious purposes!

Who is responsible for this deceit? I place the blame on a few: Patterson, Gibson, and Katz started the game with their so-called RAID concept, which kicked things off by allowing a few drives to pretend to be a single one. Data General implemented this with cache in their oh-so-clever HADA, further separating us from The True Disk. But the worst was EMC, with their fully-virtualized Symmetrix array, where there was no definite relationship at all between the LUNs presented to servers and the disks that do all the real work. Some folks would even go so far as to praise this type of post-RAID virtualized storage as innovative!

NetApp takes this “automated lying” to the extreme, forcing their innocent hardware to take honest, well-laid-out blocks of intelligent WAFL space and twist them into vast tracts of dumb pretend-disks. The nerve! Compellent, 3PAR, Dell/EqualLogic, and the rest are just as bad, scattering blocks of data willy-nilly across their disks in so-called “wide stripes“. But don’t let Chuck’s misdirection fool you: EMC is just as guilty with each of their different storage platforms, masquerading as disk drives or file servers and intelligently managing storage underneath! And don’t get me started on the twisted things VMware does to storage!

Modern? Feh! Let’s all hope Apple starts producing their no-feature Xserve RAID again!

Waiting On Angels

So every modern array emulates disks. What was Chuck’s point again? Oh yeah, that the AX Fibre Channel storage used by EMC’s NX4 is superior to the integrated Fibre Channel capability of the NetApp FAS2020! I’m sure he’s right for some use cases and wrong for others. FC on the FAS2020 is a perfect match for some, and the NX4/AX wins in a landslide in some circumstances.

The crux of the argument is the fact that NetApp does all sorts of stuff behind the scenes build and support an FC LUN that the EMC AX FC array doesn’t do. So, although it wouldn’t be fair to say that one was “emulated” and another was not, Chuck would be correct in saying that an FC LUN on an AX is more “real” than one on a NetApp FAS. But arguing over technicalities like this is all angels and pins and doesn’t matter in the real world!

What does matter? In block storage, latency is king. Generally speaking, more cogs and wheels leads to more latency. This is why storage arrays rely so much on large, intelligent caches and vendors are experimenting with all sorts of cool caching technology. But, ignoring cache, high-end arrays generally have worse latency than low-end ones because they have all sorts of translation and virtualization going on in the background. In any I/O situation, increased latency hurts throughput and the perception of performance. And there comes a point when block applications give up waiting and it’s “game over, man!”

I remember migrating from an old CLARiiON 3100 to a brand new Symmetrix 3930 and watching the Symmetrix choke on the incoming data stream. It just couldn’t write fast enough to handle full streaming reads from the (old-tech) CLARiiON. But once everything was migrated and running, the Symmetrix, with its massive (for the time) 16 GB of cache, widely-spaced data layout, and multiple internal channels, completely destroyed the CLARiiON in real-world performance. This pattern continues today, with devices like the DMX and USP offering much better real-world performance than benchmarks or theoretical techno-arguments would suggest.

So Which Is Better?

But Chuck and the rest were not talking about high-end stuff here. They are comparing the architecture of entry-level enterprise kit and drawing conclusions about which is best. I personally don’t care what the internals of the system look like. I care how well it works.

I have personally seen Microsoft Exchange running on low-end FC-connected NetApp FAS arrays, and it worked great. I also helped a customer migrate off of EMC AX that didn’t give them the performance they needed for their databases. In truth, lower-end gear is often over-sold and unable to deliver the performance, features, and reliability specified on data sheets and in vendor presentations.

That’s right, there’s more to this picture than raw performance. Consider manageability, for one. NetApp is offering a single-interface integrated system with all protocols (CIFS, NFS, iSCSI, and FC) available from one device. They also offer similar levels of integration for their (really nice) snapshot, replication, and deduplication technology. WAFL is busy doing a lot of great stuff, so I really wouldn’t be surprised if EMC’s less-integrated NX/AX offering beats them on performance at the same price point. Which is more important to you, integration, performance, or features? And I bet that, if you spent a bit more on a higher-end NetApp box, you could have it all.

On the flip side, EMC is offering a really compelling entry-enterprise combination at a nice price point. The latest NX should be on everyone’s NAS short list, and I’m sure the simple FC of the AX array would work well in a smallish Exchange, VMware, or SQL Server environment. It’s not as unified as NetApp’s offering management- or feature-wise, but it’s still pretty good.

Pick the right tool for the job, though. Neither the NX4 nor the FAS2020 is a good fit for a high-I/O application, and that’s a fact!

This post can also be found on Gestalt IT: Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

On the contrary, I suggest that the enterprise storage world we know was shaped by singular innovative products of the past. Without these, the IT world might look very different.

While you’re at it, check out my list of the coolest enterprise storage flops!

So let’s take a walk through history, identifying the ten most innovative and important enterprise storage hardware products. But let me note first that this list could be 100 items long, and we all have our favorites. Lots of the storage blogging world contributed their ideas, too!  So I’m setting some arbitrary rules to keep myself on track:

1. The products have to be reflected in the shape of modern enterprise storage for open systems. If their day came and went, they’re not listed here.
2. Hardware only – I’m working on another list for software!
3. Listed products have to have been successful in the market (or they’d probably fail rule number 1 anyway), so I’ll put together a list of cool flops later.
4. For product families, I tried to pick the most influential and innovative 2member.
5. This is an enterprise storage list – items have to be used in big companies, not little PCs.
6. Only one product per company (sorry, IBM and EMC!)

So without further ado, on with the list!

1964 IBM 2311

Let’s get one thing straight: IBM is responsible for much of modern enterprise computing, so it’s no surprise that when it comes to storage, they developed just about everything, including disk drives, floppies, tapes, arrays, and libraries. So why pick this particular piece of kit? Prior to 1964, computer components were developed as a set (the 350 storage system went with the 305 RAMAC, for example), but the 2311 changed that. It was a generic storage device, plug-compatible with a range of computers. If you claimed that the 2311 was the common ancestor of all modern enterprise storage, I would certainly agree with you!

Other notable IBM products worth mentioning:

• The original 1956 RAMAC 350 disk drive unit
• The 1961 301, which used a separate arm and head for each platter (like every disk drive produced since)
• 9-track tape, which dominated from 1964 until the 1/2-inch tape revolution in the 1990s
• The 1970 3330, which added error correction and remained in production for 13 years
• The 1971 introduction of the 23FD floppy disk drive
• The 1973 3430, whose “30/30″ code name caused people to refer to hard drives as “Winchesters”
• 1974′s 3830 “MSS” tape library
• The 1980 3380, which introduced film-head technology
• 2003′s SAN Volume Controller, the first successful SAN virtualization product (after many others failed!)

1976 Shugart SA-400 minifloppy

Alan Shugart’s floppy drive was a massive hit in the home computing market, but why include it in a list of enterprise storage technologies? Because its storage interface set the standard for plug- and protocol-compatible storage in the nascent microcomputer world.

Designing a general product for use in a multitude of systems was truly innovative, and many later computers were literally designed around both the concept and physical form factor of Shugart’s drive. Simply put, this fat floppy drive inspired computer designers to create computers that could accept standard peripherals, the very definition of open systems. The definition of “peripheral” would soon grow to include standard I/O devices like storage, terminals, printers, communications gear, and everything else we know today.

1980 Seagate ST-506

Shugart left his company in 1979, founding Seagate Technology. That company’s first product was the ST-506, a 5 MB hard disk drive that shared its physical shape with the SA-400 and used a similar interface.

Like the floppy, Shugart’s hard drive set the standard for microcomputers, eventually finding its way into enterprise systems, and catapulting Seagate to its current position of disk drive leadership. Higher-capacity derivatives of the ST-506 were fitted with interfaces using Larry Boucher of Adaptec’s SCSI protocol, which continues in use even today. Although the ST-506 wasn’t a “smart” drive, the ecosystem that developed around it was critical.

1984 DEC TK50

Digital Equipment’s introduction of the MicroVAX II in 1985 was accompanied by a new half-inch backup tape drive and cartridge called the TK50. Open-reel tapes had dominated non-disk storage before this, but cartridges quickly replaced reels, leading to the development of tape robotics and (more) reliable off-site storage.

If you picked up a TK50 tape today, you could be forgiven for mistaking it for a modern SDLT or LTO, because these use a similar cartridge and drive form factor, the same linear tape technology, and the same half-inch tape size. The TK50, DLT, and SDLT were the mainstays of open backup for decades.

1987 Auspex

This terrible photo is an actual Auspex press image!

What was the first open systems enterprise storage array? Leveraging Sun’s NFS protocol, and hiding a Sun workstation inside, Wizard of Oz style, the Auspex set the standard for everything we think of as “enterprise” in the open systems world. The company’s storage systems were truly ahead of their time, with ranks of redundant disks years before RAID became common.

Auspex was founded by Larry Boucher, father of SCSI and founder of Adaptec, and raked in sales while others struggled to figure out how to sell in the enterprise.  But their refusal to produce a smaller device would be their undoing. Today’s monolithic arrays owe as much to Auspex as they do to IBM, but the companies producing them could learn a lesson from the company’s demise.

1987 StorageTek 9310 PowderHorn

StorageTeks versatile and scalable 9310 PowderHorn defined backup

StorageTek was instrumental in many enterprise storage developments (see McData, for example), but one product literally transformed the datacenter: the 9310 “PowderHorn”. Consider the “glass house” datacenter tours that the largest companies would use to impress visitors: They would show off their mainframe, their Cray, or their PowderHorn.

This versatile system would accommodate every major tape cartridge format and system type and could scale to truly massive proportions. When disks were expressed in megabytes, PowderHorns held terabytes. 

1995 EMC Symmetrix 3000

EMC's Symmetrix 3000 family brought mainframe block storage technology to open systems

The 1990 introduction of the Symmetrix was the key turning point for the (now) giant of Hopkinton, but the first two generations were mainframe-only. In 1994, the company delivered data replication capability in the form of SRDF, moving key enterprise functionality to the storage array.

But it was the 1995 introduction of the third-generation Symmetrix 3000 that really changed the storage world. For the first time, open systems could connect to mainframe-class storage over the standard SCSI protocol and leverage features like SRDF and (in 1997′s Symm 4) TimeFinder.

One key ingredient often overlooked in the Symmetrix was in-box virtualization the likes of which hadn’t been seen before. It also featured RAID-like sub-disk data protection, though the Symmetrix line never did implement true RAID.

The Symmetrix was redesigned entirely in 2003 to become the DMX. Although it was as different from its predecessor as the New Beetle was from Volkswagen’s original, the DMX line continued many of the philosophical underpinnings set in 1990.

EMC also deserves credit for their original Celerra enterprise NAS system, which picked up where Auspex left off. The company followed these in 2002 with the Centera CAS system, which abandoned many traditional concepts of enterprise storage, like blocks- and filesystem-access and modular and monolithic architecture. The Centera is certainly innovative, but we have yet to see the impact it will have.

1994 Data General CLARiiON

Data General paired the CLARiiON (left) with their AViiON server

It’s easy to forget the world before RAID. But Data General was one of the first to market with a cached RAID system with their 1991 introduction of the HADA. Although this early system was tied to DG’s servers, it donated much of its architecture to a system that became more valuable than the rest of the company: CLARiiON.

Massively successful, and mighty impressive (PC Magazine called the first CLARiiON “amazing”), this modular block storage array set the template for over a decade. There were dozens of copycat arrays on the market within a few years of the introduction of the HADA. The CLARiiON gained Fibre Channel support, was sold to EMC, and remains a mainstay of corporate data centers, albeit with updates to every component. I have administered every generation of CLARiiON array, and can attest to their capability (when properly configured!)

1996 NetApp Multiprotocol Filer

NetApp's F330 Multiprotocol Filer was a huge hit in mid-size businesses and a major upgrade

NetApp (nee Network Appliance) was formed by a group of ex-Auspex engineers who wanted to create a more modular NAS server based on industry-standard hardware. They released their first product, the NFS-only FASServer, in 1995, but it was their August, 1996 introduction of the Windows-compatible Multiprotocol Filer software that really put them on the map.

Combined with their solid F220, F330, and F540 hardware, NetApp now had a serious challenger to Auspex, and their NAS systems blew away dedicated server-based solutions in terms of flexibility and manageability. NetApp’s unified NFS, CIFS/SMB, and HTTP access to content on their unique WAFL file system was impressive at the time, as was the quick setup and administration and the ease of adding drives to their RAID-4 sets. Plus, they brought a new level of friendliness to the data center with their bright colors and silly “toaster” nomenclature.

1999 McData Fibre Channel Director

McData has an interesting history, beginning with Storage Technology (StorageTek), intersecting with IBM, bought and spun off by EMC, and lately acquired by Brocade. Although the company spent a decade developing various peripherals for IBM mainframe systems, their late-1994 introduction of a switching director for ESCON traffic would change the storage world. We take large-scale SANs and LANs for granted today, but McData’s director was astonishing when it was introduced: There were no words to describe it or its function at the time, and period press articles are puzzling!

EMC scooped the company up a year later, but McData was spun out in 1997, adding Fibre Channel support that same year. In the ensuing years, McData became the enterprise SAN “arms dealer”, supplying IBM and EMC with the ED-5000 and ED-6064 directors. I recall commenting at the time that Fibre Channel connectivity built around the McData director was the first Storage Area Network worthy of the name. McData went IPO in 2000 and was purchased by rival Brocade in 2006. McData’s director architecture survives and thrives today against fierce competition from Brocade, Cisco, and others.

Others To Consider

What would you have included in this list? Here are some honorable mentions that I wish I had room for:

1. StorageTek “Aegis” L700 library
2. Brocade “LOOM” 2xxx switches
3. Adaptec SCSI HBAs
4. Copan MAID
5. EqualLogic iSCSI arrays
6. EMC Centera
7. HP EVA
8. HDS USP
9. Compellent Storage Center

Responses

Here’s a list of responses that others have posted:

• Marc Farley’s Top 10 List
• Alex MacDonald’s Bottom 10 List

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Top Ten Innovative Enterprise Storage Hardware Products
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.