VMware's PSA is awash in abbreviations and options

I am often questioned during my Storage for Virtual Environments seminar presentations about VMware’s Pluggable Storage Architecture (PSA). This system is fairly straightforward and concept: VMware provides native multipathing support for a variety of storage arrays, and allows third parties to substitute their own plug-ins at various points in the stack. But the profusion of acronyms and third-party options makes it difficult for end-users to figure out what is going on. In an effort to help, I present here another entry in my “VMware storage features in plain English” series.

Note: I am more of a storage guy than a virtualization expert. I consider myself one of those end-users who have had trouble figuring out what’s going on with PSA specifically, in VMware storage features in general. I welcome comments and suggestions for corrections or improvements to this and all of my articles. Thanks for your help!

Introducing Pluggable Storage Architecture (PSA)

Pluggable storage architecture was one of the major enhancements introduced in vSphere 4. Functionally similar to Microsoft’s MPIO stack for Windows, PSA includes native multipathing support and allows vendors to plug in their own advanced features.

I find the VMware diagram confusing. Is mine more or less accurate and readable?

The ESX kernel (VMkernel) walks down through three layers when communicating with storage:

1. In the top layer, VMware native NMP or third-party MPP software decides which SATP to use, or whether to use the native interface. MASK_PATH also operates at this layer.
2. The SATP layer includes native generic path selection (active/active, active/passive), standard ALUA, as well as allowing third-party plugins (SATP) to override its behavior. The SATP monitors these paths, reports changes, and initiates fail-over on the array as needed.
3. At the PSP layer, software decides which physical channel to use for I/O requests.

There are three types of PSA plugins for vSphere 4:

1. Storage Array Type Plug-In (SATP)
2. Path Selection Plug-in (PSP)
3. A complete third-party multipathing software stack (MPP)

As is the case with VAAI, VMware includes a number of third-party plug-ins in the ESXi install. Users can simply activate many of these according to their needs, though some require additional fees and licensing.

Storage Array Type Plug-in (SATP) List

Storage Array Type Plug-Ins (SATPs) to the VMware Pluggable Storage Architecture multipathing solution for the specific characteristics of the storage array. This is very important, since each storage array design differs substantially in detail and support, especially when it comes to load-balancing and failover between controllers, ports, and paths. So it is critical for VMware to have developed a standard interface to communicate with arrays.

SATPs allow load balancing across multiple paths, intelligent path selection, and over troubled conditions such as “chatter”, when passed rapidly fail back and forth between controllers.

The SATP has critical tasks to perform in the PSA stack:

1. Decide which method of communication to use with the storage (PSA or native)
2. Monitor the health of the physical I/O channels or paths
3. Report any changes in the state of the paths up the stack
4. Perform actions required to fail over storage between controllers on the array

VMware vSphere includes a variety of generic plugins for storage arrays. I’ve identified the following:

• VMW_SATP_LOCAL – Local SATP for direct-attached devices
• VMW_SATP_DEFAULT_AA – Generic for active/active arrays
• VMW_SATP_DEFAULT_AP – Generic for active/passive arrays
• VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arrays

Although I have sometimes seen other SATP plug-ins mentioned, the following plug-ins are all that are listed in the VMware ESX Hardware Compatibility List.

• VMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGI
• VMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)
• VMW_SATP_CX – EMC/Dell CLARiiON  and Celerra (also VMW_SATP_ALUA_CX)
• VMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, Invista
• VMW_SATP_INV – EMC Invista and VPLEX
• VMW_SATP_EQL – Dell EqualLogic systems

EMC PowerPath and HDS HDLM also support a variety of storage arrays, but I would classify these as full MPP replacements for PSA, rather than SATP plug-ins.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp satp list

Path selection plug-in (PSP) List

In contrast to the diversity of VAAI and SATP plug-ins, the universe of path selection plug-ins is fairly small. Most storage arrays are supported with either Most Recently Used (MRU) or Fixed path selection approaches. Many also support Round Robin (RR) path selection. The only vendor with a specific PSP that is not also part of a full MPP (like EMC PowerPath or HDS HDLM) is Dell, which offers a special routed path selection plug-in for the EqualLogic iSCSI arrays.

• VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arrays
• VMW_PSP_FIXED – Fixed – Supports hundreds of storage arrays
• VMW_PSP_RR – Round-Robin – Supports dozens of storage arrays
• DELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arrays

As mentioned, EMC PowerPath also offers path selection as a plug-in in addition to the full MPP stack. Many other vendors offer unique path selection plug-ins, over 100 in total, but these are not specifically called out in the VMware HCL apart from their existence. I would love to learn more about them, however.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp psp list

Tell Me More About PSA!

As mentioned above, I am by no means an expert in VMware Pluggable Storage Architecture. Rather, I am interested in learning more and passing on this knowledge to others. I welcome your comments and feedback, and especially your corrections to the information presented here. I will try to keep this page updated as new versions of vSphere are introduced and as I learn more about this technology. Thank you for your help and understanding!

Further Reading:

• What’s New in VMware vSphere 4: Storage
• Next-Generation ESX Storage: A Pluggable Core Storage Architecture
• Storage Changes in the VMware vSphere 4 Family
• Pluggable Storage Architecture, exploring the next version of ESX/vCenter
• Vmware vSphere 4.1 PSA (Pluggable Storage Architecture) Understanding
• Explain the Pluggable Storage Architecture (PSA) Layout

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware PSP and SATP in Plain English
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Note: This information was based on the “VMware Storage/SAN Compatibility Guide” and is being regularly updated. Please add comments here and I will add products and change and update listings as soon as they appear in the guide!

Where, Why, and What is VAAI?

I’ve previously discussed the fact that VMware’s excellent ESX hardware compatibility list (HCL) is so comprehensive than obscures basic facts about supported products. This is especially true for VAAI, since compatibility is only noted as a footnote in individual storage array listings. It does not help matters that not all VAAI plugins support all three capabilities.

Like my previous posts regarding FCoE CNA’s, SATA and PATA chipsets, and home/lab network cards, I’ll attempt to boil down the VMware ESX HCL into plain English. This data will also be part of my Storage for Virtual Servers seminar presentation, the first of which will be given on March 10 in Philadelphia.

I’m not going to spend too much time on “what is VAAI” in this post. Instead, I suggest you read the following blog posts and VMware’s excellent guide, “What’s New in VMware vSphereTM 4.1 — Storage“?

• What is VAAI, and how does it add spice to my life as a VMware admin?
• vStorage APIs for Array Integration aka VAAI
• If You Ever Needed Convincing About VAAI…
• VMware VAAI pros and cons and the hidden fourth primitive

The Three VAAI Primitives

You ought to read the updated Complete List of VMware VAAI Primitives since it’s much more thorough and informative!

In ESX 4.1, that vStorage API for Array Integration includes three basic capabilities or primitives:

1. Blocks zeroing is a communication method for thin provisioning capable storage arrays, allowing them to quickly zero out storage capacity for later reclamation.
2. Full copy commands the storage array to make a mirror or snapshot of data without any I/O on the server hardware.
3. Hardware assisted locking enables more granular control of shared storage resources in ESX clusters

In order to support VAAI, a storage array requires two things:

1. Hardware capable of supporting one or more of the three primitives listed above
2. A software plug in for ESX enabling communication and integration

VAAI Plug In Support

Creating a VAAI plug in is not a trivial task, and not all storage arrays are yet supported. I have heard grumbling from storage vendors that EMC (the storage vendor that owns VMware) has been given early access to VAAI information, allowing them to support this feature set before their competitors. However, this has not stopped a diverse set of other unrelated storage vendors from quickly producing and releasing effective and complete VAAI plugins.

As of this writing, there are 11 array-specific plugins and one general-purpose plug in available for ESX 4.1. EMC, NetApp, 3PAR (HP), HDS, FalconStor, Fujitsu, IBM, Dell (EqualLogic), and HP (LeftHand, P9000, P2000) have produced VAAI plugins supporting all three primitives. Additionally, a cloud in supporting the T10 blocks zeroing methods is available, enabling other arrays to support this one primitive. Note that the T10 primitive should support nearly any capable array, but not all have been tested and qualified for use with it.

VAAI Support Matrix

Note that similar OEM versions (for example, Fujitsu’s FibreCAT CLARiiONs, and the Gateway/Lenovo/Acer AMS line) are also supported the same as the manufacturer’s offerings. I’ve simplified and eliminated similar models (the Dell EqualLogic PS6000E, PS6000S, PS6000X, PS6000XV, and PS6000XVS are all listed simply as PS6000).

Updates:

• IBM recently added full VAAI for the XIV, SVC, and similar Storwize V7000. I’m sure they’re also working on complete VAAI plugins for the big DS8000 systems!
• EMC certified the new VNX line for VAAI (FC only for now) and the V-Max just gained iSCSI VAAI support.
• FalconStor added VAAI for NSS, enabling any storage array to be used.
• HP created VAAI plugins for the P9500 and MSA P2000 lines as well.

Stephen’s Stance

VAAI is an exciting new capability for VMware ESX, and demonstrates the enterprise readiness of vSphere 4.1. Although not all storage arrays are yet supported, the diverse assortment listed above should cover the majority of enterprise storage environments. I fully expect that the obvious holes will be filled in soon, and I look forward to updating this list when I hear news of those product releases. I also look forward to learning of additional capabilities added as VAAI primitives in the future!

The Exhaustive List

I am attempting to keep this list up to date. My authoritative source of information is the VMware Storage Compatibility Guide. This is the only source of information I will use, since only official and supported implementations belong in production. But I welcome pointers, suggestions, and referrals for updates!

This list is complete as of February 21, 2011

Full VAAI (All 3 primitives)

• Dell
  • EqualLogic
    • iSCSI (vmw_vaaip_eql)
      • Dell EqualLogic PS4000E (Dell EqualLogic PS4000X; Dell EqualLogic PS4000XV)
      • Dell EqualLogic PS5000E (Dell EqualLogic PS5000X; Dell EqualLogic PS5000XV)
      • Dell EqualLogic PS5500E
      • Dell EqualLogic PS6000E (Dell EqualLogic PS6000S; Dell EqualLogic PS6000V; Dell EqualLogic PS6000XV; Dell EqualLogic PS6000XVS)
      • Dell EqualLogic PS6010E (Dell EqualLogic PS6010S; Dell EqualLogic PS6010X; Dell EqualLogic PS6010XV; Dell EqualLogic PS6010XVS)
      • Dell EqualLogic PS6500E (Dell EqualLogic PS6500X)
      • Dell EqualLogic PS6510E (Dell EqualLogic PS6510X)
      • EqualLogic PS100E
      • EqualLogic PS200E
      • EqualLogic PS300E
      • EqualLogic PS3600X
      • EqualLogic PS3700X
      • EqualLogic PS3800XV
        • EqualLogic PS3900XV
      • EqualLogic PS400E
      • EqualLogic PS50E
      • EqualLogic PS70E
• EMC
  • FC (vmw_vaaip_cx)
    • EMC CLARiiON CX4-120 (EMC CLARiiON CX4-120 C8/C8X; Dell/EMC CLARiiON CX4-120; Fujitsu FibreCAT CX4-120)
    • EMC CLARiiON CX4-240 C8/C8X
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8/C8X; Dell/EMC CLARiiON CX4-480)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8/C8X; Dell/EMC CLARiiON CX4-960)
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC VNX5100
    • EMC VNX5300
    • EMC VNX5500
    • EMC VNX5700
    • EMC VNX7500
  • FC/iSCSI/FCoE (VMW_VAAI_SYMM)
    • EMC Symmetrix VMAX
  • iSCSI (vmw_vaaip_cx)
    • EMC Celerra CNS
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC CLARiiON CX4-120 C8
    • EMC CLARiiON CX4-240 C8
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8)
• FalconStor
• FC (unknown)
• CDP Gateway
• CDP SA
• CDP VS
• CDPx Gateway
• IPStor Enterprise
• NSS Gateway
• NSS SA
• NSS VS
• Fujitsu
  • FC (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
  • iSCSI (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
• Hitachi
  • FC (vmw_vaaip_hds)
    • Hitachi AMS 2100 (Acer AMS2100; Gateway AMS2100; HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (Acer AMS2300; Gateway AMS2300; HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Acer AS2040; Gateway GS2040
    • Hitachi AMS 2010
    • Hitachi BR1600/BR1600E/BR1600S
    • Hitachi Virtual Storage Platform (Hitachi VP9500; HP StorageWorks P9500)
    • Nihon-Unisys Sanarena 1910
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
  • iSCSI (vmw_vaaip_hds)
    • Hitachi AMS 2010
    • Hitachi AMS 2100 (HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Hitachi BR1600E
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
    • Hitachi Virtual Storage Platform
• HP
  • 3PAR
    • FC (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
    • iSCSI (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
  • P9500
  • FC (hp-vaaip-p9000)
  • HP P9500
  • P2000
  • FC/iSCSI (hp-vaaip-p2000)
  • HP MSA P2000
  • LeftHand
    • iSCSI (vmw_vaaip_lhn)
      • HP LeftHand P4500
      • HP LeftHand P4000 VSA
      • HP LeftHand P4000sb
      • HP LeftHand P4300 (HP LeftHand P4300 G2)
      • HP LeftHand P4500 (HP LeftHand P4500 G2)
      • HP LeftHand P4800
      • HP ProLiant DL380
      • Dell 2950
      • IBM x3650
      • LeftHand NSM 160
      • LeftHand NSM 185
      • LeftHand NSM 2060 (LeftHand NSM 2060 G2)
      • LeftHand NSM 2120 (LeftHand NSM 2120 G2)
      • LeftHand NSM 260
      • LeftHand NSM 320
      • LeftHand NSM 326
      • LeftHand NSM 3650
      • LeftHand NSM 380
      • LeftHand NSM 4150
      • LeftHand VSA
• IBM
  • FC/iSCSI (IBM_VAAIP_MODULE)
    • IBM XIV
    • IBM SVC
    • IBM V7000
    • Fujitsu VS850
    • Actifio
• NetApp
  • FC (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • FCoE (VMW_VAAIP_NETAPP)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • iSCSI (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series (Fujitsu Eternus NR1000F Series Model F2040)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series (Fujitsu Eternus NR1000F Series Model F3160)
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series

T10 Block Zero Only

• Bull
  • FC (vmw_vaaip_t10)
    • Bull Optima2000
  • iSCSI (vmw_vaaip_t10)
    • Bull Optima2000c
    • Bull Optima2000i
• NEC
  • FC (vmw_vaaip_t10)
    • NEC iStorage D3-30
    • NEC iStorage D4-30
  • iSCSI (vmw_vaaip_t10)
    • NEC iStorage D3-30/D3-30i
    • NEC iStorage D4-30/D4-30i
• Fujitsu
  • iSCSI (vmw_vaaip_t10)
    • Fujitsu Eternus VS850
• IBM
  • iSCSI (vmw_vaaip_t10)
    • IBM Storwize V7000
    • IBM SVC

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware VAAI Storage Array Support in Plain English
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My work
  • The Four Stages of Vendor Blogging
  • Always Punch Above Your Weight
  • Introducing The Enterprise IT Infrastructure Events Calendar
  • What Does HP Printer “Ink Cartridge Expired” Mean?
  • HP Photosmart Ink System Failure – Error: 0xc19a0035
• Other good stuff
  • Nigel Poulton did a great job covering EMC’s latest Symmetrix updates in VMAX Comes of Age
  • Chris Wolf is always worth reading; witness this week’s Hybrid Cloud Mobility: Converting the VM is the Easy Part
  • I also loved Tom Loverro’s piece, Android “Vista Pie”

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Back From the Pile: Interesting Links, December 24, 2010
This post was categorized as Enterprise storage, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Four Horsemen of Storage System Performance: These four ugly gentlemen stand between you and your data.

Why do some data storage solutions perform better than others? What tradeoffs are made for economy and how do they affect the system as a whole? These questions can be puzzling, but there are core truths that are difficult to avoid. Mechanical disk drives can only move a certain amount of data. RAM caching can improve performance, but only until it runs out. I/O channels can be overwhelmed with data. And above all, a system must be smart to maximize the potential of these components. These are the four horsemen of storage system performance, and they cannot be denied.

Overcoming the Limits of Spindles

Perhaps the previous discussion of spindles left you exhausted, imagining a spindly-legged centipede of a storage system, trying and failing to run on stilts. The Rule of Spindles would be the end of the story were it not for the second horseman: Cache. He stands in front of the spindles, quickly dispatching requests using solid state memory rather than spinning disks. Cache also acts as a buffer, allowing writes to queue up without forcing the requesters to wait in line.

Cache may be quick, but practical concerns limit its effectiveness. Solid state memory is available in many types, but all are far more expensive per gigabyte than magnetic hard disk media. DRAM has historically cost 400 times as much as disk capacity, and even NAND flash (the current darling of the industry) is more than 40 times as expensive. Practically speaking, this means that disk devices, from the drives themselves to large enterprise storage arrays, usually include a very small amount of cache relative to their total capacity.

When specifying a storage system, the mathematics of cache and spindles adhere to a simple rule: More is better for performance but worse for the budget. This leads to a trade-off, where a point of diminishing return tells us to stop adding both spindles and cache and accepting the storage system as it is.

A History of Cache

Cache was not always as common as it is today. When even a small amount of DRAM cost hundreds of dollars, adding a single RAM chip to a hard disk drive would have broken the bank. So many drives had no cache at all well into the mid 1990′s. Operating systems of the time used expensive system memory as a buffer for storage operations rather than expecting cache in the disk controller or drive – remember setting the Buffers command in config.sys?

This was not as bad as it seems, at least in theory. Operating systems stand a fighting chance of “knowing” what data will be requested next, and could therefore request it ahead of time. They also might get a hint about data that will never be used again and can thus flush that from the so-called buffer cache. Although MS-DOS wasn’t very good at this, modern systems have greatly advanced in this respect using a technology called demand paging.

Caching at the array was the key differentiator for early enterprise RAID systems, overcoming the punishing slowdowns caused by parity calculations when data was written. EMC adapted their DRAM-based solid-state storage systems to become a cache in front of hard disk drives and the Symmetrix was born. The Data General (now EMC) CLARiiON was notable as well, bringing a large intelligent write cache to the vast market of midrange systems that could never justify the high price of a Symmetrix. Today, all vendors, from IBM to HP to NetApp to HDS, have vast and clever caches.

The importance of cache on enterprise storage performance can not be over-stated. Mix together rotational latency, seek time, and RAID penalty and you get seriously-compromised I/O response time. But cache can eliminate this penalty entirely, provided there is capacity, by confirming the write and queueing it for later (a concept known as write-back caching). Busy shared storage systems would be simply unusable without cache.

Five Uses for Disk Buffers

Hard disk drives today normally contain a small amount of RAM to use as a buffer for I/O requests. This serves the following needs, though not all are found on all drives:

1. A read cache, allowing frequently-requested data to be read from memory rather than involving mechanical disk operations
2. An I/O-matching mechanism, allowing slower disks and faster interfaces to work together
3. A read-around (ahead or behind) pre-fetch cache, saving a few blocks around any requested read on the assumption that they will also be requested soon
4. A read-after-write cache, saving recently-written data to serve later read requests
5. A command queue, allowing write commands to be reordered, avoiding the “elevator seeking” common to early hard disk drives

Disk buffer size has expanded rapidly in recent years, with some devices including 64 MB or more or DRAM. Seagate’s Momentus XT drive even includes 4 GB of NAND flash as a massive read cache!

Write-Through and Write-Back Cache

There are two basic methods of caching data:
The earliest systems used read-only or write-through caches. All I/O requests pass through the cache, which usually saves the most recent and serves them up when a read is requested. They don’t buffer write requests at all, simply passing them through to the storage system to process. They are safe, since the storage device always has a consistent set of committed writes, but they do nothing to offset the RAID penalty.	Most modern storage systems use a write-back (also called “write-behind”) cache, which acknowledges writes before they are committed to disk. They use non-volatile RAM, battery-backed DRAM, or NAND flash to ensure that data is not lost in the event of a power outage. Though far more effective, this type of memory is also far more costly.

Just about every modern storage array uses caching, and most employ the write-back method to accelerate writes as well as reads. Some have very smart controllers that perform other tricks, but Smart is another Horseman for another day. As mentioned before, RAID systems would be nearly unusable without write-back cache allowing the disks to catch up with random writes.

Onward: I/O, and Smarts

The horseman of spindles is harsh, but he does not rule the day. There are many ways to overcome his limits and his three brothers often come into play. These are cache, which bypasses the spindle altogether; I/O, which can constrain even the fastest combination of disk and cache; and the intelligence of the whole system, which limits or accelerates all the rest. We will examine these horsemen in the future!

I’ve been meaning to write this up for a long time. Thanks for listening and commenting!

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | The Four Horsemen of Storage System Performance: Never Enough Cache
This post was categorized as Computer History, Enterprise storage, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What is Multipath? And Why?

The good old days: One device, one path

In the beginning, computers connected to peripherals and other computers through a single bus or channel and life was easy. Although one might mistake the names of the dominant printer connection (parallel) for some kind of multipath system when compared to the modem connection (serial), this was not the case. Only the bits traveled in parallel – the logical connection was a simple single path.

Early-90's servers might confuse admins with two SCSI connections to a single device

Then things got complicated. The SCSI protocol allowed for multiple devices in a chain, and even for two different “initiators” (computers or controllers) to interact with these “targets”. Some folks even dual-attached devices to a single computer with multiple controllers.

Why would one device and one computer need more than one connection? It boils down to two factors:

1. Performance – I/O channels have typically been slower than the computer could handle, so multiple channels might be used to increase the amount of data that can flow in and out.
2. Reliability – If one connection failed, the other might still be usable, reducing the risk of an outage.

Late-90's enterprise systems might have four or more paths to a single storage array

Pretty soon, enterprise computer architecture had gotten incredibly complex. I remember connecting a massive HP V-class server to an EMC Symmetrix with eight separate Fibre Channel cables. Each disk “LUN” showed up twice, and we had hundreds of them. We managed all of these virtual storage paths using HP’s PVLinks dynamic multipathing software. We used Veritas DMP and EMC PowerPath to do pretty much the same thing on Solaris and other UNIX systems.

Active/Passive to Active/Active

The earliest path management software provided two incredibly important functions: It figured out which of the SCSI targets it saw were actually different names for the same one, and it allowed the operating system to choose one and fail over to the other in case of an interruption. These were Active/Passive links – no matter how many paths were presented (and Fibre Channel switches sometimes presented eight or more), only one was active at any one time.

Modern systems have abstracted and virtual I/O channels, making path management much more important

But the EMC Symmetrix and similar high-end storage systems changed all this. Symmetrix storage was fully virtualized – the presentation of LUNs to servers was entirely disconnected from the actual disks and RAID sets in the array. This meant the Symmetrix could handle I/O requests across different paths and controllers for the same LUN. EMC and the rest responded with Active/Active path management software, allowing I/O to travel in parallel for the first time.

How is Dual Active Different?

Not everything called Active/Active is created equal. In fact, many supposed Active/Active setups really shouldn’t be called that since they don’t use both paths for all data. Instead, I like to call these Dual Active – both paths are active but with different data.

Consider the differences between the following two solutions:

A true active/active setup uses all paths for all data all the time

A "dual active" setup uses both paths, but each target is directed to one or the other

See the difference? Although the paths are active in both cases, they are not the same. Both approaches have merit, and neither is inherently superior, but they should have different names applied. Even active/passive has its place, since simplicity is often a virtue.

Dual Active Outside Storage

These same concepts apply outside the field of storage and I/O. Many server clustering systems use the same terminology, right down to the misapplication of “active/active” when “dual active” is more appropriate. It’s easy to miss the significance of this difference, but it can make more of an impact in clustering since CPU workloads are harder to balance.

Let me know what you think. If there is interest, I might dive into path management strategies like round robin!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Multipath: Active/Passive, Dual Active, and Active/Active
This post was categorized as Computer History, Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

EMC mentioned this schedule on their public earnings call, repeated it in discussions with Enterprise Storage Forum, and corporate PR confirmed it to me this morning. FAST is, if you will excuse the pun, fast-approaching!

There’s actually a very-significant difference between FAST versions 1 and 2:

• FAST version 1 (coming in December 2009) can automatically migrate an entire LUN between tiers of storage on the Symmetrix V-Max based on access patterns. This is a nice-to-have feature, but does not yet live up to the promise of automated tiered storage, as pioneered by Compellent and offered in various forms by many other storage companies. I am told that FAST version 1 also does not support virtually-provisioned (thin) volumes. Although FAST will make it much easier to take advantage of solid state flash drives (EFDs to EMC), it will do nothing to improve utilization.
• FAST version 2 (coming in May mid-2010?) will automatically move smaller sub-LUN pieces of storage between tiers of storage, and will reportedly support virtual provisioning. It is likely that FAST would use the same extent size supported by virtual provisioning, which is 12 tracks or 768 KB. The combination of virtual provisioning and fully-automated storage tiering will finally EMC the ammunition they need to defend their turf against smaller challengers when it comes to these sticky technical features.

So there you have it. EMC V-Max gets FAST in December 2009 and May mid-2010! Now how about a schedule for geo-distributed clustering?

Update: Fixed two typos above. I wrote “May 2009″ instead of “May 2010″. You should see my checkbook!

Note: Although Lou Przystas told Enterprise Storage Forum that FAST V2 would come in May, EMC has apparently not set a hard date and are saying “mid-2010″ at this point. Even this is an aggressive ship date, but I won’t say EMC missed it unless we haven’t got it by the end of summer!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | EMC V-Max FAST: Coming in December … And 2010!
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Apple

• MacBook Users: Turn off This Bluetooth Default Setting Now – Now I know what turned on my MacBook Pro in the bag: My BlueTooth mouse!

Enterprise Computing

• Windows Storage Server-Based Systems Step Into 2008 – My Gestalt IT coverage of the features of WSS08.
• EMC Symmetrix V-Max: When Does It Get FAST and Virtual? – Another Gestalt IT piece, pointing out that the V-Max isn’t going to be fully realized for a very long time
• Microsoft Unveils Final Windows Storage Server – Paul Shread runs with the “Windows Storage Server 2008″ theme, quoting yours truly extensively.
• What is a best practice and Are Best Practices Just Shared Opinions? – W. Curtis Preston and I pickup an old conversation about the definition of “best practice”.
• Infosmack Episode 2 – EMC, David Donatelli and Non-Compete Agreements – Tony Asaro and I join Marc Farley and Greg Knieriemen on the Storage Monkeys podcast!
• The US Federal Government defines Cloud Computing - Ruven Cohen covers the US Federal Government’s cloud computing meetings and gives us a sneak peek at the new definition of cloud computing from the world’s number one IT user.
• Support and Q&A for Solid-State Drives – Lots of great info on how Windows 7 (and probably Server 2008 R2) deals with solid state drives.
• VMware Slaps Enterprise and Cisco In Face, Opens Door For Competitors – More on the VMware vSphere 4 licensing changes, and what it might mean.

Misc

• Non-competes are evil – Robin Harris lays it out. I happen to agree.
• Five Technologies Our Kids Won’t Even Recognize – Wired’s Gadget Lab makes us all feel old!
• Chrysler HQ Designed To Convert Into Shopping Mall - Wait Wait Don’t Tell Me says it’s true, so I believe it!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Back From the Pile: Interesting Content From the Week of May 9, 2009
This post was categorized as Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The podcast is now available on iTunes!

Although the topic is an open roundtable, I thought I might give my readers a peek into the areas I plan to bring up:

• Why storage is so critical to VMware admins: It’s all about the IOPS!
• EMC’s new Symmetrix V-Max and why it’s important to VMware shops
• The great FCoE/CNA debate and Cisco’s UCS home run
• Super low-end storage for the small business or lab

Of course, what will actually get discussed is anyone’s guess, but it’s good to go in with a list of topics to avoid dead air!

So dial in or listen online, and let the discussion start!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Interested in VMware and Storage? Tune In to the VMware Communities Podcast!
This post was categorized as Enterprise storage, Gestalt IT, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This news is so fresh you would leave footprints if you step in it, but here’s a collection of my favorite links and coverage so far. I’ll be updating this list throughout the next few days:

• Let’s start with independents:
  • My first post on the topic can be found over at Gestalt IT: EMC Symmetrix V-Max Is Neither Monolithic Nor Midrange
  • Chris Evans was first but promises more than Enterprise Computing: EMC Announced Next Generation V-Max Architecture
  • Storagebod supposes that there will be No DMX5
  • Duncan Epping is watching from a server virtualization perspective: EMC announced the Symmetrix V-Max!
  • Jerome Wendt: EMC Symmetrix V-Max – It’s About Time
• And the press:
  • Beth Pariseau is always smart, as her SearchStorage piece shows: EMC clusters Symmetrix high-end disk arrays
  • And Beth on StorageSoup: EMC launches Symmetrix V-Max, may add spin-down
  • Chris Mellor is dead on in his piece for The Register, Symmetrix and the death of monolithic arrays
  • Lucas Mearian of Computerworld focused his piece on the private cloud: EMC introduces x86-based Symmetrix array for cloud storage
  • Dave Raffo at StorageSoup: EMC V-Max: V stands for bigger
  • Paul Travis at Byte and Switch: EMC Clusters New Storage Systems for Virtual Data Centers
  • And Dave Simpson brings up the rear on his personal blog, but still no word from InfoStor: EMC V-Max: Locked (in) and loaded
• EMC’s bloggers are first out of the gate:
  • Barry Burke has typically prolific and detailed coverage: 
  • • 1.055: symmetrix v-max – a revolutionary evolution
    • 1.056: inside the virtual matrix architecture (my favorite)
    • 1.057: symmetrix v-max – scale up, scale out, scale away!
    • 1.058: v-max does what hi-star can’t?
    • 1.059: fully automated storage tiering (fast)
  • Chuck Hollis’ Symmetrix V-Max: Storage Architecture Redefined is a solid overview
  • Chad Sakac is always good, and his EMC’s VMware Storage Strategy – The 3rd Shoe Drops is a must-read on V-Max, though I’m still left wondering about EMC’s VMware strategy!
  • Mark Twomey’s there with three posts and counting:
    • V-Max: The Scale Out Symmetrix
    • VMax: Auto-provisioning Groups
    • V-Max: SRDF Evolves
  • Dick Sullivan promises You Ain’t Seen Nothing Yet
  • Steve Todd:
    • V-Max Quality
    • Managing V-Max at Scale
  • Len Devanna comments on the Social Launch
• The opposition is moving fast, too!
  • Marc Farley preempted the announcement with a hilarious must-watch video, EMC insider interview is a fake, and an expose on What’s a tigon anyway?
  • Val Bercovici from NetApp observes that the world might have moved past the need for a Symmetrix, noting “Frames are so 90’s”
  • Hu Yoshida focuses on what the V-Max is not: A storage virtualization platform. Don’t Confuse Symmetrix V-Max with Storage Virtualization
  • Claus Mikkelsen of HDS was also predictably underwhelmed, asking why EMC introduced this roadmap now? EMC – Catching Up With the Past
  • Tony Asaro blogs for HDS about 10 Questions on V-Max

That’s the news that’s fit to print so far! Stick with this post and Gestalt IT’s continuing coverage of the V-Max!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Tracking EMC’s Symmetrix V-Max Launch
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What was he getting at? Read the comments in Chuck’s post and you’ll understand his implication: Chuck suggests that NetApp “emulates” Fibre Channel in their FAS/OnTap devices on top of the WAFL “file system”, while EMC’s AX storage (behind the NX4) uses “real” Fibre Channel, so it’s better. He goes on to say that EMC is doing a brisk business replacing misfit NetApp FC arrays with real FC kit from EMC. But, as is so often the case, the truth is a little more complex than this: All enterprise storage arrays “emulate” Fibre Channel drives to one extent or another, and using the wrong tool for the job will always lead to trouble.

Is It Real Or Is It Virtual?

Let’s knock one thing out right away: Chuck is right, NetApp is emulating Fibre Channel drives with their FAS arrays. They really do lay out chunks of storage on something kind of like a file system with a bunch of logic mixed in and then pretend that this space is a plain-jane SCSI drive connected over Fibre Channel. And I’ll add to the “scandal” by pointing out that NetApp does exactly the same thing with their iSCSI drives!

Now let’s move on to an even more important point: All modern storage arrays emulate SCSI drives! That’s right, every enterprise storage array is lying, pretending to serve up basic drives but really slicing and dicing them in the background for their own nefarious purposes!

Who is responsible for this deceit? I place the blame on a few: Patterson, Gibson, and Katz started the game with their so-called RAID concept, which kicked things off by allowing a few drives to pretend to be a single one. Data General implemented this with cache in their oh-so-clever HADA, further separating us from The True Disk. But the worst was EMC, with their fully-virtualized Symmetrix array, where there was no definite relationship at all between the LUNs presented to servers and the disks that do all the real work. Some folks would even go so far as to praise this type of post-RAID virtualized storage as innovative!

NetApp takes this “automated lying” to the extreme, forcing their innocent hardware to take honest, well-laid-out blocks of intelligent WAFL space and twist them into vast tracts of dumb pretend-disks. The nerve! Compellent, 3PAR, Dell/EqualLogic, and the rest are just as bad, scattering blocks of data willy-nilly across their disks in so-called “wide stripes“. But don’t let Chuck’s misdirection fool you: EMC is just as guilty with each of their different storage platforms, masquerading as disk drives or file servers and intelligently managing storage underneath! And don’t get me started on the twisted things VMware does to storage!

Modern? Feh! Let’s all hope Apple starts producing their no-feature Xserve RAID again!

Waiting On Angels

So every modern array emulates disks. What was Chuck’s point again? Oh yeah, that the AX Fibre Channel storage used by EMC’s NX4 is superior to the integrated Fibre Channel capability of the NetApp FAS2020! I’m sure he’s right for some use cases and wrong for others. FC on the FAS2020 is a perfect match for some, and the NX4/AX wins in a landslide in some circumstances.

The crux of the argument is the fact that NetApp does all sorts of stuff behind the scenes build and support an FC LUN that the EMC AX FC array doesn’t do. So, although it wouldn’t be fair to say that one was “emulated” and another was not, Chuck would be correct in saying that an FC LUN on an AX is more “real” than one on a NetApp FAS. But arguing over technicalities like this is all angels and pins and doesn’t matter in the real world!

What does matter? In block storage, latency is king. Generally speaking, more cogs and wheels leads to more latency. This is why storage arrays rely so much on large, intelligent caches and vendors are experimenting with all sorts of cool caching technology. But, ignoring cache, high-end arrays generally have worse latency than low-end ones because they have all sorts of translation and virtualization going on in the background. In any I/O situation, increased latency hurts throughput and the perception of performance. And there comes a point when block applications give up waiting and it’s “game over, man!”

I remember migrating from an old CLARiiON 3100 to a brand new Symmetrix 3930 and watching the Symmetrix choke on the incoming data stream. It just couldn’t write fast enough to handle full streaming reads from the (old-tech) CLARiiON. But once everything was migrated and running, the Symmetrix, with its massive (for the time) 16 GB of cache, widely-spaced data layout, and multiple internal channels, completely destroyed the CLARiiON in real-world performance. This pattern continues today, with devices like the DMX and USP offering much better real-world performance than benchmarks or theoretical techno-arguments would suggest.

So Which Is Better?

But Chuck and the rest were not talking about high-end stuff here. They are comparing the architecture of entry-level enterprise kit and drawing conclusions about which is best. I personally don’t care what the internals of the system look like. I care how well it works.

I have personally seen Microsoft Exchange running on low-end FC-connected NetApp FAS arrays, and it worked great. I also helped a customer migrate off of EMC AX that didn’t give them the performance they needed for their databases. In truth, lower-end gear is often over-sold and unable to deliver the performance, features, and reliability specified on data sheets and in vendor presentations.

That’s right, there’s more to this picture than raw performance. Consider manageability, for one. NetApp is offering a single-interface integrated system with all protocols (CIFS, NFS, iSCSI, and FC) available from one device. They also offer similar levels of integration for their (really nice) snapshot, replication, and deduplication technology. WAFL is busy doing a lot of great stuff, so I really wouldn’t be surprised if EMC’s less-integrated NX/AX offering beats them on performance at the same price point. Which is more important to you, integration, performance, or features? And I bet that, if you spent a bit more on a higher-end NetApp box, you could have it all.

On the flip side, EMC is offering a really compelling entry-enterprise combination at a nice price point. The latest NX should be on everyone’s NAS short list, and I’m sure the simple FC of the AX array would work well in a smallish Exchange, VMware, or SQL Server environment. It’s not as unified as NetApp’s offering management- or feature-wise, but it’s still pretty good.

Pick the right tool for the job, though. Neither the NX4 nor the FAS2020 is a good fit for a high-I/O application, and that’s a fact!

This post can also be found on Gestalt IT: Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.