It isn't always easy to get where you need to go!

Consider the following situation: You go to lunch with your good friends, John and Mary. Halfway through a rousing discussion of the latest Hollywood movie, Mary starts talking about the fantastic action sequences while John criticizes the romantic angle. You realize something mine-bending has happened: John now has Mary’s personality, and vice versa. It’s like they have switched brains or something!

This truly weird situation isn’t likely to happen in person, but occurs all the time in the data center. Virtualization of server, network, and storage services eliminates the fixed link between physical resources and functional applications. A running virtual machine can instantly move from one server, network adapter, HBA, or LUN to another. And when it happens, traditional components have no idea how to react.

The Challenges of Mobility

Mobility is perhaps the “killer app” of virtualization, but it is also the killer of traditional IT systems. Let’s consider the challenges of this “Twilight Zone” moment.

• The operating system expects a consistent hardware environment, which is exactly what the hypervisor creates
• The LAN must be prepared to redirect all network traffic instantly and seamlessly to one or more new physical interfaces
• The SAN similarly must be able to reroute all I/O to a new pair of HBA’s without missing a beat
• The storage array must be able to re-present capacity to a new physical device, and must maintain snapshots and other configurations
• The backup system must also be able to maintain consistency over time even as machines relocate to different server and storage locations

All of this must be done while maintaining quality of service (QoS), access control, reporting, and appropriate segmentation at all levels. This is an incredibly challenging task, and no conventional protocol (IP, Ethernet, NFS, SCSI, Fibre Channel, etc.) is anymore ready then you are when your good friends switch personalities.

Two Paths

So much of the development that is currently taking place in IT focuses on accommodating this “mobility issue”. Two key approaches have emerged to take on this challenge:

• “In a vacuum” technologies (like VXLAN) assume that no other changes will be made, so the focus is on maintaining complete compatibility in front and behind
• “Clean sheet” technologies (usually from startups) take a different approach, throwing out compatibility in favor of technical elegance

Both of these approaches have merit. Attempting to maintain compatibility only works so far (just ask a Windows API programmer), but it leverages the existing environment and recognizes that most people are not ready for wholesale change. Clean sheet designs always make more sense, but they rarely attain mass acceptance. Nearly every technology we rely on today is full of bolt-ons in the name of compatibility. Some, like Ethernet and x86, actually work pretty well, too.

The Stack of Lies

The difference between virtualization and cloud computing is exactly this same distinction. Hypervisors, NPV, NAT, thin provisioning, and so many other virtualization technologies exist mainly to maintain compatibility in a vacuum. In contrast, true cloud computing dispenses with the entire stack and creates a new platform for applications.

This is, perhaps, the reason that cloud computing is not taken off in the enterprise. Simply put, IT is not prepared to ditch everything they have ever used even in the face of a demonstrably superior alternative. Currently, the highest use of cloud is behind gateways and virtualization engines that bring it back down to earth.

This brings us to the stack of lies called server virtualization. Any “modern” virtualized data center is built on lie after lie, with each level telling the other what it wants to hear. The volume manager lies to the operating system; the hypervisor lies to the volume manager; and the storage array lies to the hypervisor. The same sad state of affairs allows networking and even memory to function in a virtual world.

But these shaky stacks of lies have difficulty adapting to motion, since no level truly “knows” the reality of the world around. The depressing truth is that a bowl of spaghetti like VXLAN is perhaps the highest form of art we can expect in a virtual data center.

Stephen’s Stance

As a techie, I am always drawn to clean sheet designs that offer technical elegance along with functionality. But I know that, realistically, products that assume nothing about the world around them and bend over backward to maintain compatibility are more likely to succeed. Still, I maintain hope that the issues of virtual machine mobility will be solved in an elegant way, rather than adding to the “stack of lies”.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Terrifying True Story Of Virtual Machine Mobility
This post was categorized as Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

When is a gigabyte not a gigabyte? When you're not buying gigabytes!

I stepped into a hornet nest this week when I posted a write-up about a new flash storage array from Pure Storage. The controversy had nothing to do with the underlying technology, which seems quite sound. Rather, it was all about pricing, with Pure’s competitors calling foul on their price comparisons.

You’re Not Buying Gigabytes

In a world of 3 TB drives, storage capacity is almost irrelevant. Capacity is what people think they are getting when they buy enterprise storage devices, but capacity is only one aspect of the purchase, and it’s not a very significant one in most cases.

So what are enterprise storage buyers buying?

• Performance, especially I/O operations (IOPS), is much more critical than capacity in most cases, and it takes lots of spindles or expensive flash chips to deliver it.
• Data protection features like snapshots are increasingly important, and often cost extra.
• Compatibility is paramount, as is long-term supportability from all vendors involved.
• Integration and management features are often the deciding factor in purchases, especially when it comes to server virtualization applications.
• High availability and product support are “must-haves” that can multiply the cost of a solution.
• Power, cooling, and floor space can be very important for some applications and entirely inconsequential in others.
• Capacity is sometimes important, but many applications require just a few TB or less and thin provisioning, data deduplication, and compression are really blurring the lines here.

So although a typical customer will say “I need 200 GB for this application” they likely need nothing of the sort. They really need 100 IOPS, snapshots, a line on the HCL, VAAI and vCenter plugins, and redundant everything. Even the capacity number is questionable: Most applications grow over time, and few need much capacity really.

Since you can’t buy a 1 GB storage array and can’t fill a 10 TB device to 100%, pricing per GB is complete nonsense. Plain old storage space just sort of tags along for the ride once you build a system capable of meeting all these other needs.

Data Reduction or Pricing Obfuscation?

Utilization of storage capacity has always been terrible, but improving capacity efficiency is worthless. The best you can do is over-tax your array or put all your data “eggs” in too few drive “baskets”. Achieving impressive capacity utilization just concentrates I/O, and this is the last thing you want to do with spinning hard disk drives.

This is why I suggest redirecting the conversation away from capacity requirements. The amount of GB to be used and the efficiency of that storage doesn’t matter all that much except for certain massive and rare applications. Once the array is big enough to handle the data, everything else is a wash.

This is also why I’m skeptical of data reduction technologies. Most applications would be better off optimizing for performance not reducing capacity used. And data reduction techniques like compression and deduplication quickly lead down the “your mileage may vary” rat hole.

Comparing Apples to Apples

Also read Grapples and Tangelos: Why it’s Impossible to Compare Fairly

There is only one way to do a real fair comparison between different storage devices: Specify all the requirements and let each vendor put forward whatever they have that meets all of them. Who really cares if vendor A’s disk-based solution is 10% utilized while vendor B’s flash array needs 1/5 the capacity? As long as you have a place to put it (and enough power to feed it) it’ll still work fine.

One serious challenge in enterprise storage is the rise of flash memory as a storage medium. Flash chips are expensive on a data capacity basis but amazingly cheap in terms of performance and environmental efficiency. Put another way, an SSD can’t storage as much data as a hard disk, but it delivers massive I/O capability in a tiny, rugged, low-power footprint.

Since most enterprise applications need only a few hundred GB of capacity, a few SSDs can be a compelling alternative to a “refrigerator” full of disks. It can be hard to convince the boss, but you really can fit a whole datacenter’s worth of storage I/O into a few rack units!

Pure and Nimbus

This is the issue facing flashy solid state devices from many companies, and the root of my headaches this week. Pure Storage hasn’t finalized pricing yet, but are claiming that their new device costs $5 per usable gigabyte. This is incredibly cheap for an array that can blow the doors off most enterprise gear!

Nimbus Data, on the other hand, sells their all-flash enterprise storage array for about $10 per GB. But this is not the end of the story, and Pure might even be more expensive than Nimbus! Or maybe not. It all depends on what you’re comparing.

Pure claims that their cost is half the price of most comparable flash storage array competitors, but this is where the questions start to appear. Is that $5 gigabyte usable or raw? Does it include high availability? And can I really store any old gigabyte of data there or is that a compressed/deduplicated gigabyte?

It turns out that the real cost of Pure Storage capacity is $20 per GB including RAID and an extra mirrored array for high availability. But since every byte written to the array is thin provisioned, deduplicated, and compressed, many customers will pay much less for actual data stored. And since it’s an all-SSD array, it’ll perform way better than a disk-based system, too.

Muddying the Waters

So why not just call it $5 per GB and be done with it? It’s confusing, that’s why, and your mileage will vary widely.  Pure’s own slides show some applications getting 4:1 data reduction and others all the way up to 17:1. So these applications would be paying as low as $1.18 per GB or as high as $5.

But you can’t buy just 1 GB of storage from Pure. Their smallest array (which includes one controller and one shelf of SSDs) provides 5.5 TB of raw capacity, presumably using 24 256 GB SSDs. A high-availability configuration would include two controllers and two shelves of SSDs for 11 TB of raw storage. That’s going to cost almost a quarter of a million dollars according to my calculator. That’s one expensive gigabyte!

Of course no one would buy this array to store just a thousand megabytes. They would buy it to support a bunch of applications that need capacity and performance and efficiency and integration and everything else. And they can buy a mirrored pair of arrays from Pure Storage or Nimbus or Violin Memory or Texas Memory Systems or others at a variety of price points.

The only way to really compare these products is to spec them out on equal footing and see what the price tag looks like. These comparisons would include data reduction, but they would also have to bring in high availability and every other requirement of the applications they will support.

Stephen’s Stance

It’s way too difficult for me to do the pricing math for these systems, so I’m throwing in the towel. I’m thrilled to see all-flash arrays made available to IT buyers. This wouldn’t be possible without clever use of thin provisioning and data reduction, as well as smart software to overcome the limits of SSD.

I’m going to guess that Pure and Nimbus will cost about the same for similar configurations, though I’ll bet each believes they’re cheaper. Rather than get in the middle, I invite each company to post a comment below stating their case. I’ll even embed their responses into a future blog post on the subject if they get too long. Just don’t ask me to be the referee.

Update: Pure Storage responds with an outline of their pricing:

• How Pure Storage Delivers All-Flash Storage at Below the Price of Spinning Disk

Image credit: Rotten Apple by Wappas

© sfoskett for Stephen Foskett, Pack Rat, 2011. | When Pricing Gets Squishy Competition Heats Up
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

IBM today announced VAAI support for SVC, and promised it for the DS8000

VMware’s vStorage API for Array Integration (VAAI) is one of the most-important storage technology advances of the decade, allowing the VMware ESX to integrate and coordinate operations with supported enterprise storage arrays. After an explosion of initial support, however, VAAI implementation slowed. IBM was notably absent from the party, listing only partial support for a single product line for the better part of the year. But they’ve turned on the VAAI heat, releasing full support for the XIV and SVC and promising DS8000 in the near future.

VAAIing for Attention

For the complete updated VMware VAAI support matrix, see my post, VMware VAAI Storage Array Support in Plain English

Under vSphere 4.1, VAAI includes three “primitives”, or supported features:

1. Full copy enables the hypervisor to direct the storage array to copy data without sending it all the way “up the stack”
2. Block zeroing supports enhanced thin provisioning communication
3. Hardware-assisted locking allows the hypervisor to more-efficiently share a SCSI LUN

IBM’s original VAAI support amounted only to the basic T10 zeroing support using the generic “vmw_vaaip_t10″ plugin on the SVC and related Storwize V7000. This fell well short of the specialized three-primitive plugins offered by rivals like EMC and HDS.

IBM VAAI Support is Spreading

Far from being asleep at the wheel, IBM was busy creating a complete VAAI implementation that would be portable across a variety of storage platforms. XIV was the first beneficiary of this work, gaining full VAAI support and a listing in the VMware ESX HCL. This came in version 10.2.4 of the XIV software, released and approved in April of 2011. But insiders hinted something more was on the way.

Today, IBM announced that their “IBM_VAAIP_MODULE” plugin would support the SAN Volume Controller (SVC) and related Storwize V7000 as well. The array-side functionality will be added in version 6.2 of the software, which IBM promises to deliver in June of this year.

Now, at the IBM Storage Summit, I hear that this same code will enable the DS8000 to support all three VAAI primitives shortly. Once this is delivered, IBM’s key platforms (XIV, SVC, and DS8k) will all support VAAI, putting IBM on par with their competitors.

Stephen’s Stance

I am pleased to see IBM planning full VAAI support, since I feel this is absolutely key to future storage developments. I am further impressed that they developed a cross-platform VAAI codebase that could be deployed across such diverse architectures as XIV, SVC, and DS8k. Once this is released, IBM will finally be up to speed with their key competition.

I won’t recommend anything that’s not on the official VMware hardware compatibility list (HCL), and I’m glad to see that V7000 and XIV is now listed. It will take months DS8k to appear, though. By that time, vSphere 5 (with its rumored VAAI enhancements) will be here. Hopefully IBM will be out day-and-date with support.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | IBM Adds VAAI Support to XIV and SVC
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing
  • Processing and Scheduling Thin Provisioning is another entry in my State of the Art Thin Provisioning series
  • I posted another napkin-tastic infographic: Real-World Port Throughput Relative To Light Peak
  • Wednesday saw the Thunderbolt leak, Will Apple Call Light Peak “Thunderbolt”?, but USB 3.0 For Mac Is Here (sort of)
  • Then there was the real reveal on Thursday: Thunderbolt = Light Peak = Mini DisplayPort + PCI Express
  • I had to dispel some silly speculation: Why the iPad 2 Won’t Include Thunderbolt
  • I didn’t write the post, but I did write the DCIG 2011 Small Enterprise Storage Array Buyer’s Guide!
• Other stuff
  • Good stuff from Nasuni: Blocks vs. Files: Which Approach is Better for Cloud Gateways?
  • IPv6 was a topic of discussion at Tech Field Day, so this one from Eric Siebert was interesting: IPv6 support in vSphere
  • Terry Slattery also has some IPv6 ideas: Are You Ready for IPv6?
  • Another great TFD post from Jeff Fry: IP Address Management (IPAM) and INFOBLOX
  • A response to my VAAI questions from IBM: XIV VAAI Support – whats the story?
  • I’ve been wondering about this. What’s your opinion on Gabrie van Zanten’s post? To USB or not to USB, how do you boot your ESXi host?
  • And now for something completely different: Pants Down on Madison: Salinger’s Slightly Salingeresque Boswell Wants to Be Left Alone

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, February 25, 2011
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although the core issues with thin provisioning revolve around communication, it presents unique challenges to the storage array as well. We talked about granularity of pages, and the comments for that piece were extremely enlightening. Now let’s consider another key factor: Scheduling.

Note that the “provisioning” part is relatively easy to do on the fly: An array just has to allocate additional capacity as writes come in, which is something it does anyway. It’s the thin reclamation that poses a challenge, since this involves zero detection across a whole page of data in many cases.

Just like de-duplication, thin provisioning challenges the resources of the storage array to do background number crunching. And just like dedupe, the array engineers have a choice of when to do the reclamation processing: Well after writing or “in-line”. The extreme ends of this spectrum fall into two equally disappointing categories: Wholly ineffective or ridiculously intensive.

Let’s start with the “intensive” side: You could have the controller do thin provisioning automatically; that’s kind of what IBM does with SVC, for example, and 3PAR claims to do this too. The trouble is that the controller has to literally watch everything, and it’s got to reassemble whole pages, perhaps 42 MB or even one GB in cache. If it didn’t have all that data, it would have to go fetch it, put it into cache, look at it, make sure it was all zeros, then get rid of it. It’s really, really difficult to do automatic, in-line, thin provisioning. It’s a good thing to do, but it’s a hard thing to do.

So most vendors schedule thinning for later. In the “10 terabytes of zeros” example, they’re actually going to write 10 terabytes to disk, or at least through to cache. Then, at some point in the future, they’ll go back and reclaim that space. Some are pretty aggressive and reclaim capacity very frequently. Others are fairly lazy: The Drobo seems to reclaim only once or twice a day. A lot of people who have them are surprised when the thing springs to life and starts going, “Bada-bada-bada-bada-bada-bada.” Apparently it’s reclaiming storage at that time.

Some thin provisioning systems are even manually-initiated, and this is really pretty ineffective. The storage administrator has better things to do than reclaim storage all the time, so they are probably going to set a cron job to do it regularly at a specified time. If the system only does it on demand, that means that it doesn’t have the horsepower to do it automatically. Ergo, it’s sometimes going to conflict with “real work” and cause a problem.

I would look for a system that was fairly aggressive with thin reclamation. I was talking to the guys at Nimbus Data, for example, and they claim to do thin provisioning in-line all the time. I hope that we see more storage arrays that are doing that, and less that are doing it manually, on demand, because that’s just not as useful.

But considering that thin provisioning used to be almost useless, the fact that it’s now at least somewhat useful is gratifying.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Processing and Scheduling Thin Provisioning
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Note: This information was based on the “VMware Storage/SAN Compatibility Guide” and is being regularly updated. Please add comments here and I will add products and change and update listings as soon as they appear in the guide!

Where, Why, and What is VAAI?

I’ve previously discussed the fact that VMware’s excellent ESX hardware compatibility list (HCL) is so comprehensive than obscures basic facts about supported products. This is especially true for VAAI, since compatibility is only noted as a footnote in individual storage array listings. It does not help matters that not all VAAI plugins support all three capabilities.

Like my previous posts regarding FCoE CNA’s, SATA and PATA chipsets, and home/lab network cards, I’ll attempt to boil down the VMware ESX HCL into plain English. This data will also be part of my Storage for Virtual Servers seminar presentation, the first of which will be given on March 10 in Philadelphia.

I’m not going to spend too much time on “what is VAAI” in this post. Instead, I suggest you read the following blog posts and VMware’s excellent guide, “What’s New in VMware vSphereTM 4.1 — Storage“?

• What is VAAI, and how does it add spice to my life as a VMware admin?
• vStorage APIs for Array Integration aka VAAI
• If You Ever Needed Convincing About VAAI…
• VMware VAAI pros and cons and the hidden fourth primitive

The Three VAAI Primitives

You ought to read the updated Complete List of VMware VAAI Primitives since it’s much more thorough and informative!

In ESX 4.1, that vStorage API for Array Integration includes three basic capabilities or primitives:

1. Blocks zeroing is a communication method for thin provisioning capable storage arrays, allowing them to quickly zero out storage capacity for later reclamation.
2. Full copy commands the storage array to make a mirror or snapshot of data without any I/O on the server hardware.
3. Hardware assisted locking enables more granular control of shared storage resources in ESX clusters

In order to support VAAI, a storage array requires two things:

1. Hardware capable of supporting one or more of the three primitives listed above
2. A software plug in for ESX enabling communication and integration

VAAI Plug In Support

Creating a VAAI plug in is not a trivial task, and not all storage arrays are yet supported. I have heard grumbling from storage vendors that EMC (the storage vendor that owns VMware) has been given early access to VAAI information, allowing them to support this feature set before their competitors. However, this has not stopped a diverse set of other unrelated storage vendors from quickly producing and releasing effective and complete VAAI plugins.

As of this writing, there are 11 array-specific plugins and one general-purpose plug in available for ESX 4.1. EMC, NetApp, 3PAR (HP), HDS, FalconStor, Fujitsu, IBM, Dell (EqualLogic), and HP (LeftHand, P9000, P2000) have produced VAAI plugins supporting all three primitives. Additionally, a cloud in supporting the T10 blocks zeroing methods is available, enabling other arrays to support this one primitive. Note that the T10 primitive should support nearly any capable array, but not all have been tested and qualified for use with it.

VAAI Support Matrix

Note that similar OEM versions (for example, Fujitsu’s FibreCAT CLARiiONs, and the Gateway/Lenovo/Acer AMS line) are also supported the same as the manufacturer’s offerings. I’ve simplified and eliminated similar models (the Dell EqualLogic PS6000E, PS6000S, PS6000X, PS6000XV, and PS6000XVS are all listed simply as PS6000).

Updates:

• IBM recently added full VAAI for the XIV, SVC, and similar Storwize V7000. I’m sure they’re also working on complete VAAI plugins for the big DS8000 systems!
• EMC certified the new VNX line for VAAI (FC only for now) and the V-Max just gained iSCSI VAAI support.
• FalconStor added VAAI for NSS, enabling any storage array to be used.
• HP created VAAI plugins for the P9500 and MSA P2000 lines as well.

Stephen’s Stance

VAAI is an exciting new capability for VMware ESX, and demonstrates the enterprise readiness of vSphere 4.1. Although not all storage arrays are yet supported, the diverse assortment listed above should cover the majority of enterprise storage environments. I fully expect that the obvious holes will be filled in soon, and I look forward to updating this list when I hear news of those product releases. I also look forward to learning of additional capabilities added as VAAI primitives in the future!

The Exhaustive List

I am attempting to keep this list up to date. My authoritative source of information is the VMware Storage Compatibility Guide. This is the only source of information I will use, since only official and supported implementations belong in production. But I welcome pointers, suggestions, and referrals for updates!

This list is complete as of February 21, 2011

Full VAAI (All 3 primitives)

• Dell
  • EqualLogic
    • iSCSI (vmw_vaaip_eql)
      • Dell EqualLogic PS4000E (Dell EqualLogic PS4000X; Dell EqualLogic PS4000XV)
      • Dell EqualLogic PS5000E (Dell EqualLogic PS5000X; Dell EqualLogic PS5000XV)
      • Dell EqualLogic PS5500E
      • Dell EqualLogic PS6000E (Dell EqualLogic PS6000S; Dell EqualLogic PS6000V; Dell EqualLogic PS6000XV; Dell EqualLogic PS6000XVS)
      • Dell EqualLogic PS6010E (Dell EqualLogic PS6010S; Dell EqualLogic PS6010X; Dell EqualLogic PS6010XV; Dell EqualLogic PS6010XVS)
      • Dell EqualLogic PS6500E (Dell EqualLogic PS6500X)
      • Dell EqualLogic PS6510E (Dell EqualLogic PS6510X)
      • EqualLogic PS100E
      • EqualLogic PS200E
      • EqualLogic PS300E
      • EqualLogic PS3600X
      • EqualLogic PS3700X
      • EqualLogic PS3800XV
        • EqualLogic PS3900XV
      • EqualLogic PS400E
      • EqualLogic PS50E
      • EqualLogic PS70E
• EMC
  • FC (vmw_vaaip_cx)
    • EMC CLARiiON CX4-120 (EMC CLARiiON CX4-120 C8/C8X; Dell/EMC CLARiiON CX4-120; Fujitsu FibreCAT CX4-120)
    • EMC CLARiiON CX4-240 C8/C8X
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8/C8X; Dell/EMC CLARiiON CX4-480)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8/C8X; Dell/EMC CLARiiON CX4-960)
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC VNX5100
    • EMC VNX5300
    • EMC VNX5500
    • EMC VNX5700
    • EMC VNX7500
  • FC/iSCSI/FCoE (VMW_VAAI_SYMM)
    • EMC Symmetrix VMAX
  • iSCSI (vmw_vaaip_cx)
    • EMC Celerra CNS
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC CLARiiON CX4-120 C8
    • EMC CLARiiON CX4-240 C8
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8)
• FalconStor
• FC (unknown)
• CDP Gateway
• CDP SA
• CDP VS
• CDPx Gateway
• IPStor Enterprise
• NSS Gateway
• NSS SA
• NSS VS
• Fujitsu
  • FC (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
  • iSCSI (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
• Hitachi
  • FC (vmw_vaaip_hds)
    • Hitachi AMS 2100 (Acer AMS2100; Gateway AMS2100; HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (Acer AMS2300; Gateway AMS2300; HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Acer AS2040; Gateway GS2040
    • Hitachi AMS 2010
    • Hitachi BR1600/BR1600E/BR1600S
    • Hitachi Virtual Storage Platform (Hitachi VP9500; HP StorageWorks P9500)
    • Nihon-Unisys Sanarena 1910
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
  • iSCSI (vmw_vaaip_hds)
    • Hitachi AMS 2010
    • Hitachi AMS 2100 (HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Hitachi BR1600E
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
    • Hitachi Virtual Storage Platform
• HP
  • 3PAR
    • FC (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
    • iSCSI (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
  • P9500
  • FC (hp-vaaip-p9000)
  • HP P9500
  • P2000
  • FC/iSCSI (hp-vaaip-p2000)
  • HP MSA P2000
  • LeftHand
    • iSCSI (vmw_vaaip_lhn)
      • HP LeftHand P4500
      • HP LeftHand P4000 VSA
      • HP LeftHand P4000sb
      • HP LeftHand P4300 (HP LeftHand P4300 G2)
      • HP LeftHand P4500 (HP LeftHand P4500 G2)
      • HP LeftHand P4800
      • HP ProLiant DL380
      • Dell 2950
      • IBM x3650
      • LeftHand NSM 160
      • LeftHand NSM 185
      • LeftHand NSM 2060 (LeftHand NSM 2060 G2)
      • LeftHand NSM 2120 (LeftHand NSM 2120 G2)
      • LeftHand NSM 260
      • LeftHand NSM 320
      • LeftHand NSM 326
      • LeftHand NSM 3650
      • LeftHand NSM 380
      • LeftHand NSM 4150
      • LeftHand VSA
• IBM
  • FC/iSCSI (IBM_VAAIP_MODULE)
    • IBM XIV
    • IBM SVC
    • IBM V7000
    • Fujitsu VS850
    • Actifio
• NetApp
  • FC (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • FCoE (VMW_VAAIP_NETAPP)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • iSCSI (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series (Fujitsu Eternus NR1000F Series Model F2040)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series (Fujitsu Eternus NR1000F Series Model F3160)
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series

T10 Block Zero Only

• Bull
  • FC (vmw_vaaip_t10)
    • Bull Optima2000
  • iSCSI (vmw_vaaip_t10)
    • Bull Optima2000c
    • Bull Optima2000i
• NEC
  • FC (vmw_vaaip_t10)
    • NEC iStorage D3-30
    • NEC iStorage D4-30
  • iSCSI (vmw_vaaip_t10)
    • NEC iStorage D3-30/D3-30i
    • NEC iStorage D4-30/D4-30i
• Fujitsu
  • iSCSI (vmw_vaaip_t10)
    • Fujitsu Eternus VS850
• IBM
  • iSCSI (vmw_vaaip_t10)
    • IBM Storwize V7000
    • IBM SVC

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware VAAI Storage Array Support in Plain English
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although I consider it the main stumbling block for thin provisioning, communication (or lack thereof) is being addressed with metadata monitoring, WRITE_SAME, the Veritas Thin API, and other ideas. But communication isn’t the only issue.

Let’s talk about page sizes. You’ll often see vendors tossing this “softball” objection at their competitors, claiming that their (smaller) page size makes for more-effective thin provisioning. And that’s true, to a some extent, but perhaps not the end of the story.

Look at the top block in this stack. The light background box is the page, and the colored boxes represent data. If your storage is written in “pages” of this size, you can’t thin it.

What if we used a smaller page? What if my page is a quarter of that size, as in the second row? I still can’t thin it out, because my data is spread all over the place.

Remember worrying about fragmentation back in the days of DOS and Windows and FAT filesystems? It’s kind of like this.

Because we’re using zero page reclaim, the whole page has to be zero to be reclaimed. If your data is all over the place, if there’s even one bit that’s not zero on a page, we’re not going to reclaim that whole page.

Now let’s return to our illustration. If we use a little bit smaller page, as in the bottom two rows, we can reclaim some space. If we use a really tiny page, we can reclaim half the space even.

We’re still not reclaiming all the space, though. At the beginning of this series, I showed the “simplified perfect-world” thin provisioning illustration. In that picture, the half-empty barrel was perfectly reclaimed thanks to this technology. We will never get there unless we are using really minuscule pages. But we can get somewhat close. Maybe we can thin out three-quarters of the empty space.

But some vendors use really big pages. Some folks made fun of Hitachi for using 42 megabyte pages, since, if there’s one bit in 42 megabytes of potential ones or zeros, the Hitachi will not thin that. It also won’t migrate it for automated storage tiering. But others use even-bigger pages; up to a gigabyte in size. And 42 MB isn’t that bad in practice.

I know of a company that’s doing four-kilobyte pages. And EMC actually allocates one-gigabyte slices of storage for writing on the CLARiiON, even though their thin size is 8 KB. So is the CLARiiON page size 8 KB or 1 GB? It’s very confusing to me (and probably the customer too)…

The trouble with 4 K or 8 K pages is it makes an awful lot of pages to keep track of. Consider the analogy of hard disk drive sector sizes. An ATA disk could only get to 2.1 terabytes until recently, because they still used 512-byte sectors. And 512 bytes times the biggest 32-bit number is 2048 GB. So 512 bytes makes for greater efficiency in theory, but hurts scalability in practice. So, the disk drive industry is moving to 4 K sectors.

It’s exactly the same thing as with thin provisioning. So, you’ve got to keep track of all these gazillions and gazillions of pages. So, from a vendor perspective, you can save a lot of horsepower and make it a lot easier to implement if you have bigger pages. It also means you’re not moving stuff around as much when using these big pages for automated tiering.

I’m not going to throw rocks at HDS or anyone else over page sizes. I actually don’t think 42 MB is that bad, because the biggest problem with underutilization is not inside a file system. In my experience, the big problem is storage that’s not used at all.

When I used to do storage assessments, it was very common to find LUNs that were allocated ant not used at all; not even touched. Your page size doesn’t matter if a LUN is not even touched: It’s going to be thinned out no matter what. So, regardless of the page size, thin provisioning will probably save more space outside a filesystem than within one, especially if your systems administrators are doing a reasonably good job of storage management. And even if they’re not doing a good job, there’s probably 42 megs of zeros that can be thinned out anyway.

So, I’m not as worried about the size of the pages. Granularity is an architectural decision, and larger pages are not the end of the world. Ask your vendor if they support thin provisioning and what the granularity or page size is, and think about how that’s going to affect you. At the end of the day, it’s probably going to yield about the same result no matter what the page size is.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Granularity of Thin Provisioning Approaches
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My State of the Art Thin Provisioning series
  • Storage is Not Getting Cheaper
  • Thin Provisioning: Attacking Storage Utilization
  • De-Allocating is the Core Issue for Thin Provisioning
  • Thin Provisioning: Playing the Telephone Game
  • Monitoring Filesystem Metadata For Thin Provisioning
  • Zero Page Reclaim: Savior of Thin Provisioning?
  • What is WRITE_SAME? Green Eggs and Ham!
  • The Bridge: Veritas Thin (Provisioning) API
  • Is TRIM Useful For Thin Provisioning?
• The Raising the Floor podcast series
  • Raising the Floor 1: Cloud Storage Gateways
  • Talking Cloud Storage Gateways With Nasuni and Cirtas
  • Raising the Floor: How the Cloud Impacts Enterprise IT
• Tech Field Day links
  • Data Robotics Is First Three-Time Tech Field Day Presenter
  • The Nerd’s Going to Have a Field Day! | The Networking Nerd
• And more links!
  • I wrote this piece for SearchStorage: Is PCIe SSD right for you? Deploying PCI Express SSD devices
  • I was interviewed for another thin story: Thin provisioning driven by technology advances, best practices
  • My friend Erin Collopy from HP is starting a blog: A new year, a fresh slate « Social Swimmer
  • Bob Plankers always impresses with his technical writing: How To Create and Measure NTFS & ext3 Disk Fragmentation
  • I loved this idea from Freedom to Tinker: Monitoring all the electrical and hydraulic appliances in your house
  • Chris Mellor spills the beans on EMC’s big announcement next week: Storage Virtualization, Unified and the VNX
  • Chris also posted a great piece about InPhase: How I watched a holographic storage company implode
  • Then there’s this Winextra bit on Microsoft’s Biggest Mistake of 2010
  • A new EMCer blog, Storage Savvy, has a nice pNFS piece: Why pNFS can be a big deal even if NFS4.1 isn’t…
  • Martin Glassborow is always worth reading: Reading the Ruins
  • Floyd Strimling makes a go at prognostication: Fun Alert: Insane 2011 Predictions That May Come True
  • Another great Bill Hill post on cloud: Cloud Crossroads
  • And Ian Foster isn’t happy with Amazon: Amazon – Masters of marketing or perfecting the art of customer dissatisfaction?

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, January 7, 2011
This post was categorized as Computer History, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

If WRITE_SAME can be a semaphore for thin un-provisioning, what about TRIM? It sounds like a perfect fit, and has wider implementation to boot! Let’s take a deeper look.

First, I’d like to remind the reader that I’m not some specialist storage engineer. I am regular guy who wanted to know about this TRIM thing. I welcome corrections and feedback on this piece!

We should note that TRIM was invented for SSDs, not thin provisioning. TRIM lets an operating system tell a storage device that certain blocks of data are no longer used. Although this sounds like a thin provisioning tool, it’s not. TRIM was never intended to be used this way.

Even though Microsoft supports TRIM in the file system, and even though, hopefully, Apple will introduce TRIM support in OS X Lion, this doesn’t do what we need it to do for thin provisioning. In fact, TRIM is not anything for enterprise storage folks to be getting all that excited about.

One reason for is is that TRIM is an ATA command, and most enterprise systems are not connected to their storage using any kind of ATA-based protocol. Maybe there are SATA disks somewhere down the line, but, whether they use iSCSI, FC, SAS, or FCoE, the servers speak SCSI.

Even though there are SCSI analogs of of TRIM, these commands were really not designed for thin provisioning. They are designed for SSDs, which have a big performance problem: An SSD can delete data and write data just fine, but in order to update data, they actually have to read, delete, and write a big page of data. What happen is that, as an SSD gets full, it starts slowing down. And that’s a problem because people buy SSDs for speed. So TRIM is a way to tell the SSD, “Sometime in the future, you can go ahead and delete this if you feel like it,” and the SSD will take care of that later.

There’s great OS support for this, but unfortunately, this is not something that enterprise storage vendors are looking at. The only vendor (that I know of) that’s working on this, NetApp, did propose to T10 a mechanism to do thin provisioning based on TRIM. I don’t think it got anywhere. They say that it did, but I just don’t see it. HDS and EMC, seem to like UNMAP and SCSI, which are peripherally related, but I don’t know much about what they’re doing either. Perhaps they’ll let me know in comments on this post.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Is TRIM Useful For Thin Provisioning?
This post was categorized as Apple, Enterprise storage, Everything, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Thin provisioning needs communication to function, and zero page reclaim is only the array side of the story. WRITE_SAME helps reduce I/O load, but the server needs to use it. Wouldn’t it be nice if the operating system, file system, or volume manager would use these commands to help recover capacity?

We need a bridge. We need to cross the gulf between the storage array (that expects all these zeros or some other way to recover this space), and the host (who has this information)

That’s the thing that gets me riled up. The application and the file system know that this data is no longer used. If there was only some way, some bridge, that allowed them to talk to the storage array and say, “Hey, you can get rid of that now. I don’t need it anymore.” See? We need a bridge.

Well, guess what? We have some bridges. One of my favorite developments in storage of the last year is this Thin Reclamation API that Symantec has. I really like what they’re doing.

Symantec (formerly Veritas) have the file system and the volume manager. If you’re an enterprise guy like me (I was using HP-UX in 1996) you have used the Veritas File System and Veritas Volume Manager before.

Symantec calls all this Veritas Storage Foundation now. It has the information thin provisioning storage arrays need, and they’ve come up with a way to communicate: Being a software company, Symantec implemented just about every communication method, because not everybody uses this WRITE_SAME, and not everybody uses zero page reclaim. Some of them have their own API for recovery. So Symantec tried to implement as many thin methods as they possibly could right in the file system and volume manager.

I’ve talked to a lot of people who are using it, and they say that it works. They say that it’s brilliant. Because you turn on this feature and, suddenly, the file system starts telling the array “I don’t need this data anymore.” Suddenly one of my major complaints about thin provisioning goes away.

But I have a question: Why doesn’t everybody do this? Why isn’t this in the Linux file systems and volume managers? Why isn’t Microsoft working on this? Why doesn’t everybody have this capability? It’s not rocket science. It just takes a little bit of development effort to try to come up with some kind of way to have the file systems talk.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Bridge: Veritas Thin (Provisioning) API
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.