The great battle of enterprise storage is on!

The time has come to take sides on the core question of storage for virtual servers: Do you want storage intelligence to live in the hypervisor or the array? Most administrators are already lining up on one side or the other, unintentionally casting their vote while the rest flounder. But the storage industry must wake up and embrace the divide.

Hypervisor Huggers Unite!

Do you "heart" virtualization?

VMware’s vSphere dominates the world of enterprise server virtualization and has become the core element of the modern open systems datacenter. Microsoft recognizes this but has been unable to field a competitive hypervisor ecosystem for the virtual datacenter. Today, vSphere is the state of the art and nowhere is this more apparent than in storage.

In just a few years, VMware has delivered and updated a host of advanced storage functionality, from provisioning to migration and load balancing to backup and data protection. vSphere 5 includes an advanced and scalable storage virtualization layer, delivering everything a datacenter needs. VMFS sculpts basic block storage into a shared resource for virtual machines, with snapshots, policy-based layout and movement, and flexible allocation and thin provisioning.

Most VMware administrators are “server guys” and relish these features. They have never experienced an automated “storage service” like this, and the enterprise storage world has never been able to construct anything remotely as flexible, user-friendly, and functional. And Hypervisor Huggers don’t need complex enterprise storage arrays to do it: They can use basic iSCSI or Fibre Channel devices to provide performance and capacity and let VMware do the rest!

Storage DRS is exemplary of the new virtual datacenter world. Introduced in vSphere 5 (and restricted to the pricey Enterprise Plus license), Storage DRS uses the core technology of Storage vMotion to dynamically balance I/O and capacity across a diverse pool of storage. Storage DRS even uses Policy-Driven Storage and VASA to enforce tiered storage and data placement strategy. This kind of virtualization has been a “holy grail” quest for the enterprise storage industry, but they’ve never delivered on their promises.

Cheers for Storage Stalwarts!

Do you wish all this virtualization nonsense would just go away?

But not every IT environment wants be 100% vSphere focused, and many aren’t convinced that dumb storage is the smartest place for their data. These Storage Stalwarts want smarter and better-integrated storage arrays, and VMware is innovating here as well.

VMware’s Storage APIs for Array Integration (VAAI) technology is squarely aimed at this market. VAAI allows vSphere to hand off heavy storage operations to the high-end storage arrays from the major players. It works transparently, too, coordinating cloning without the kind of scripting and hair-pulling that used to require. VAAI in vSphere 4.1 also includes block zeroing support and something called “atomic test and set” which we’ll get to in a moment. Microsoft announced their own cloning integration, ODX, but it won’t ship until Windows Server 8 appears sometime next year.

But cloning is just the tip of the iceberg when it comes to Hypervisor-to-array integration. The rising army of NAS users have seen the glory of more-complete array integration for a while, and they’re not quiet about it. They love that VMware’s NFS protocol support makes storage “disappear” in vCenter, becoming just another resource with integrated thin provisioning and flexible allocation and movement.

VMware is moving aggressively to please their Storage Stalwarts, adding more VAAI support for block and file in vSphere 5. But, as the company laid out at VMworld 2011, neither access method is ideal for virtual servers. So VMware has been pushing the enterprise array vendors for ever-greater integration. They see a future where a VAAI-based protocol enables arrays to de-multiplex I/O streams from the hypervisor and intelligently handle per-VM data.

Stephen’s Stance

You can spot a Hypervisor Hugger by their big LUNs: They would rather treat storage as a bulk commodity, and array vendors should be lining up to get their business. Storage Stalwarts will jump on each new VMware innovation, finally making use of the capabilities they have spent over a decade paying for but not utilizing. The only untenable stance is trying to keep a foot in both worlds: It’s foolish to buy an enterprise array and use it as bulk storage!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Are You a Hypervisor Hugger or a Storage Stalwart?
This post was categorized as Computer History, Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

EMC mentioned this schedule on their public earnings call, repeated it in discussions with Enterprise Storage Forum, and corporate PR confirmed it to me this morning. FAST is, if you will excuse the pun, fast-approaching!

There’s actually a very-significant difference between FAST versions 1 and 2:

• FAST version 1 (coming in December 2009) can automatically migrate an entire LUN between tiers of storage on the Symmetrix V-Max based on access patterns. This is a nice-to-have feature, but does not yet live up to the promise of automated tiered storage, as pioneered by Compellent and offered in various forms by many other storage companies. I am told that FAST version 1 also does not support virtually-provisioned (thin) volumes. Although FAST will make it much easier to take advantage of solid state flash drives (EFDs to EMC), it will do nothing to improve utilization.
• FAST version 2 (coming in May mid-2010?) will automatically move smaller sub-LUN pieces of storage between tiers of storage, and will reportedly support virtual provisioning. It is likely that FAST would use the same extent size supported by virtual provisioning, which is 12 tracks or 768 KB. The combination of virtual provisioning and fully-automated storage tiering will finally EMC the ammunition they need to defend their turf against smaller challengers when it comes to these sticky technical features.

So there you have it. EMC V-Max gets FAST in December 2009 and May mid-2010! Now how about a schedule for geo-distributed clustering?

Update: Fixed two typos above. I wrote “May 2009″ instead of “May 2010″. You should see my checkbook!

Note: Although Lou Przystas told Enterprise Storage Forum that FAST V2 would come in May, EMC has apparently not set a hard date and are saying “mid-2010″ at this point. Even this is an aggressive ship date, but I won’t say EMC missed it unless we haven’t got it by the end of summer!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | EMC V-Max FAST: Coming in December … And 2010!
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3

Expanded Support for Enhanced vmxnet Adapter

Not specifically a storage change, but the enhanced vmxnet adapter introduced back in the original release of ESX 3.5 now works with most versions of Windows Server 2003 and XP Pro. Look for improved performance when using guest-side SMB and NFS as well as the guest iSCSI initiator. Note that you cannot select this driver when configuring non-Enterprise Edition machines; you have to select Windows Server 2003 Enterprise Edition (64-bit) regardless of which version of Server 2003 you are using.

Expanded SAS and SATA Controller Support

If you’d like to install ESX on a server equipped with a PMC 8011, Intel ICH9 or ICH10, CERC 6/I SATA/SAS Integrated RAID Controller, or HP Smart Array P700m Controller, you’ll find happiness in Update 4.

The Intel controllers are especially important, as we’re seeing them used more and more and this driver is more full-featured than the earlier Broadcom HT 1000 and Intel ICH7 drivers. The Intel ICH9/ICH10 is a dual-mode (IDE/ATA and AHCI/SATA) driver, supports SATA hard drives, SSDs, and optical drives, and now enables VMFS support when in AHCI/SATA mode. It’s not clear whether VMware actually supports VMFS datastores on ICH9/10 SATA, but it says it works. Anyone want to try it out? One thing is certain: You can’t use SATA drives in a shared/clustered environment because SATA does not include reservations. See this tech note and especially this question:

Earlier, it was mentioned that we can create VMFS if we use AHCI/SATA mode. If so, why did VMware not claim VMFS support when using SATA controller running in AHCI/SATA mode?

VMware might decide to add support in the near future. There is no strong need to have VMFS support on a SATA drive, because native SATA protocol does not support reserve/release. Reserve/release is needed if VMFS is used as clustered file system in a shared disk environment.

PXE Boot Support

Rich at VM/ETC points out that Update 4 includes experimental PXE boot support for ESX and ESXi. As he notes, this has major implications for cloud computing platforms, since it means that ESX servers can boot guests without local storage at all. Very interesting! Let’s bet that Update 5 (expected in June or July) will include this as a supported option.

Updated QLogic, Emulex, and LSI Drivers

Like most ESX updates, this one included updated Fibre Channel drivers.

• The QLogic Fibre Channel Adapter driver and firmware (versions 7.08-vm66 and 4.04.06, respectively) include bug fixes and enhanced NPIV support.
• On the Emulex side, driver version 7.4.0.40 supports the company’s HBAnyware 4.0 management software.
• Users of SAS and SCSI LSI MegaRAIDs will find driver version 3.19vmw (megaraid_sas) and 2.6.48.18 vmw (mptscsi) which improves performance and enhances event handling capabilities.

Expanded Sun Storage Array Support

All you StorageTek loyalists out there will be happy to see support for Sun’s low-end StorageTek 2530 SAS array as well as the modular 6580 and 6780 Fibre Channel arrays. It looks like just about every model in Sun’s current storage lineup is now supported in ESX.

Expanded Network Card Support

Support for Gigabit cards is greatly expanded, including HP’s quad-port NC375i and dual-port NC362i and NC360m, Intel’s Gigabit CT and 82574L, and NetXtreme’s BCM5722, BCM5755, BCM5755M, and BCM5756. Intel’s widely-used 10-gig 82598EB cards are now supported as well.

Tweaks and Fixes

Looking through the release notes, a few storage-related tweaks and fixes stand out:

1. WMware can optionally automatically throttle back the queue depth when congestion is encountered. See Controlling LUN queue depth throttling in VMware ESX for 3PAR Storage Arrays for more information.
2. VMklinux module heap size can now be adjusted as LUN queue-depth values are increased. Since tuning LUN queue depths is one common trick of the storage trade to improve performance, especially in queue-stingy systems like ESX, this is welcome news. But call VMware support before you monkey with it!
3. An RDM-related issue where SCSI inquiry data over 36 bytes was truncated or corrupted (for example when using Microsoft VSS and NetApp SnapDrive) has been resolved.

Well, that’s all folks. I suggest you all read the release notes for yourself, and please leave a comment if you see an error in what I wrote here or have something to add!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Changes in VMware ESX 3.5 Update 4
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It has been the core technology behind the storage industry since day one, but the sun is setting on traditional RAID technology. After two decades of refinement and fragmentation, we are abandoning the core concepts of disk-centric data protection as storage and servers go virtual. Next-generation storage products will feature refined and integrated capabilities based on pools of storage rather than combinations of disk drives, and we will all benefit from improved reliability and performance.

RAID Classic

Early storage systems were revolutionary, in physically removing storage from the CPU, in enabling sharing of storage between multiple CPUs, and especially in virtualizing disk drives using RAID. When Patterson, Gibson, and Katz proposed the creation of a redundant array of inexpensive disks (RAID) in 1987, they specified five numbered “levels”. Each level had its own features and benefits, but all centered on the idea that a static set of disk drives would be grouped together and presented to higher-level systems as a single drive. Storage devices, as a rule, mapped host data back to these integral disk sets, sometimes sharing a single RAID group among multiple “LUNs”, but never spreading data more broadly. Storage has remained stuck with small sets of drives ever since.

The core insight of the 1980s remains true: More spindles means better performance. Although additional overhead dulls the impact somewhat, the benefit of spreading data across multiple drives can be tremendous. A typical RAID set offers much better performance than the drives alone, and can handle a mechanical failure as a bonus.

Cracks are appearing in the RAID veneer, however. Double drive failures are much more common than one would expect, leading to the development of hot spare drives and dual-parity RAID 6. If four drives perform well, then forty drives perform much better, leading to the common practice of “stacking” one RAID set on others. Caches and specialized processors were introduced to overcome the performance issues related to parity calculation.

But traditional RAID cannot overcome today’s most critical storage issues. As drives have become larger, the tiny chance of an unrecoverable media error compounds, becoming a certainty. Even dual-parity will not be able to guarantee data protection on the massive disks predicted for the near future – statistics cannot be denied. The latest disks contain so much data, without commensurate improvements in throughput, that rebuild times have skyrocketed, resulting in hours or days of reduced data protection.

RAID is also ill-suited to the demands of virtualized systems, where predictable I/O patterns become fragmented. It cannot provide tiered storage or account for changing requirements over time. It cannot take advantage of the latest high-performance solid state storage technology. It cannot be used in cloud architectures, with massive numbers of small devices clustered together. It interferes with power-saving spin-down ideas. Most RAID implementations cannot even grow or shrink with the addition or removal of a disk. In short, traditional RAID cannot do what we now need storage to do.

RAID is Dead

Although most vendors still use the name, nearly every one has abandoned much of the classic RAID technology. EMC’s Symmetrix pioneered the idea of sub-disk RAID, pairing just a portion of each disk with others to reduce the impact of “hot spots”. HP’s AutoRAID added the ability to dynamically move data from one RAID type to another to balance performance. And NetApp paired disk management so closely with their filesystem that they were able to use RAID 4 and the flexibility it brings.

Today, a new generation of devices has even evolved beyond RAID’s concept of coherent disk sets. Compellent, Dell EqualLogic, 3PAR and others focus on blocks of data, moving portions of a LUN between RAID sets, disk drive types, and even inner or outer tracks based on access patterns. With these devices, a single LUN could encompass data on every drive in the storage array. And the latest clustered arrays can spread data across multiple storage nodes to scale performance and protection.

These innovative devices point the way to a future in which virtual storage is serviced and protected very differently than in the past. Perhaps software like Sun’s ZFS serves to illustrate this future best: It unifies storage as a single pool, intelligently protecting it and presenting flexible storage volumes to the operating system. Although Sun calls its data protection scheme “RAID-Z”, it has little in common with its namesake. Like NetApp’s WAFL, the copy-on-write ZFS filesystem is totally integrated with the layout of data on disk, allowing mobility and efficient use of storage. A single pool can include striping, single- or dual-parity, and mirroring, and disks can be added as needed. Importantly, ZFS also checksums all reads, detecting disk errors.

Long Live RAID

The post-RAID future will see these concepts spread across all enterprise storage devices. Disks will be pooled rather than segregated into RAID sets. Tight integration between layout and data protection will allow for much greater flexibility, integrating tiering and differing data protection strategies in a unified whole. Storage virtualization will allow mobility of data within these future storage arrays, and clustering will enable massive scalability.

Two things will likely remain to remind us of Patterson, Gibson, and Katz, however. First, the core principle that multiple drives working as one yields dividends in terms of performance and data protection. And second, that whatever we use should be called RAID, even though the definition of that term has changed beyond recognition in the last two decades.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Turning the Page on RAID
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What’s Wrong With Thin Provisioning?

As I have explained in my storage seminars, thin provisioning is the opposite of what storage management professionals should be doing: Instead of managing usage, we just throw up our hands and say “you want 500 GB? Fine, you’ve got it!” while all the while only provisioning a fraction of that space. It’s a lie, and is thus bound to catch up with us sooner or later, and probably at just the wrong time.

People use disk space like money – their needs tend to expand to use up all they can get. Tell the users that you just added another 8 TB to the file server and watch their usage spike. Tell a database manager that they need to buy 20 TB and watch as their tablespaces magically start using 19. It’s human nature, and fighting this impulse to consume is precisely what management is all about. Traditional thin provisioning (or “virtual provisioning” in EMC-speak) does exactly this – it “tells” the downstream users of a storage resource that they have more capacity than is actually assigned to them and then grows capacity as it is used. To say that it is controversial is an understatement.

In certain instances, including Drobo and VMware growable disks, this can be beneficial since it’s a pain for most end-user OS configurations to resize a volume after the fact. In these special cases, I concede that thin provisioning is the right way to go. The same could be said for deduplicating or compressed storage – these simply have to be thin provisioned, since the actual allocation is completely abstracted by the compression algorithm. Thin provisioning can also help (slightly) for the OS volumes of virtual servers. But mainstream enterprise users have storage, server, and application managers, so they shouldn’t resort to “tricks” like thin provisioning – instead, they should manage their storage!

But the worst thing about thin provisioning is that it can’t un-provision storage. Let’s say a user uses your thin-provisioned file server as a temporary landing zone while switching to a new laptop. Or your database folks load their LUNs up with SQL dumps after an outage. Or your application folks fill up their test servers prior to going into production. Predictably, that thin-provisioned storage will expand, using up real disk capacity, to take the load (presuming enough capacity is available). The problem arises when they delete this temporary data – the storage array has no way of knowing that those blocks are no longer in use, so it cannot un-provision them. Suddenly your 500 GB thin-provisioned LUN is really taking up 400 GB even though it only has 20 GB of actual data on it, and you feel like a chump. Time to go manage your storage…

Zero-detection helps a little

Now back to 3PAR. Their new T-class InServ storage array has a special ASIC designed to attack a small chunk of the un-provisioning problem. It scans allocated storage, looking for blocks filled with zeros, and de-provisions them. This is nice – it’s a great tool to convert traditional storage to thin-provisioned storage. It’s also the first practical un-provisioning approach I’ve heard about, and might yield some capacity improvements for already-provisioned LUNs in certain special cases, though I’m not sure 3PAR is aiming for this market.

See, it’ll only work for zeroed-out storage, which is sadly extremely rare in the world of storage. It will detect capacity that has never been used, but most filesystems simply change their pointers when a file is deleted – leaving the data just where it was. 3PAR’s effort won’t work in this case. Even decommissioned servers often leave their LUNs full of old data, a security risk to be sure, and not a case that 3PAR could deal with, either.

The only way to make this work for already-used storage would be to add another step to the decommissioning process – zero out LUNs that are no longer in use as a way to send a signal to the storage array that it can un-provision that storage. But of course, we could also just send an email to the storage administrator to de-allocate the LUNs, leaving us in a much better position since we no longer have unused LUNs sitting on the storage array. Maybe we could modify the filesystem to zero out unused storage. Anyone have the source code for NTFS?

Seriously, though, this is a practical step in the right direction. We need better communication between applications, operating systems, and storage in order to enable lots of beneficial features. 3PAR is trying to enable some communication, and I applaud them for that. Just don’t expect too much.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | 3PAR’s Thin Un-Provisioning is Slightly Less Bad
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.