The great battle of enterprise storage is on!

The time has come to take sides on the core question of storage for virtual servers: Do you want storage intelligence to live in the hypervisor or the array? Most administrators are already lining up on one side or the other, unintentionally casting their vote while the rest flounder. But the storage industry must wake up and embrace the divide.

Hypervisor Huggers Unite!

Do you "heart" virtualization?

VMware’s vSphere dominates the world of enterprise server virtualization and has become the core element of the modern open systems datacenter. Microsoft recognizes this but has been unable to field a competitive hypervisor ecosystem for the virtual datacenter. Today, vSphere is the state of the art and nowhere is this more apparent than in storage.

In just a few years, VMware has delivered and updated a host of advanced storage functionality, from provisioning to migration and load balancing to backup and data protection. vSphere 5 includes an advanced and scalable storage virtualization layer, delivering everything a datacenter needs. VMFS sculpts basic block storage into a shared resource for virtual machines, with snapshots, policy-based layout and movement, and flexible allocation and thin provisioning.

Most VMware administrators are “server guys” and relish these features. They have never experienced an automated “storage service” like this, and the enterprise storage world has never been able to construct anything remotely as flexible, user-friendly, and functional. And Hypervisor Huggers don’t need complex enterprise storage arrays to do it: They can use basic iSCSI or Fibre Channel devices to provide performance and capacity and let VMware do the rest!

Storage DRS is exemplary of the new virtual datacenter world. Introduced in vSphere 5 (and restricted to the pricey Enterprise Plus license), Storage DRS uses the core technology of Storage vMotion to dynamically balance I/O and capacity across a diverse pool of storage. Storage DRS even uses Policy-Driven Storage and VASA to enforce tiered storage and data placement strategy. This kind of virtualization has been a “holy grail” quest for the enterprise storage industry, but they’ve never delivered on their promises.

Cheers for Storage Stalwarts!

Do you wish all this virtualization nonsense would just go away?

But not every IT environment wants be 100% vSphere focused, and many aren’t convinced that dumb storage is the smartest place for their data. These Storage Stalwarts want smarter and better-integrated storage arrays, and VMware is innovating here as well.

VMware’s Storage APIs for Array Integration (VAAI) technology is squarely aimed at this market. VAAI allows vSphere to hand off heavy storage operations to the high-end storage arrays from the major players. It works transparently, too, coordinating cloning without the kind of scripting and hair-pulling that used to require. VAAI in vSphere 4.1 also includes block zeroing support and something called “atomic test and set” which we’ll get to in a moment. Microsoft announced their own cloning integration, ODX, but it won’t ship until Windows Server 8 appears sometime next year.

But cloning is just the tip of the iceberg when it comes to Hypervisor-to-array integration. The rising army of NAS users have seen the glory of more-complete array integration for a while, and they’re not quiet about it. They love that VMware’s NFS protocol support makes storage “disappear” in vCenter, becoming just another resource with integrated thin provisioning and flexible allocation and movement.

VMware is moving aggressively to please their Storage Stalwarts, adding more VAAI support for block and file in vSphere 5. But, as the company laid out at VMworld 2011, neither access method is ideal for virtual servers. So VMware has been pushing the enterprise array vendors for ever-greater integration. They see a future where a VAAI-based protocol enables arrays to de-multiplex I/O streams from the hypervisor and intelligently handle per-VM data.

Stephen’s Stance

You can spot a Hypervisor Hugger by their big LUNs: They would rather treat storage as a bulk commodity, and array vendors should be lining up to get their business. Storage Stalwarts will jump on each new VMware innovation, finally making use of the capabilities they have spent over a decade paying for but not utilizing. The only untenable stance is trying to keep a foot in both worlds: It’s foolish to buy an enterprise array and use it as bulk storage!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Are You a Hypervisor Hugger or a Storage Stalwart?
This post was categorized as Computer History, Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

FalconStor NSS 7.0 brings VMware VAAI support to any storage array

FalconStor Software announced this week that Network Storage Server (NSS), their SAN virtualization solution, now supports VMware VAAI. This is big news for two reasons: Is the first time a smaller storage company has added VAAI support, and this is the first software virtualization product to do so. Users of this FalconStor solution will be able to use the three ESX 4.1 VAAI primitives with a wide variety of third-party storage devices.

You would probably also be interested in reading VMware VAAI Storage Array Support in Plain English

VMworld 2011 was a flurry of activity, but the FalconStor booth caught my eye. There, on the backdrop, were the four letters I’ve been tracking throughout the storage industry: VAAI. VMware’s array integration API brings powerful capabilities for the hypervisor to directly communicate with supported storage devices. Once enabled, VAAI “just works”, accelerating storage the motion and file locking and improving storage efficiency through thin provisioning.

But VAAI support is very difficult for storage vendors to bring to market. It requires capabilities that may not be present in some storage systems as well as the development of a plug-in for VMware vSphere. Then, vendors must go through certification with VMware before being added to the hardware compatibility list.

When it was initially announced as part of vSphere 4.1, VAAI support was limited to a few large companies. Since then, only IBM has added substantial support for the API, and no small vendor has been able to step up. This has somewhat dampened my enthusiasm for VAAI, since I would like to see widespread support across the industry.

Now that vSphere 5 has been announced, along with an extended version of VAAI, I hope more vendors will join the party. But until then, the FalconStor announcement brings hope to those who own unsupported systems. NSS is a block storage virtualization platform, and version 7.0 brings VAAI to any storage array. It is available as a software product or virtual appliance or as a physical appliance solution.

In this way, it is similar to the IBM SVC and HDS USP products. But FalconStor NSS is substantially less expensive than those offerings, more suitable for smaller businesses with “alternative” storage devices. FalconStor software is also widely offered by storage resellers, who will now be able to offer VAAI compliant solutions from vendors other than the big names.

Stephen’s Stance

I do not necessarily endorse or recommend FalconStor NSS over competing products from more familiar names, but I commend them for adding VAAI support. There the first small vendor to do so, and their software virtualization platform spreads the availability of this important software capability. For that, they should be commended!

No disclaimer: I’m not associated with FalconStor in any way.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | FalconStor Brings VAAI Support To Every Storage Array
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

IBM today announced VAAI support for SVC, and promised it for the DS8000

VMware’s vStorage API for Array Integration (VAAI) is one of the most-important storage technology advances of the decade, allowing the VMware ESX to integrate and coordinate operations with supported enterprise storage arrays. After an explosion of initial support, however, VAAI implementation slowed. IBM was notably absent from the party, listing only partial support for a single product line for the better part of the year. But they’ve turned on the VAAI heat, releasing full support for the XIV and SVC and promising DS8000 in the near future.

VAAIing for Attention

For the complete updated VMware VAAI support matrix, see my post, VMware VAAI Storage Array Support in Plain English

Under vSphere 4.1, VAAI includes three “primitives”, or supported features:

1. Full copy enables the hypervisor to direct the storage array to copy data without sending it all the way “up the stack”
2. Block zeroing supports enhanced thin provisioning communication
3. Hardware-assisted locking allows the hypervisor to more-efficiently share a SCSI LUN

IBM’s original VAAI support amounted only to the basic T10 zeroing support using the generic “vmw_vaaip_t10″ plugin on the SVC and related Storwize V7000. This fell well short of the specialized three-primitive plugins offered by rivals like EMC and HDS.

IBM VAAI Support is Spreading

Far from being asleep at the wheel, IBM was busy creating a complete VAAI implementation that would be portable across a variety of storage platforms. XIV was the first beneficiary of this work, gaining full VAAI support and a listing in the VMware ESX HCL. This came in version 10.2.4 of the XIV software, released and approved in April of 2011. But insiders hinted something more was on the way.

Today, IBM announced that their “IBM_VAAIP_MODULE” plugin would support the SAN Volume Controller (SVC) and related Storwize V7000 as well. The array-side functionality will be added in version 6.2 of the software, which IBM promises to deliver in June of this year.

Now, at the IBM Storage Summit, I hear that this same code will enable the DS8000 to support all three VAAI primitives shortly. Once this is delivered, IBM’s key platforms (XIV, SVC, and DS8k) will all support VAAI, putting IBM on par with their competitors.

Stephen’s Stance

I am pleased to see IBM planning full VAAI support, since I feel this is absolutely key to future storage developments. I am further impressed that they developed a cross-platform VAAI codebase that could be deployed across such diverse architectures as XIV, SVC, and DS8k. Once this is released, IBM will finally be up to speed with their key competition.

I won’t recommend anything that’s not on the official VMware hardware compatibility list (HCL), and I’m glad to see that V7000 and XIV is now listed. It will take months DS8k to appear, though. By that time, vSphere 5 (with its rumored VAAI enhancements) will be here. Hopefully IBM will be out day-and-date with support.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | IBM Adds VAAI Support to XIV and SVC
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Iomega ix4-200d is a sleek 4-drive SOHO RAID system that does just about everything, from NAS to Time Machine to iSCSI for a list price right around $700

Iomega is well into its second coming as EMC’s entry-level storage division. First, they applied EMC’s compact and full-featured LifeLine home storage software to existing gear, giving birth to the Home Media Network Hard Drive, StorCenter ix2, and StorCenter Pro ix4-100. Then they wooed the small-business community with the rack-mount StorCenter ix4-200r, adding iSCSI target support and VMware compatibility.

Today, they are back with the new ix4-200d, probably Iomega’s best product yet. It includes every feature of the rack-mount ix4-200r, including NAS and iSCSI target mode plus great new stuff like one-touch synchronization. All of this is packaged in a Drobo-like desktop system with a starting list price of just $700, or less than half the cost of a comparable ix4-200r!

Desktop Storage

External desktop storage products, exemplified by Western Digital’s wildly successful My Book series, have been a huge retail hit. Priced just over $100, these drives pack a terabyte or more and offer plug and play simplicity. I recently visited a small business with a WD or Seagate USB enclosure on every single desk. I’ve purchased five USB- or FireWire-connected hard drives myself over the last two years!

But these single-drive desktop wonders are a disaster waiting to happen:

• They fail frequently (like my Maxtor 3200), instantly wiping out the data they contained
• They are targets for thieves, so data loss prevention (DLP) experts warn against their use
• They aren’t shareable natively, so most people resort to sneakernet swapping rather than fight with Windows to present them as a network share
• When they’re full, they’re full, forcing the purchase of a whole new drive

Many vendors sell grown-up versions that address some or all of these concerns with multiple drives, network connections, and encryption, but these have been slow to catch on. Since they contain redundant drives and extra hardware and software, they are much more expensive than their little cousins. Iomega has done battle in this arena with their original StorCenter ix2 and the ix4-100, but these have yet to catch on. Even Drobo, with their fanatical user-friendly focus, has failed to convince many buyers.

Then there is the world of business storage. Way down at the bottom of the enterprise storage pyramid lies the realm of small 4- and 8-drive storage arrays. These SMB storage arrays offer a lot of capacity and reliability for the money but very little in the way of features. Iomega’s StorCenter Pro ix4-200r, in contrast, wowed the techies with a full iSCSI target stack that was certified for VMware ESX. But the price, over $1500, definitely limited sales to the home hobbyist.

“Just Right” Storage

Although the name is similar to the StorCenter Pro ix4-200r launched this Spring, Iomega went back to the drawing board for the ix4-200d. They built an entirely new device that could offer the impressive features of their StorCenter Pro line at a price closer to consumer storage offerings. The result boasts everything the Pro has and more at less than half the price:

• You want connectivity? The ix4 sports dual gigabit Ethernet ports that can be teamed up for performance or split off for redundancy.
• You want NAS? The ix4 supports NFS, SMB, and even Apple’s AFP, plus it’s Active Directory compatible.
• How about iSCSI? It’s a full-featured iSCSI target, certified for Microsoft Server 2003 and 2008.
• Want to host virtual machines? The ix4 is certified with VMware ESX 4 vSphere using both NFS and iSCSI, on the Xen HCL, and that Microsoft logo means it will work with Hyper-V as well.
• Need backup? The ix4 comes with EMC’s Retrospect and supports OS X Time Machine over AFP just like an Apple Time Capsule.
• Looking for weird features? How about support for up to 5 Axis network cameras, BlueTooth Picture Transfer Protocol, and UPnP/DLNA media service!
• Iomega also added a new feature, QuickTransfer, to synchronize files between devices.

This is one seriously feature-rich storage system. In fact, this glut of features is the ix4′s Achilles heel: How do you effectively communicate the value of a device that does so much? Most of the buying public has never heard of most of these features, so the price remains hard for some to justify.

Every ix4 configuration comes loaded with four hard disk drives

Another hurdle for the ix4 is Iomega’s decision to fill it with hard drives. One cannot buy an empty ix4-200d, and both the 2 TB and 4 TB configurations come loaded with four hard disk drives. This raises the price of entry and scares off the very techies who might be interested in the device. The majority of Drobos are purchased with no drives at all, and storage geeks like me love the idea that disks can be added as-needed in the future. Although Iomega is open to users swapping out the drives in their own StorCenter device, this is not its intended use case. Iomega’s decision to sell the ix4-200d as a loaded appliance seems counter to the price sensitivity and flexibility needs of buyers.

QuickTransfer: Data Synchronization for Everything

One nifty new feature included with the ix4-200d is QuickTransfer, a one-touch data synchronization capability. Leveraging rsync technology, but hiding this complexity with a more-friendly wizard-based GUI, QuickTransfer allows users to set up synchronization jobs between the ix4 and a variety of targets:

• USB drives can be plugged into one of the three USB 2.0 ports and synchronized with a subset of the content of the ix4. For example, a portable USB drive could be “recharged” with the latest set of data before one heads out of the office.
• Two Iomega ix4′s, or other NAS systems for that matter, can be synchronized over the Ethernet/IP LAN. This would provide a robust and bandwidth-friendly remote office backup or data replication solution.
• A PC or Mac can also be synchronized over a network share, providing a simple alternative to the bundled backup software.

QuickTransfer is exclusive to the ix4-200d for now, but Iomega assured me that it would be included in future StorCenter products and added to the StorCenter Pro ix4-200r in the near future. It is unknown if or when other existing StorCenter devices will get QuickTransfer, however.

Iomega’s Next Move

What will Iomega do next? They must be ready to announce their vSphere 4 and Microsoft Windows Server certification soon, since both company’s web sites already list the device in their compatibility lists. Iomega ought to try to take advantage of the interest among VMware users with a big VMworld splash. They will be there, but it is awfully hard to get noticed at such a large event. I am looking forward to the event to get a hands-on test.

The ix4 should begin showing up for sale at online stores very quickly. Amazon listed the rackmount product within days of its release, and we expect the same this time. But will Iomega offer this cheaper device in retail stores? It would be great to have it available at Staples and Best Buy, but shelf space for a storage system this expensive would be hard to get. Instead, expect it at specialty outlets like Fry’s and perhaps Micro Center.

How much does the StorCenter ix4-200d cost?

• The 2 TB model (SKU# 34546 with four 500 GB drives) lists at $699.99
• The 4 TB model (SKU# 34549 with four 1 TB drives) lists at $899.99
• The 8 TB model (SKU# 34563 with four 2 TB drives) lists at $1,899.99

It also seems likely that the 2-bay product is up for a refresh in the near future. The ix2 can’t be said to be very attractive, so hopefully the company will do something about this with the next-generation product. It is unclear whether the inexpensive 2-bay device will get iSCSI support, but I suspect it will. Beyond this, might Iomega move further upmarket with an 8-drive unit? We shall see!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Iomega’s ix4-200d: A Killer Desktop Storage Array
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Most important: Please register to be a marrow donor, especially if you might be a match for Nick Glasgow of EMC!

Misc

• The week of May 16 will forever be known as the week Twitter lost control of their eponymous service. There were some great posts, many of which went beyond whining and offered context on why Twitter is so important, but the one that sticks in my head most was a simple statement: “Goodbye People I Never Knew“
• But it wasn’t all bad news related to Twitter. Witness this awesome idea: A Modest Proposal For More Microstructure: Twitter /Locations
• My friend Donavon was featured in Baltimore Business Journal talking about his nifty home servers, and pictured with his humidor!
• A simple pearl of MBA wisdom: Ignore sunk costs
• And if you’re using LinkedIn, here are Five Dos and Don’ts
• The best new product of May? Easily it’s Lego’s new Frank Lloyd Wright models!
• Wilco’s new album was leaked, but rather than freak out they put up a free streaming version. It’s Grrrrrrreat!

Enterprise Computing

• vSphere finally hit, and lots of in-depth information appeared. I loved learning more about vSphere Memory/CPU Hotplug, a topic I knew nothing about! Also, see my Gestalt IT post answering Will the vSphere 4 Upgrade Require Another VMFS Update?
• The Doeswijk Data Model presents a wonderful way of thinking about storage capacity and growth.
• One of my favorite Windows 7/Server 2008 R2 features is the way that VHDs are handled: Learn more in Native VHD Support in Windows 7 and Less Virtual, More Machine – Windows 7 and the magic of Boot to VHD
• New to storage? Brent Ozar (aka The Man) wrote a series on SAN Multipathing. See Part 1: What are Paths? and Part 2: What Multipathing Does
• As everyone following my Twitter or Flickr feed knows, I was in Prague last week at the Sys-Con Cloud Expo. While there, I met up with a few others, including Guy Bunker, Sam Johnston, Roman Stanek, and others. See Time for a debrief!, Review of the Reviews, and the Photo Album

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Back From The Pile: Interesting Links From Mid-May
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Apple

• MacBook Users: Turn off This Bluetooth Default Setting Now – Now I know what turned on my MacBook Pro in the bag: My BlueTooth mouse!

Enterprise Computing

• Windows Storage Server-Based Systems Step Into 2008 – My Gestalt IT coverage of the features of WSS08.
• EMC Symmetrix V-Max: When Does It Get FAST and Virtual? – Another Gestalt IT piece, pointing out that the V-Max isn’t going to be fully realized for a very long time
• Microsoft Unveils Final Windows Storage Server – Paul Shread runs with the “Windows Storage Server 2008″ theme, quoting yours truly extensively.
• What is a best practice and Are Best Practices Just Shared Opinions? – W. Curtis Preston and I pickup an old conversation about the definition of “best practice”.
• Infosmack Episode 2 – EMC, David Donatelli and Non-Compete Agreements – Tony Asaro and I join Marc Farley and Greg Knieriemen on the Storage Monkeys podcast!
• The US Federal Government defines Cloud Computing - Ruven Cohen covers the US Federal Government’s cloud computing meetings and gives us a sneak peek at the new definition of cloud computing from the world’s number one IT user.
• Support and Q&A for Solid-State Drives – Lots of great info on how Windows 7 (and probably Server 2008 R2) deals with solid state drives.
• VMware Slaps Enterprise and Cisco In Face, Opens Door For Competitors – More on the VMware vSphere 4 licensing changes, and what it might mean.

Misc

• Non-competes are evil – Robin Harris lays it out. I happen to agree.
• Five Technologies Our Kids Won’t Even Recognize – Wired’s Gadget Lab makes us all feel old!
• Chrysler HQ Designed To Convert Into Shopping Mall - Wait Wait Don’t Tell Me says it’s true, so I believe it!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Back From the Pile: Interesting Content From the Week of May 9, 2009
This post was categorized as Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Enterprise IT

• Reduce, Reuse, and Recycle Your Reliance On Backup Tapes – What’s wrong with backup tapes? They’re inaccessible, making them unsuitable for most applications. My latest post for my Enterprise Storage Strategies blog.
• Is Licensing Turning vSphere Into Vista? – A group post for Gestalt IT suggesting that VMware’s tiered licensing strategy for the vSphere 4 family might cause anguish for customers
• Infosmack Episode 2 – EMC, David Donatelli and Non-Compete Agreements – I joined the StorageMonkeys Infosmack podcast to discuss EMC, Dave Donatelli, and con-compete agreements with Greg Knieriemen, Marc Farley of 3Par and StorageRap, and Tony Asaro of the INI Group
• IT utilities, the biggest game in town – Chris Mellor of The Register suggests that the biggest IT vendors are positioning to lock up the IT market.
• The Non-Compete Clause Debate….. - Steve Duplessie muses on the pros and cons of non-compete agreements and what they mean for the IT industry
• Who’s the pot & who’s the kettle? - W. Curtis Preston reacts to the skirmishes between Tony Asaro and the EMC bloggers
• The Only VMware-certified NAS in its Category: Why the Iomega StorCenter ix4-200r Will Resonate with Small Businesses - Jerome Wendt expands on the Iomega StorCenter Pro ix4-200r
• Don’t focus too much on costs, cloud computing is about business agility - CapGemini reminds us that cloud computing isn’t just about reducing costs
• What we talk about when we talk about cloud computing - The Google Enterprise blog lays down some of the hard facts about building a cloud on your own
• Review: Sun Storage 7000 Unified Storage System - Chris Evans provides one of the best reviews yet of Sun’s Amber Road open source-based storage systems
• Brocade Pokes Cisco in the Eye, Switches for IBM - A great headline and solid reporting by Om Malik on one of the first reactions to Cisco UCS
• TheInfoPro: Analysis or Marketing? - StorageMonkeys continues their probing of the storage analyst community
• A “Tweeterview” with Stephen Foskett, Nirvanix Consulting Director - Sunshine interviews me about Nirvanix, storage, and my new role

Apple

• Achieving Email Bliss with IMAP, Gmail, and Apple Mail – A deep, long post on GMail, IMAP, Mac OS X Mail, and how to get all three to play nicely together.
• Monitor Your Mac Remotely With iStat for iPhone - I love iStat for Mac OS X and the iPhone and had been meaning to write a review. This AppleBlog post is just about what I might have written!

Life

• There Goes Chrysler – NPR’s awesome Planet Money blog reported the breaking news of Chrysler’s bankruptcy
• The Venture Capital Math Problem and The Venture Capital Math Problem (continued) – Fred Wilson points out that there can be no such thing as unlimited venture capital
• Is it time to cut the Ethernet access cable? – Has pervasive wireless Ethernet made wired ports unnecessary?
• Interview with Micah Baldwin, Father of FollowFriday – Micah calls me out as someone who is doing interesting things on Twitter!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Back From the Pile: Interesting Content From the Week of May 2, 2009
This post was categorized as Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Driving Massive I/O

Chuck Hollis treated us to a discussion of vSphere as an I/O Engine on his blog this morning with some background on multipath IO (MPIO for short), but I’m not sure he did the topic justice. In my opinion, server virtualization is the greatest I/O driver ever brought into the data center, and it messes with all of our preconceived notions about I/O at the same time.

What’s so special about server virtualization?

1. Hypervisors concentrate I/O, shifting loads that were formerly distributed to a large number of I/O channels into a far fewer channels. Picture 10 servers doing what they do. Now put all 10 in a single physical box. All of their storage access must now share a bus, a host adapter, a cable, and perhaps a LUN on the storage system. It’s the difference between lemonade and lemon juice!
2. Hypervisors randomize I/O, chunking everything up and mixing it together. Forget about the carefully-designed read-ahead algorithms and caching used in enterprise storage – VMware, Hyper-V and the rest throw those expectations out the window! Virtualization is a blender – it grinds up your lemons, skin, seeds, and all!
3. Hypervisors demand low I/O latency, forcing infrastructure to get quicker, not just faster. This is one reason that caching, solid state disks, and 10 GbE are going to be huge in virtual environments – all reduce latency by orders of magnitude! As any car guy will tell you, quick and fast are two very different things!

The upshot of all of this is that virtual servers are very very hard to satisfy when it comes to I/O. And the “back end” has always been a bit of a bottleneck for virtualization software. Now we have VMware claiming that vSphere 4 can push over 300,000 I/O operations per second (IOPS) without resorting to VMDirectPath and similar “cheater” measures. Of course not all IOPS are equal, and I doubt that that 300k number would hold up with a real-world workload, but it’s impressive nonetheless!

A Brief History of MPIO

Let’s turn back to multipath I/O. PowerPath/VE is just the latest in a long line of path managers, not all of which have been well-loved. Back in my HP-UX days I learned to make the most of PVlinks, the native path management on that operating system. It wasn’t always easy to get it to work well, but it sure was nice to have a path manager built into the operating system! Veritas also offered a multi-platform path manager, DMP, which worked with a variety of array types. Back in the day, both were limited to simple failover and lacked the “intelligence” to deal with the peculiarities of the weird storage arrays we learned to not hate.

Array-specific path managers from storage vendors were much more successful. CLARiiONs used ATF, Hitachi arrays used HDLM, IBM had SDD, and of course EMC had PowerPath. EMC introduced PowerPath in 1997, the software reportedly having been developed by Conley Corporation, which EMC acquired the next year and turned into its Cambridge (MA) development center. After acquiring Data General, EMC adapted PowerPath to support CLARiiON, pushing ATF off stage right. Then they kept right on developing the software, adding support for IBM, HDS, and HP arrays and data migration.

Meanwhile, Microsoft decided that HP and Veritas were on to something when they developed standard path management software, so they began working on a standard multi-path IO (MPIO) driver for Windows. But Microsoft learned a thing or two from the mediocre device support in those old solutions, so they decided to allow vendors to plug their own smarts into the standard Windows Server 2000/2003 MPIO framework. Microsoft provided basic failover capability and third parties, including EMC, wrote their own device-specific modules (DSMs). This MPIO support evolved and spread, standard on Microsoft’s iSCSI initiator and Hyper-V virtualization platform. PowerPath 5.2.1 for Windows already supported Hyper-V thanks to this.

PowerPath and VMware PSA

VMware also learned a thing or two from HP and Microsoft. Although basic path failover support has been included in ESX for years, vSphere 4 takes it to a new level with pluggable storage architecture (PSA). Every version of ESX 4 includes native multipathing (NMP), but Enterprise Plus licensees can use vendor-supplied plugins to enable more advanced path management. As I noted on Tuesday, there are two three different levels of path selection: Basic path-selection plugins (PSPs), more advanced storage array type plugins (SATPs), and complete multi-path plugins (MPPs).

This is what EMC has introduced: An MPP for vSphere 4 called PowerPath/VE. Like the DSM for Windows MPIO, PowerPath/VE for vSphere slots right into an existing MPIO framework and enables advanced path selection and load balancing without mucking with the internals of the hypervisor. PowerPath/VE has all sorts of smarts in it. It has eight different predictive load balancing policies, proactive disconnect, bus testing, and HBA monitoring.

Super VMware guy Chad Sakac described PowerPath/VE as part of the launch. He notes that EMC is first out of the gate with a multipathing plugin for vSphere, but I suspect that just about every vendor will release similar functionality pretty quickly. In particular I expect support to come from NetApp and 3PAR, since they’re so interested in VMware support.

Licensing Questions

One thing really stuck out in the vSphere launch: PSA is only included in the top-of-the-line Enterprise Plus license. Presumably, this means that, in addition to paying for a PowerPath/VE license, users will have to spring for maximum ESX, too. This is a dumb move, if you ask me. Microsoft made MPIO successful by giving it away with every copy of Windows. They even included it in the free iSCSI initiator download. VMware, in contrast, seems to be actively limiting PSA’s usefulness to the top tier of users. If it was up to me, I would set the VMware MPIO free!

I’m working with EMC and VMware to determine the extent of the NMP/PSA/PowerPath licensing mess. I’ll update this post as I find out the answers!

1. Does every edition of ESX 4 include the basic VMware native multipathing (NMP)?
2. Can one use a vendor-supplied PSA plugin like PowerPath/VE without an enterprise plus license?
3. Does it matter (to licensing) if the plugin is a PSP or an SATP?
4. If “no” to 2 or 3, can PSA be added separately without the plus license if someone wants to use something like PowerPath/VE?

Update: I received a nice email from an EMC engineer correcting me about the plugin types. This kind of open communication is why the web is so great! It turns out that PowerPath/VE is a sort of super plugin called an MPP, not “just” an SATP or PSP. I’ve updated the section above!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | PowerPath To The Virtual People
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3
• Storage Changes in VMware ESX 3.5 Update 4
  

One first step is VMware’s whitepaper, “What’s New in VMware vSphere 4: Storage“.

Licensing and Availability of Features

The most important change at a glance is in the licensing model for VMware.

• A new low-end “standard” level includes all of the components a small business might need, including thin provisioning of storage, at a very attractive price.
• The “advanced” level includes advances in the area of data protection.
• At the “enterprise” level, live migration of storage is enabled.
• Top of the heap is “enterprise plus”, which enables the intriguing plug-in third-party multipathing support previously rumored.
• Enterprise customers with current support will get vStorage thin provisioning and data protection when they upgrade.

Each step up the licensing ladder enables important new storage features (source: VMware)

Thin Provisioning

One of the most widespread storage features is native thin provisioning. VMware ESX 4 will allocate storage in 1 MB chunks as capacity is used. This isn’t really completely new – similar support was enabled by default for virtual disks on NFS in VI 3, and thin provisioning could be enabled on the command line for block-based storage as well. It was also present in VMware’s desktop products, including my own copy of Fusion. And ESX allows thick-to-thin conversion during Storage VMotion.

The difference with vSphere 4 is that thin provisioning is fully supported and integrated into every version of ESX. Although many storage arrays now also offer thin storage, the addition of native, integrated thin provisioning right in ESX is huge. This alone will be a major capacity (and thus, cost) savings feature! VMware claims 50% storage savings in their lab tests.

Dynamic Expansion of VMFS Volumes

VMFS volumes can now grow (and, in some cases, shrink) online without resorting to spanning to a new LUN. Under vSphere 4, VMFS volumes can grow to take advantage of expanded LUNs (up to 2 TB per LUN). The old method still works as well, and multi-LUN spanned VMFS volumes can grow when any of their LUNs is expanded.

Enhanced Storage VMotion

Like thin provisioning, Storage VMotion has been elevated to first-class status, supported just about everywhere you’d want it. It’s in all the likely spots within vCenter.

Storage VMotion gives serious storage flexibility now, enabling (almost) any-to-any migration of VMFS volumes: Pick up a Fibre Channel, iSCSI, or NFS disk image and move it to another datastore running any of those protocols to convert live. And you can do thick-to-thin provisioning at the same time.

Under the hood, the whole infrastructure has been revised. Storage VMotion leverages VMware’s change block tracking instead of disk snapshots now, speeding up the migration process and reducing the (formerly excessive) memory and CPU requirements of Storage VMotion in 3.5. This is the same technology leveraged by vSphere’s High Availability features, by the way.

Pluggable Storage Architecture (PSA)

Only “Enterprise Plus” licensees will get to use it, but the vSphere family also sports a new pluggable storage architecture (PSA) which will initially be leveraged to deliver vendor-specific multipath support. Note that the native multipath support in vSphere continues to be a basic round-robin or fail-over system – it will not dynamically load balance I/O across multiple paths or make more intelligent decisions about which paths to use.

vSphere 4's Pluggable Storage Architecture allows third-party developers to replace ESX's storage I/O stack (source: VMware)

As you may gather from this VMware illustration (but would probably miss since it’s not all that comprehensible), there are two classes of third-party plug-ins:

1. Basic path-selection plugins (PSPs) will merely optimize the choice of which path to use, ideal for active/passive type arrays
2. Full storage array type plugins (SATPs) will allow load balancing across multiple paths in addition to path selection for active/active arrays

EMC also announced PowerPath/VE for vSphere, integrating their popular multi-platform path management software directly into ESX. It’s not clear at this point whether PowerPath will require an Enterprise Plus license (or if it will come with one) or if it will work with all editions, but I’m sure that will be clarified soon. My EMC contacts do tell me that PowerPath/VE is licensed on a per-socket basis (like VMware of yore) and that EMC sales reps have some room to get creative on licensing.

iSCSI Software Initiator Enhancements

It’s harder to claim it as a new feature, but the iSCSI software initiator has also been tweaked and tuned to use less CPU time and deliver better throughput. The iSCSI configuration process has also been smoothed out so one no longer needs to have a live Service Console connection in order to communicate with an iSCSI target. And changes made in the general tab are now global, so they’ll propagate down to each target. Bi-directional CHAP is also added, so the target can now be authenticated in addition to the initiator.

vSphere also includes a paravirtualized iSCSI driver (PVSCSI) which works like vmxnet to present a higher-performance iSCSI adapter within certain supported guest OSes.

More information on multipathing iSCSI can be found at Yellow Bricks and in Chad Sakac’s mega-post on the topic.

High-Performance I/O

vSphere really pours on the I/O power. They claim a 3x increase, to “over 300,000 I/O operations per second”, but mentioned in the launch that they’ve gotten to 400,000 in some workloads. This is really amazing, and if it’s true in real-world use means that VMware ESX can now host just about any application you can think of.

It’s funny, but just a few weeks before the announcement VMware’s Chief Data Center architect, Scott Davis, told me at Storage Networking World that vSphere would double I/O performance. I thought this was a bold statement at the time, but VMware went further in their official documents, claiming 3x improvement. Now we have an on-stage admission that it could be up to 4x! This is serious stuff, folks: Can VMware really do 400,000 IOPS?

One question is whether these IOPS improvements require the new VMDirectPath I/O for Storage, which binds a physical Fibre Channel HBA to a single guest OS, or if they’re generalized across all systems.

vCenter Improvements

vCenter now includes more storage information:

• Each VM and ESX in the inventory has a tab showing storage information and allowing users to set alarms on storage use – setting capacity alarms becomes extremely important when thin provisioning is used!
• vCenter inventory has a view showing datastore details
• There’s also a nice storage topology map

Data Protection APIs (Updated)

VMware also enhanced the APIs for consolidated backup (VCB) into “vStorage Data Protection”. Available only with Advanced or Enterprise licenses, these APIs allow enterprise backup vendors to develop specific integration with VMware without VCB in the middle.  All versions of ESX include this vStorage API and boast VCB support. Standard and Essentials licenses do not include vCenter Data Recovery, a standalone disk-to-disk backup capability for virtual machines.

Conclusion

vSphere is loaded with storage improvements, though many seem like incremental updates rather than new features. Admins will welcome thin provisioning support, and the full integration of Storage VMotion is certainly welcome, but I think the changes to vCenter will be most noticeable. I’ll be looking into EMC’s PowerPath/VE and the amazing I/O improvements as we move forward and learn more about the next ESX!

Duncan Epping posted much more vSphere Linkage on his Yellow Bricks blog if you’re interested in non-storage features. I also suggest reading Chris Evans’ outline of vSphere storage changes.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Changes in the VMware vSphere 4 Family
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.