Thursday, September 15

One of the highlights of VMworld with the debut of Nutanix. They received a lot of attention, as well as a “Best of VMworld” award, for their scale out “compute cluster”, which incorporates servers storage into a blade form factor. We’re very pleased to have the fine folks from Nutanix joining us at Tech Field Day as well! Tune in at 8:00 AM Pacific time on Thursday to learn more.

A company I’m very glad to see at Tech Field Day is Nasuni, which is much more than a cloud storage gateway. I’ve been working with Nasuni for almost a year now, and am very familiar with their offerings and technology. But I’m betting the delegates might not know what they’re up to. And I’m very pleased to let you know that Nasuni CEO Andres Rodriguez will be covering some excellent new information during their 10:30 AM Thursday session!

Symantec is perhaps the most-frequent Tech Field Day presenter, but this is a break from the past Backup Exec- and NetBackup-focused sessions. This time we’ve got the Symantec Storage folks (i.e. Veritas) and they’re spilling the beans on the long-awaited version 6.0 of Storage Foundation. I’m a big fan of this software, having used it for over a decade, and am looking forward to hearing what’s new at 1:30 PM.

While touring the show floor at the NAB show earlier this year, I ran into DataDirect Networks, probably the biggest and most successful storage company most people have never heard of! DDN sells tons of storage in the media and entertainment business (explaining their focus on the NAB show), but they have a lot of great products that are gaining attention in other markets as well. We are very pleased that DDN will give us all a peek at what they are working on at 4 PM Thursday!

Party With The ExecEvent

Every Tech Field Day event has featured a Thursday-evening party so the presenting companies and invited delegates can get to know each other better. In the past, we’ve visited venues as diverse as the Computer History Museum, The Museum of Flight, and Fenway Park, and have invited friends and locals to join us. This time we’re doing something different again.

We coordinated the date of Tech Field Day with our friends over at The ExecEvent so we could throw a joint Thursday evening bash. Joining our presenters and delegates will be the representatives of various companies who are meeting in Santa Clara for that event. This gives everyone involved another reason to attend the party, and will likely lead to many interesting industry discussions.

Friday, September 16

I’ve previously written about the all-flash enterprise arrays announced by Pure Storage at VMworld, where they got lots of attention. Now it’s time for the Tech Field Day delegates to take a closer look at their technology. Join in Friday morning at 8 AM Pacific to hear what they’ve got to say.

Arista Networks can be considered a “friend of the event” after presenting at Networking Field Day last year, and more so since they are allowing us to use their facility for the rest of the day Friday. We’ll start off with Arista themselves at 10:30 Pacifc. They’ll undoubtedly cover their high-performance Ethernet switches with a twist: VMware integration (another “Best of VMworld” winner) and the ability to run certain applications right in the chassis. Nifty stuff.

SolidFire is generating tremendous buzz in the industry with their all SSD storage solution. Targeted at storage service providers, the SolidFire storage appliances deliver massive performance and efficiency as well as automated management. We are thrilled that SolidFire will be able to demonstrate their product to the Tech Field Day delegates at 1:30 PM Friday.

We’ll wrap up with a backup industry veteran, Arkeia Software. Although perhaps not the most well-known name, Arkeia is staking a claim on distributed data protection leveraging a special twist on data deduplication and even offering a credible solution for cloud servers. Tune in at 4 PM Friday to learn more!

Stephen’s Stance

Tech Field Day 8 really is the most diverse and startup-laden event we’ve ever hosted. I am personally very excited to see every presentation, and invite you to join in live on Thursday and Friday between 8 and 6 Pacific. Live streaming video will be at TechFieldDay.com!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Tech Field Day 8 Presenter Lineup
This post was categorized as Enterprise storage, Everything, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Thin provisioning needs communication to function, and zero page reclaim is only the array side of the story. WRITE_SAME helps reduce I/O load, but the server needs to use it. Wouldn’t it be nice if the operating system, file system, or volume manager would use these commands to help recover capacity?

We need a bridge. We need to cross the gulf between the storage array (that expects all these zeros or some other way to recover this space), and the host (who has this information)

That’s the thing that gets me riled up. The application and the file system know that this data is no longer used. If there was only some way, some bridge, that allowed them to talk to the storage array and say, “Hey, you can get rid of that now. I don’t need it anymore.” See? We need a bridge.

Well, guess what? We have some bridges. One of my favorite developments in storage of the last year is this Thin Reclamation API that Symantec has. I really like what they’re doing.

Symantec (formerly Veritas) have the file system and the volume manager. If you’re an enterprise guy like me (I was using HP-UX in 1996) you have used the Veritas File System and Veritas Volume Manager before.

Symantec calls all this Veritas Storage Foundation now. It has the information thin provisioning storage arrays need, and they’ve come up with a way to communicate: Being a software company, Symantec implemented just about every communication method, because not everybody uses this WRITE_SAME, and not everybody uses zero page reclaim. Some of them have their own API for recovery. So Symantec tried to implement as many thin methods as they possibly could right in the file system and volume manager.

I’ve talked to a lot of people who are using it, and they say that it works. They say that it’s brilliant. Because you turn on this feature and, suddenly, the file system starts telling the array “I don’t need this data anymore.” Suddenly one of my major complaints about thin provisioning goes away.

But I have a question: Why doesn’t everybody do this? Why isn’t this in the Linux file systems and volume managers? Why isn’t Microsoft working on this? Why doesn’t everybody have this capability? It’s not rocket science. It just takes a little bit of development effort to try to come up with some kind of way to have the file systems talk.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Bridge: Veritas Thin (Provisioning) API
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What is Multipath? And Why?

The good old days: One device, one path

In the beginning, computers connected to peripherals and other computers through a single bus or channel and life was easy. Although one might mistake the names of the dominant printer connection (parallel) for some kind of multipath system when compared to the modem connection (serial), this was not the case. Only the bits traveled in parallel – the logical connection was a simple single path.

Early-90's servers might confuse admins with two SCSI connections to a single device

Then things got complicated. The SCSI protocol allowed for multiple devices in a chain, and even for two different “initiators” (computers or controllers) to interact with these “targets”. Some folks even dual-attached devices to a single computer with multiple controllers.

Why would one device and one computer need more than one connection? It boils down to two factors:

1. Performance – I/O channels have typically been slower than the computer could handle, so multiple channels might be used to increase the amount of data that can flow in and out.
2. Reliability – If one connection failed, the other might still be usable, reducing the risk of an outage.

Late-90's enterprise systems might have four or more paths to a single storage array

Pretty soon, enterprise computer architecture had gotten incredibly complex. I remember connecting a massive HP V-class server to an EMC Symmetrix with eight separate Fibre Channel cables. Each disk “LUN” showed up twice, and we had hundreds of them. We managed all of these virtual storage paths using HP’s PVLinks dynamic multipathing software. We used Veritas DMP and EMC PowerPath to do pretty much the same thing on Solaris and other UNIX systems.

Active/Passive to Active/Active

The earliest path management software provided two incredibly important functions: It figured out which of the SCSI targets it saw were actually different names for the same one, and it allowed the operating system to choose one and fail over to the other in case of an interruption. These were Active/Passive links – no matter how many paths were presented (and Fibre Channel switches sometimes presented eight or more), only one was active at any one time.

Modern systems have abstracted and virtual I/O channels, making path management much more important

But the EMC Symmetrix and similar high-end storage systems changed all this. Symmetrix storage was fully virtualized – the presentation of LUNs to servers was entirely disconnected from the actual disks and RAID sets in the array. This meant the Symmetrix could handle I/O requests across different paths and controllers for the same LUN. EMC and the rest responded with Active/Active path management software, allowing I/O to travel in parallel for the first time.

How is Dual Active Different?

Not everything called Active/Active is created equal. In fact, many supposed Active/Active setups really shouldn’t be called that since they don’t use both paths for all data. Instead, I like to call these Dual Active – both paths are active but with different data.

Consider the differences between the following two solutions:

A true active/active setup uses all paths for all data all the time

A "dual active" setup uses both paths, but each target is directed to one or the other

See the difference? Although the paths are active in both cases, they are not the same. Both approaches have merit, and neither is inherently superior, but they should have different names applied. Even active/passive has its place, since simplicity is often a virtue.

Dual Active Outside Storage

These same concepts apply outside the field of storage and I/O. Many server clustering systems use the same terminology, right down to the misapplication of “active/active” when “dual active” is more appropriate. It’s easy to miss the significance of this difference, but it can make more of an impact in clustering since CPU workloads are harder to balance.

Let me know what you think. If there is interest, I might dive into path management strategies like round robin!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Multipath: Active/Passive, Dual Active, and Active/Active
This post was categorized as Computer History, Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Driving Massive I/O

Chuck Hollis treated us to a discussion of vSphere as an I/O Engine on his blog this morning with some background on multipath IO (MPIO for short), but I’m not sure he did the topic justice. In my opinion, server virtualization is the greatest I/O driver ever brought into the data center, and it messes with all of our preconceived notions about I/O at the same time.

What’s so special about server virtualization?

1. Hypervisors concentrate I/O, shifting loads that were formerly distributed to a large number of I/O channels into a far fewer channels. Picture 10 servers doing what they do. Now put all 10 in a single physical box. All of their storage access must now share a bus, a host adapter, a cable, and perhaps a LUN on the storage system. It’s the difference between lemonade and lemon juice!
2. Hypervisors randomize I/O, chunking everything up and mixing it together. Forget about the carefully-designed read-ahead algorithms and caching used in enterprise storage – VMware, Hyper-V and the rest throw those expectations out the window! Virtualization is a blender – it grinds up your lemons, skin, seeds, and all!
3. Hypervisors demand low I/O latency, forcing infrastructure to get quicker, not just faster. This is one reason that caching, solid state disks, and 10 GbE are going to be huge in virtual environments – all reduce latency by orders of magnitude! As any car guy will tell you, quick and fast are two very different things!

The upshot of all of this is that virtual servers are very very hard to satisfy when it comes to I/O. And the “back end” has always been a bit of a bottleneck for virtualization software. Now we have VMware claiming that vSphere 4 can push over 300,000 I/O operations per second (IOPS) without resorting to VMDirectPath and similar “cheater” measures. Of course not all IOPS are equal, and I doubt that that 300k number would hold up with a real-world workload, but it’s impressive nonetheless!

A Brief History of MPIO

Let’s turn back to multipath I/O. PowerPath/VE is just the latest in a long line of path managers, not all of which have been well-loved. Back in my HP-UX days I learned to make the most of PVlinks, the native path management on that operating system. It wasn’t always easy to get it to work well, but it sure was nice to have a path manager built into the operating system! Veritas also offered a multi-platform path manager, DMP, which worked with a variety of array types. Back in the day, both were limited to simple failover and lacked the “intelligence” to deal with the peculiarities of the weird storage arrays we learned to not hate.

Array-specific path managers from storage vendors were much more successful. CLARiiONs used ATF, Hitachi arrays used HDLM, IBM had SDD, and of course EMC had PowerPath. EMC introduced PowerPath in 1997, the software reportedly having been developed by Conley Corporation, which EMC acquired the next year and turned into its Cambridge (MA) development center. After acquiring Data General, EMC adapted PowerPath to support CLARiiON, pushing ATF off stage right. Then they kept right on developing the software, adding support for IBM, HDS, and HP arrays and data migration.

Meanwhile, Microsoft decided that HP and Veritas were on to something when they developed standard path management software, so they began working on a standard multi-path IO (MPIO) driver for Windows. But Microsoft learned a thing or two from the mediocre device support in those old solutions, so they decided to allow vendors to plug their own smarts into the standard Windows Server 2000/2003 MPIO framework. Microsoft provided basic failover capability and third parties, including EMC, wrote their own device-specific modules (DSMs). This MPIO support evolved and spread, standard on Microsoft’s iSCSI initiator and Hyper-V virtualization platform. PowerPath 5.2.1 for Windows already supported Hyper-V thanks to this.

PowerPath and VMware PSA

VMware also learned a thing or two from HP and Microsoft. Although basic path failover support has been included in ESX for years, vSphere 4 takes it to a new level with pluggable storage architecture (PSA). Every version of ESX 4 includes native multipathing (NMP), but Enterprise Plus licensees can use vendor-supplied plugins to enable more advanced path management. As I noted on Tuesday, there are two three different levels of path selection: Basic path-selection plugins (PSPs), more advanced storage array type plugins (SATPs), and complete multi-path plugins (MPPs).

This is what EMC has introduced: An MPP for vSphere 4 called PowerPath/VE. Like the DSM for Windows MPIO, PowerPath/VE for vSphere slots right into an existing MPIO framework and enables advanced path selection and load balancing without mucking with the internals of the hypervisor. PowerPath/VE has all sorts of smarts in it. It has eight different predictive load balancing policies, proactive disconnect, bus testing, and HBA monitoring.

Super VMware guy Chad Sakac described PowerPath/VE as part of the launch. He notes that EMC is first out of the gate with a multipathing plugin for vSphere, but I suspect that just about every vendor will release similar functionality pretty quickly. In particular I expect support to come from NetApp and 3PAR, since they’re so interested in VMware support.

Licensing Questions

One thing really stuck out in the vSphere launch: PSA is only included in the top-of-the-line Enterprise Plus license. Presumably, this means that, in addition to paying for a PowerPath/VE license, users will have to spring for maximum ESX, too. This is a dumb move, if you ask me. Microsoft made MPIO successful by giving it away with every copy of Windows. They even included it in the free iSCSI initiator download. VMware, in contrast, seems to be actively limiting PSA’s usefulness to the top tier of users. If it was up to me, I would set the VMware MPIO free!

I’m working with EMC and VMware to determine the extent of the NMP/PSA/PowerPath licensing mess. I’ll update this post as I find out the answers!

1. Does every edition of ESX 4 include the basic VMware native multipathing (NMP)?
2. Can one use a vendor-supplied PSA plugin like PowerPath/VE without an enterprise plus license?
3. Does it matter (to licensing) if the plugin is a PSP or an SATP?
4. If “no” to 2 or 3, can PSA be added separately without the plus license if someone wants to use something like PowerPath/VE?

Update: I received a nice email from an EMC engineer correcting me about the plugin types. This kind of open communication is why the web is so great! It turns out that PowerPath/VE is a sort of super plugin called an MPP, not “just” an SATP or PSP. I’ve updated the section above!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | PowerPath To The Virtual People
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Update: More details have come out about Symantec’s new capability. Read my follow-up post on Symantec’s Thin API for clarification!

Here’s the gist (as near as I can tell). Veritas Storage Foundation will include the following capabilities:

1. Native thin provisioning which will cause Veritas volume manager to allocate space in the Veritas filesystem only when data is written
2. A thin provisioning API which allows the volume manager to communicate with certain storage arrays (3PAR is highlighted, along with HDS and HP) to allocate and reclaim space as it is used in the filesystem
3. A thin migration API which allows the volume manager to tell these storage arrays to migrate existing volumes to thin-provisioned space

This is great stuff, and ought to allow folks to make better use of thin provisioning. In fact, it nicely addresses many of my concerns about thin provisioning, including un-provisioning. It’s especially nice to see that Symantec already has interested partners, and that one system (3PAR) is up and running already. Way to go!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Symantec’s Thin API Is A Step In The Right Direction
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I’m kicking off with one of my favorite topics, logical volume management on servers. This is the longest-standing and most-successful face of storage virtualization, and the one I was first exposed to. Put simply, volume managers abstract block storage (LUNs, disks, partitions, what have you) into virtual “volumes” for use by the server. They look the same to the OS, and still need a filesystem, but are much more flexible, as we’ll see.

Volume managers are very common today – all modern OSes (except for that one from Apple!) have volume managers built in. Windows has the Logical Disk Manager, which I’m told was co-engineered (or something) with Veritas way back when and which I’ve covered in my Storage Magazine columns. Linux has an implementation of LVM, which I wrote something about way back when, and which has now not been supplanted by EVMS as had once been supposed. AIX it’s own twist on the original LVM, as does HP-UX. Solaris has the variously-named Solstice DiskSuite/Volume Manager which has evolved substantially in the 15 or so years I’ve been using it. And everyone has Symantec’s Veritas Volume Manager/Foundation Suite, which we in “the biz” view with considerable admiration and some skepticism, as is the case with all good front runners!

Folks mostly use volume managers for flexibility. It’s really quite amazing what you can do when your servers run them, enough that you often wonder how you got along without them!

• You can resize volumes (aka file systems or drives) on the fly (if your file system supports this as most modern ones do)
• You can protect data with RAID, even if your storage doesn’t support it (think bare disk drives)
• You can add storage capacity on the fly by concatenating new to old or (maybe) expanding existing stripes and RAID sets
• You can mirror volumes, create snapshots, and even replicate data to remote locations (this functionality varies by product, of course)
• One of the most powerful things (to me) was the ability to migrate live volumes from one storage device to another when making infrastructure changes

In short, volume management = server-based storage virtualization! So even if you were skeptical about the claims about storage virtualization, you might already be using it! Amazingly, a good volume manager can do anything a storage virtualization appliance or enterprise storage array can do. In fact, some virtualization appliances have more than a little volume management source code in them…

And the only cost for all this great stuff is the impact on your server’s CPU, memory, bus, access control, etc…

The chart below compares the major volume managers, and includes a little easter egg at the bottom… But we’ll cover that on another day.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Volume Management: Virtualizing Host Storage
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I for one was surprised at the small impact EMC has had on VMware. One systems administrator I talked to enthusiastically pointed out how great it was that big bad old EMC didn’t ruin VMware, but it cuts both ways. EMC has tremendous knowledge of the realities of managing storage in the enterprise, yet this hasn’t much rubbed off on VMware. After all, last I heard, they were still telling users to provision a single LUN for multiple virtual servers, and mixing OS and application data on the virtual C: drive…

Maybe we’ll see a response from EMC. EMC added volume management to PowerPath 4.0 back in 2003, and rumor had it they were developing their own full-featured multi-platform alternative to Veritas Volume Manager/Foundation Suite. Could this be headed into the VMware codebase? That would be a great way to gain some traction outside the storage space!

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Storage Management Integrated with Server Virtualization (Where’s EMC?)
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

H3C already has a long list of products, most based on in-house hardware and OEM software. On the storage side, the company makes an iSCSI storage array platform dubbed “Neocean”. This storage platform, selling strongly in China, is alleged to leverage technology licensed from FalconStor (on the low-end IX1000), Intransa (on the bigger IX5000), as well as iVivity and Xyratex. OEM storage developer Ciprico today announced that it will be working with H3C on the next generation. H3C also sells a WAFS accelerator leveraging Expand Networks software. All of these should be coming to the United States next year.

Huawei itself is also getting back into the storage market in the form of a joint venture with Symantec, creatively called Huawei-Symantec. This company is set to be coming out with a line of network devices with Veritas-based software built in. We’re hearing about virus scanning and content indexing appliances, as well as NAS and SAN arrays which will include storage foundation software from Symantec right out of the box.

Who knows what’s next from Huawei? I’d guess expanded services, more resellers in the West, and more OEM deals to create bigger systems. In a few years, they might give Hitachi and EMC trouble in the enterprise market, especially when big server vendors like Sun, SGI, Dell, and HP start rethinking their OEM strategies…

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Storage from behind the great wall
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.