Frankenweenie saves young Victor in Tim Burton's macabre short film

When is a solution integrated and when is it a Frankenstein-like mashup of tangled tech? Apparently, that line is crossed when it’s your competitor’s offering…

In my time in the storage industry, I’ve seen enough franken-storage come and go to make me skeptical whenever a new “integrated” solution is announced. But a lot of this stuff works just fine, so I also know that integrated solutions aren’t always bad!

The latest industry blog flame war centers around NetApp’s recently-announced solid state storage solution, which pairs a V-Series NAS head and a Texas Memory Systems RamSan-500 flash storage system. Perhaps NetApp’s Val Bercovici did get a bit over-excited in his post on the topic, but he wasn’t just talking about the RamSan: He was laying out how NetApp’s WAFL technology can work in an SSD world, and using some recent performance test numbers on that solution as well as their PAM cache cards as an illustration of this.

The next thing you know, we have EMC’s Storagezilla and IBM’s Barry Whyte calling the company out for what they (and others. like Storagebod) see as an underwhelming product offering. That’s all well and good, and I’ll let the reader decide if NetApp’s moves warranted a press release, but now things have gotten uglier…

EMC’s Chuck Hollis called the whole RamSan idea to account, saying it was “Frankenstorage“, causing NetApp’s Alex MacDonald to engage in a little “I know you are but what am I” in reference to EMC’s CLARiiON/Celerra “unified storage” solutions.

It’s time to bring some sanity to this whole integrated solution concept. Every product in the storage world is an amalgamation of OEM parts to one extent or another, and there are always integration issues. Certainly many of EMC’s offerings could be the subject of name-calling: They use STEC SSD drives in the DMX, they use Quantum deduplication engines in their CDLs, and their Celerra NS platform does include a complete Fibre Channel SAN behind the curtain. But they’re not alone, and not even wrong in doing this: Every vendor relies on OEMs, and as a wise man said, “working with an OEM gives you the flexibility to pick best of breed technologies” and that’s exactly what customers want. Any objective person would welcome qualification and integration of TMS’ RamSan with a solid platform like the NetApp V-Series – it’s a certifiable win for the customer. Just like they would be happy to see EMC leveraging great technology from Quantum and STEC.

Chuck goes on to point out some downsides to these OEM combinations, and they’re certainly fair criticisms:

• When you’re buying this from this guy and that from that guy, it’s bound to cost more because everyone needs their cut.
• Since all attempts at unified heterogeneous device management have failed, a combo is certainly harder to manage than a single device.
• With multiple vendors in the mix, fingerpointing is common once support is needed.

But these criticisms can be mitigated by the vendors themselves. They can give up some margin in order to gain market share. They can create unified management interfaces for the combinations they sell and support. And they can really support what they sell, refusing to give in to the temptation to say “not my problem” when the going gets rough. And companies deal with these problems all the time! Frankenstorage doesn’t have to be so scary…

This post can also be found on Gestalt IT: The Difference Between “Integration” and “Frankenstein”

© sfoskett for Stephen Foskett, Pack Rat, 2009. | The Difference Between “Integration” and “Frankenstein”
This post was categorized as Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What was he getting at? Read the comments in Chuck’s post and you’ll understand his implication: Chuck suggests that NetApp “emulates” Fibre Channel in their FAS/OnTap devices on top of the WAFL “file system”, while EMC’s AX storage (behind the NX4) uses “real” Fibre Channel, so it’s better. He goes on to say that EMC is doing a brisk business replacing misfit NetApp FC arrays with real FC kit from EMC. But, as is so often the case, the truth is a little more complex than this: All enterprise storage arrays “emulate” Fibre Channel drives to one extent or another, and using the wrong tool for the job will always lead to trouble.

Is It Real Or Is It Virtual?

Let’s knock one thing out right away: Chuck is right, NetApp is emulating Fibre Channel drives with their FAS arrays. They really do lay out chunks of storage on something kind of like a file system with a bunch of logic mixed in and then pretend that this space is a plain-jane SCSI drive connected over Fibre Channel. And I’ll add to the “scandal” by pointing out that NetApp does exactly the same thing with their iSCSI drives!

Now let’s move on to an even more important point: All modern storage arrays emulate SCSI drives! That’s right, every enterprise storage array is lying, pretending to serve up basic drives but really slicing and dicing them in the background for their own nefarious purposes!

Who is responsible for this deceit? I place the blame on a few: Patterson, Gibson, and Katz started the game with their so-called RAID concept, which kicked things off by allowing a few drives to pretend to be a single one. Data General implemented this with cache in their oh-so-clever HADA, further separating us from The True Disk. But the worst was EMC, with their fully-virtualized Symmetrix array, where there was no definite relationship at all between the LUNs presented to servers and the disks that do all the real work. Some folks would even go so far as to praise this type of post-RAID virtualized storage as innovative!

NetApp takes this “automated lying” to the extreme, forcing their innocent hardware to take honest, well-laid-out blocks of intelligent WAFL space and twist them into vast tracts of dumb pretend-disks. The nerve! Compellent, 3PAR, Dell/EqualLogic, and the rest are just as bad, scattering blocks of data willy-nilly across their disks in so-called “wide stripes“. But don’t let Chuck’s misdirection fool you: EMC is just as guilty with each of their different storage platforms, masquerading as disk drives or file servers and intelligently managing storage underneath! And don’t get me started on the twisted things VMware does to storage!

Modern? Feh! Let’s all hope Apple starts producing their no-feature Xserve RAID again!

Waiting On Angels

So every modern array emulates disks. What was Chuck’s point again? Oh yeah, that the AX Fibre Channel storage used by EMC’s NX4 is superior to the integrated Fibre Channel capability of the NetApp FAS2020! I’m sure he’s right for some use cases and wrong for others. FC on the FAS2020 is a perfect match for some, and the NX4/AX wins in a landslide in some circumstances.

The crux of the argument is the fact that NetApp does all sorts of stuff behind the scenes build and support an FC LUN that the EMC AX FC array doesn’t do. So, although it wouldn’t be fair to say that one was “emulated” and another was not, Chuck would be correct in saying that an FC LUN on an AX is more “real” than one on a NetApp FAS. But arguing over technicalities like this is all angels and pins and doesn’t matter in the real world!

What does matter? In block storage, latency is king. Generally speaking, more cogs and wheels leads to more latency. This is why storage arrays rely so much on large, intelligent caches and vendors are experimenting with all sorts of cool caching technology. But, ignoring cache, high-end arrays generally have worse latency than low-end ones because they have all sorts of translation and virtualization going on in the background. In any I/O situation, increased latency hurts throughput and the perception of performance. And there comes a point when block applications give up waiting and it’s “game over, man!”

I remember migrating from an old CLARiiON 3100 to a brand new Symmetrix 3930 and watching the Symmetrix choke on the incoming data stream. It just couldn’t write fast enough to handle full streaming reads from the (old-tech) CLARiiON. But once everything was migrated and running, the Symmetrix, with its massive (for the time) 16 GB of cache, widely-spaced data layout, and multiple internal channels, completely destroyed the CLARiiON in real-world performance. This pattern continues today, with devices like the DMX and USP offering much better real-world performance than benchmarks or theoretical techno-arguments would suggest.

So Which Is Better?

But Chuck and the rest were not talking about high-end stuff here. They are comparing the architecture of entry-level enterprise kit and drawing conclusions about which is best. I personally don’t care what the internals of the system look like. I care how well it works.

I have personally seen Microsoft Exchange running on low-end FC-connected NetApp FAS arrays, and it worked great. I also helped a customer migrate off of EMC AX that didn’t give them the performance they needed for their databases. In truth, lower-end gear is often over-sold and unable to deliver the performance, features, and reliability specified on data sheets and in vendor presentations.

That’s right, there’s more to this picture than raw performance. Consider manageability, for one. NetApp is offering a single-interface integrated system with all protocols (CIFS, NFS, iSCSI, and FC) available from one device. They also offer similar levels of integration for their (really nice) snapshot, replication, and deduplication technology. WAFL is busy doing a lot of great stuff, so I really wouldn’t be surprised if EMC’s less-integrated NX/AX offering beats them on performance at the same price point. Which is more important to you, integration, performance, or features? And I bet that, if you spent a bit more on a higher-end NetApp box, you could have it all.

On the flip side, EMC is offering a really compelling entry-enterprise combination at a nice price point. The latest NX should be on everyone’s NAS short list, and I’m sure the simple FC of the AX array would work well in a smallish Exchange, VMware, or SQL Server environment. It’s not as unified as NetApp’s offering management- or feature-wise, but it’s still pretty good.

Pick the right tool for the job, though. Neither the NX4 nor the FAS2020 is a good fit for a high-I/O application, and that’s a fact!

This post can also be found on Gestalt IT: Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It has been the core technology behind the storage industry since day one, but the sun is setting on traditional RAID technology. After two decades of refinement and fragmentation, we are abandoning the core concepts of disk-centric data protection as storage and servers go virtual. Next-generation storage products will feature refined and integrated capabilities based on pools of storage rather than combinations of disk drives, and we will all benefit from improved reliability and performance.

RAID Classic

Early storage systems were revolutionary, in physically removing storage from the CPU, in enabling sharing of storage between multiple CPUs, and especially in virtualizing disk drives using RAID. When Patterson, Gibson, and Katz proposed the creation of a redundant array of inexpensive disks (RAID) in 1987, they specified five numbered “levels”. Each level had its own features and benefits, but all centered on the idea that a static set of disk drives would be grouped together and presented to higher-level systems as a single drive. Storage devices, as a rule, mapped host data back to these integral disk sets, sometimes sharing a single RAID group among multiple “LUNs”, but never spreading data more broadly. Storage has remained stuck with small sets of drives ever since.

The core insight of the 1980s remains true: More spindles means better performance. Although additional overhead dulls the impact somewhat, the benefit of spreading data across multiple drives can be tremendous. A typical RAID set offers much better performance than the drives alone, and can handle a mechanical failure as a bonus.

Cracks are appearing in the RAID veneer, however. Double drive failures are much more common than one would expect, leading to the development of hot spare drives and dual-parity RAID 6. If four drives perform well, then forty drives perform much better, leading to the common practice of “stacking” one RAID set on others. Caches and specialized processors were introduced to overcome the performance issues related to parity calculation.

But traditional RAID cannot overcome today’s most critical storage issues. As drives have become larger, the tiny chance of an unrecoverable media error compounds, becoming a certainty. Even dual-parity will not be able to guarantee data protection on the massive disks predicted for the near future – statistics cannot be denied. The latest disks contain so much data, without commensurate improvements in throughput, that rebuild times have skyrocketed, resulting in hours or days of reduced data protection.

RAID is also ill-suited to the demands of virtualized systems, where predictable I/O patterns become fragmented. It cannot provide tiered storage or account for changing requirements over time. It cannot take advantage of the latest high-performance solid state storage technology. It cannot be used in cloud architectures, with massive numbers of small devices clustered together. It interferes with power-saving spin-down ideas. Most RAID implementations cannot even grow or shrink with the addition or removal of a disk. In short, traditional RAID cannot do what we now need storage to do.

RAID is Dead

Although most vendors still use the name, nearly every one has abandoned much of the classic RAID technology. EMC’s Symmetrix pioneered the idea of sub-disk RAID, pairing just a portion of each disk with others to reduce the impact of “hot spots”. HP’s AutoRAID added the ability to dynamically move data from one RAID type to another to balance performance. And NetApp paired disk management so closely with their filesystem that they were able to use RAID 4 and the flexibility it brings.

Today, a new generation of devices has even evolved beyond RAID’s concept of coherent disk sets. Compellent, Dell EqualLogic, 3PAR and others focus on blocks of data, moving portions of a LUN between RAID sets, disk drive types, and even inner or outer tracks based on access patterns. With these devices, a single LUN could encompass data on every drive in the storage array. And the latest clustered arrays can spread data across multiple storage nodes to scale performance and protection.

These innovative devices point the way to a future in which virtual storage is serviced and protected very differently than in the past. Perhaps software like Sun’s ZFS serves to illustrate this future best: It unifies storage as a single pool, intelligently protecting it and presenting flexible storage volumes to the operating system. Although Sun calls its data protection scheme “RAID-Z”, it has little in common with its namesake. Like NetApp’s WAFL, the copy-on-write ZFS filesystem is totally integrated with the layout of data on disk, allowing mobility and efficient use of storage. A single pool can include striping, single- or dual-parity, and mirroring, and disks can be added as needed. Importantly, ZFS also checksums all reads, detecting disk errors.

Long Live RAID

The post-RAID future will see these concepts spread across all enterprise storage devices. Disks will be pooled rather than segregated into RAID sets. Tight integration between layout and data protection will allow for much greater flexibility, integrating tiering and differing data protection strategies in a unified whole. Storage virtualization will allow mobility of data within these future storage arrays, and clustering will enable massive scalability.

Two things will likely remain to remind us of Patterson, Gibson, and Katz, however. First, the core principle that multiple drives working as one yields dividends in terms of performance and data protection. And second, that whatever we use should be called RAID, even though the definition of that term has changed beyond recognition in the last two decades.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Turning the Page on RAID
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.