The Four Horsemen of Storage System Performance: These four ugly gentlemen stand between you and your data.

Why do some data storage solutions perform better than others? What tradeoffs are made for economy and how do they affect the system as a whole? These questions can be puzzling, but there are core truths that are difficult to avoid. Mechanical disk drives can only move a certain amount of data. RAM caching can improve performance, but only until it runs out. I/O channels can be overwhelmed with data. And above all, a system must be smart to maximize the potential of these components. These are the four horsemen of storage system performance, and they cannot be denied.

Overcoming the Limits of Spindles

Perhaps the previous discussion of spindles left you exhausted, imagining a spindly-legged centipede of a storage system, trying and failing to run on stilts. The Rule of Spindles would be the end of the story were it not for the second horseman: Cache. He stands in front of the spindles, quickly dispatching requests using solid state memory rather than spinning disks. Cache also acts as a buffer, allowing writes to queue up without forcing the requesters to wait in line.

Cache may be quick, but practical concerns limit its effectiveness. Solid state memory is available in many types, but all are far more expensive per gigabyte than magnetic hard disk media. DRAM has historically cost 400 times as much as disk capacity, and even NAND flash (the current darling of the industry) is more than 40 times as expensive. Practically speaking, this means that disk devices, from the drives themselves to large enterprise storage arrays, usually include a very small amount of cache relative to their total capacity.

When specifying a storage system, the mathematics of cache and spindles adhere to a simple rule: More is better for performance but worse for the budget. This leads to a trade-off, where a point of diminishing return tells us to stop adding both spindles and cache and accepting the storage system as it is.

A History of Cache

Cache was not always as common as it is today. When even a small amount of DRAM cost hundreds of dollars, adding a single RAM chip to a hard disk drive would have broken the bank. So many drives had no cache at all well into the mid 1990′s. Operating systems of the time used expensive system memory as a buffer for storage operations rather than expecting cache in the disk controller or drive – remember setting the Buffers command in config.sys?

This was not as bad as it seems, at least in theory. Operating systems stand a fighting chance of “knowing” what data will be requested next, and could therefore request it ahead of time. They also might get a hint about data that will never be used again and can thus flush that from the so-called buffer cache. Although MS-DOS wasn’t very good at this, modern systems have greatly advanced in this respect using a technology called demand paging.

Caching at the array was the key differentiator for early enterprise RAID systems, overcoming the punishing slowdowns caused by parity calculations when data was written. EMC adapted their DRAM-based solid-state storage systems to become a cache in front of hard disk drives and the Symmetrix was born. The Data General (now EMC) CLARiiON was notable as well, bringing a large intelligent write cache to the vast market of midrange systems that could never justify the high price of a Symmetrix. Today, all vendors, from IBM to HP to NetApp to HDS, have vast and clever caches.

The importance of cache on enterprise storage performance can not be over-stated. Mix together rotational latency, seek time, and RAID penalty and you get seriously-compromised I/O response time. But cache can eliminate this penalty entirely, provided there is capacity, by confirming the write and queueing it for later (a concept known as write-back caching). Busy shared storage systems would be simply unusable without cache.

Five Uses for Disk Buffers

Hard disk drives today normally contain a small amount of RAM to use as a buffer for I/O requests. This serves the following needs, though not all are found on all drives:

1. A read cache, allowing frequently-requested data to be read from memory rather than involving mechanical disk operations
2. An I/O-matching mechanism, allowing slower disks and faster interfaces to work together
3. A read-around (ahead or behind) pre-fetch cache, saving a few blocks around any requested read on the assumption that they will also be requested soon
4. A read-after-write cache, saving recently-written data to serve later read requests
5. A command queue, allowing write commands to be reordered, avoiding the “elevator seeking” common to early hard disk drives

Disk buffer size has expanded rapidly in recent years, with some devices including 64 MB or more or DRAM. Seagate’s Momentus XT drive even includes 4 GB of NAND flash as a massive read cache!

Write-Through and Write-Back Cache

There are two basic methods of caching data:
The earliest systems used read-only or write-through caches. All I/O requests pass through the cache, which usually saves the most recent and serves them up when a read is requested. They don’t buffer write requests at all, simply passing them through to the storage system to process. They are safe, since the storage device always has a consistent set of committed writes, but they do nothing to offset the RAID penalty.	Most modern storage systems use a write-back (also called “write-behind”) cache, which acknowledges writes before they are committed to disk. They use non-volatile RAM, battery-backed DRAM, or NAND flash to ensure that data is not lost in the event of a power outage. Though far more effective, this type of memory is also far more costly.

Just about every modern storage array uses caching, and most employ the write-back method to accelerate writes as well as reads. Some have very smart controllers that perform other tricks, but Smart is another Horseman for another day. As mentioned before, RAID systems would be nearly unusable without write-back cache allowing the disks to catch up with random writes.

Onward: I/O, and Smarts

The horseman of spindles is harsh, but he does not rule the day. There are many ways to overcome his limits and his three brothers often come into play. These are cache, which bypasses the spindle altogether; I/O, which can constrain even the fastest combination of disk and cache; and the intelligence of the whole system, which limits or accelerates all the rest. We will examine these horsemen in the future!

I’ve been meaning to write this up for a long time. Thanks for listening and commenting!

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | The Four Horsemen of Storage System Performance: Never Enough Cache
This post was categorized as Computer History, Enterprise storage, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What was he getting at? Read the comments in Chuck’s post and you’ll understand his implication: Chuck suggests that NetApp “emulates” Fibre Channel in their FAS/OnTap devices on top of the WAFL “file system”, while EMC’s AX storage (behind the NX4) uses “real” Fibre Channel, so it’s better. He goes on to say that EMC is doing a brisk business replacing misfit NetApp FC arrays with real FC kit from EMC. But, as is so often the case, the truth is a little more complex than this: All enterprise storage arrays “emulate” Fibre Channel drives to one extent or another, and using the wrong tool for the job will always lead to trouble.

Is It Real Or Is It Virtual?

Let’s knock one thing out right away: Chuck is right, NetApp is emulating Fibre Channel drives with their FAS arrays. They really do lay out chunks of storage on something kind of like a file system with a bunch of logic mixed in and then pretend that this space is a plain-jane SCSI drive connected over Fibre Channel. And I’ll add to the “scandal” by pointing out that NetApp does exactly the same thing with their iSCSI drives!

Now let’s move on to an even more important point: All modern storage arrays emulate SCSI drives! That’s right, every enterprise storage array is lying, pretending to serve up basic drives but really slicing and dicing them in the background for their own nefarious purposes!

Who is responsible for this deceit? I place the blame on a few: Patterson, Gibson, and Katz started the game with their so-called RAID concept, which kicked things off by allowing a few drives to pretend to be a single one. Data General implemented this with cache in their oh-so-clever HADA, further separating us from The True Disk. But the worst was EMC, with their fully-virtualized Symmetrix array, where there was no definite relationship at all between the LUNs presented to servers and the disks that do all the real work. Some folks would even go so far as to praise this type of post-RAID virtualized storage as innovative!

NetApp takes this “automated lying” to the extreme, forcing their innocent hardware to take honest, well-laid-out blocks of intelligent WAFL space and twist them into vast tracts of dumb pretend-disks. The nerve! Compellent, 3PAR, Dell/EqualLogic, and the rest are just as bad, scattering blocks of data willy-nilly across their disks in so-called “wide stripes“. But don’t let Chuck’s misdirection fool you: EMC is just as guilty with each of their different storage platforms, masquerading as disk drives or file servers and intelligently managing storage underneath! And don’t get me started on the twisted things VMware does to storage!

Modern? Feh! Let’s all hope Apple starts producing their no-feature Xserve RAID again!

Waiting On Angels

So every modern array emulates disks. What was Chuck’s point again? Oh yeah, that the AX Fibre Channel storage used by EMC’s NX4 is superior to the integrated Fibre Channel capability of the NetApp FAS2020! I’m sure he’s right for some use cases and wrong for others. FC on the FAS2020 is a perfect match for some, and the NX4/AX wins in a landslide in some circumstances.

The crux of the argument is the fact that NetApp does all sorts of stuff behind the scenes build and support an FC LUN that the EMC AX FC array doesn’t do. So, although it wouldn’t be fair to say that one was “emulated” and another was not, Chuck would be correct in saying that an FC LUN on an AX is more “real” than one on a NetApp FAS. But arguing over technicalities like this is all angels and pins and doesn’t matter in the real world!

What does matter? In block storage, latency is king. Generally speaking, more cogs and wheels leads to more latency. This is why storage arrays rely so much on large, intelligent caches and vendors are experimenting with all sorts of cool caching technology. But, ignoring cache, high-end arrays generally have worse latency than low-end ones because they have all sorts of translation and virtualization going on in the background. In any I/O situation, increased latency hurts throughput and the perception of performance. And there comes a point when block applications give up waiting and it’s “game over, man!”

I remember migrating from an old CLARiiON 3100 to a brand new Symmetrix 3930 and watching the Symmetrix choke on the incoming data stream. It just couldn’t write fast enough to handle full streaming reads from the (old-tech) CLARiiON. But once everything was migrated and running, the Symmetrix, with its massive (for the time) 16 GB of cache, widely-spaced data layout, and multiple internal channels, completely destroyed the CLARiiON in real-world performance. This pattern continues today, with devices like the DMX and USP offering much better real-world performance than benchmarks or theoretical techno-arguments would suggest.

So Which Is Better?

But Chuck and the rest were not talking about high-end stuff here. They are comparing the architecture of entry-level enterprise kit and drawing conclusions about which is best. I personally don’t care what the internals of the system look like. I care how well it works.

I have personally seen Microsoft Exchange running on low-end FC-connected NetApp FAS arrays, and it worked great. I also helped a customer migrate off of EMC AX that didn’t give them the performance they needed for their databases. In truth, lower-end gear is often over-sold and unable to deliver the performance, features, and reliability specified on data sheets and in vendor presentations.

That’s right, there’s more to this picture than raw performance. Consider manageability, for one. NetApp is offering a single-interface integrated system with all protocols (CIFS, NFS, iSCSI, and FC) available from one device. They also offer similar levels of integration for their (really nice) snapshot, replication, and deduplication technology. WAFL is busy doing a lot of great stuff, so I really wouldn’t be surprised if EMC’s less-integrated NX/AX offering beats them on performance at the same price point. Which is more important to you, integration, performance, or features? And I bet that, if you spent a bit more on a higher-end NetApp box, you could have it all.

On the flip side, EMC is offering a really compelling entry-enterprise combination at a nice price point. The latest NX should be on everyone’s NAS short list, and I’m sure the simple FC of the AX array would work well in a smallish Exchange, VMware, or SQL Server environment. It’s not as unified as NetApp’s offering management- or feature-wise, but it’s still pretty good.

Pick the right tool for the job, though. Neither the NX4 nor the FAS2020 is a good fit for a high-I/O application, and that’s a fact!

This post can also be found on Gestalt IT: Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Of Emulated Fibre Channel, Virtualization, And The Right Tool For The Job
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

On the contrary, I suggest that the enterprise storage world we know was shaped by singular innovative products of the past. Without these, the IT world might look very different.

While you’re at it, check out my list of the coolest enterprise storage flops!

So let’s take a walk through history, identifying the ten most innovative and important enterprise storage hardware products. But let me note first that this list could be 100 items long, and we all have our favorites. Lots of the storage blogging world contributed their ideas, too!  So I’m setting some arbitrary rules to keep myself on track:

1. The products have to be reflected in the shape of modern enterprise storage for open systems. If their day came and went, they’re not listed here.
2. Hardware only – I’m working on another list for software!
3. Listed products have to have been successful in the market (or they’d probably fail rule number 1 anyway), so I’ll put together a list of cool flops later.
4. For product families, I tried to pick the most influential and innovative 2member.
5. This is an enterprise storage list – items have to be used in big companies, not little PCs.
6. Only one product per company (sorry, IBM and EMC!)

So without further ado, on with the list!

1964 IBM 2311

Let’s get one thing straight: IBM is responsible for much of modern enterprise computing, so it’s no surprise that when it comes to storage, they developed just about everything, including disk drives, floppies, tapes, arrays, and libraries. So why pick this particular piece of kit? Prior to 1964, computer components were developed as a set (the 350 storage system went with the 305 RAMAC, for example), but the 2311 changed that. It was a generic storage device, plug-compatible with a range of computers. If you claimed that the 2311 was the common ancestor of all modern enterprise storage, I would certainly agree with you!

Other notable IBM products worth mentioning:

• The original 1956 RAMAC 350 disk drive unit
• The 1961 301, which used a separate arm and head for each platter (like every disk drive produced since)
• 9-track tape, which dominated from 1964 until the 1/2-inch tape revolution in the 1990s
• The 1970 3330, which added error correction and remained in production for 13 years
• The 1971 introduction of the 23FD floppy disk drive
• The 1973 3430, whose “30/30″ code name caused people to refer to hard drives as “Winchesters”
• 1974′s 3830 “MSS” tape library
• The 1980 3380, which introduced film-head technology
• 2003′s SAN Volume Controller, the first successful SAN virtualization product (after many others failed!)

1976 Shugart SA-400 minifloppy

Alan Shugart’s floppy drive was a massive hit in the home computing market, but why include it in a list of enterprise storage technologies? Because its storage interface set the standard for plug- and protocol-compatible storage in the nascent microcomputer world.

Designing a general product for use in a multitude of systems was truly innovative, and many later computers were literally designed around both the concept and physical form factor of Shugart’s drive. Simply put, this fat floppy drive inspired computer designers to create computers that could accept standard peripherals, the very definition of open systems. The definition of “peripheral” would soon grow to include standard I/O devices like storage, terminals, printers, communications gear, and everything else we know today.

1980 Seagate ST-506

Shugart left his company in 1979, founding Seagate Technology. That company’s first product was the ST-506, a 5 MB hard disk drive that shared its physical shape with the SA-400 and used a similar interface.

Like the floppy, Shugart’s hard drive set the standard for microcomputers, eventually finding its way into enterprise systems, and catapulting Seagate to its current position of disk drive leadership. Higher-capacity derivatives of the ST-506 were fitted with interfaces using Larry Boucher of Adaptec’s SCSI protocol, which continues in use even today. Although the ST-506 wasn’t a “smart” drive, the ecosystem that developed around it was critical.

1984 DEC TK50

Digital Equipment’s introduction of the MicroVAX II in 1985 was accompanied by a new half-inch backup tape drive and cartridge called the TK50. Open-reel tapes had dominated non-disk storage before this, but cartridges quickly replaced reels, leading to the development of tape robotics and (more) reliable off-site storage.

If you picked up a TK50 tape today, you could be forgiven for mistaking it for a modern SDLT or LTO, because these use a similar cartridge and drive form factor, the same linear tape technology, and the same half-inch tape size. The TK50, DLT, and SDLT were the mainstays of open backup for decades.

1987 Auspex

This terrible photo is an actual Auspex press image!

What was the first open systems enterprise storage array? Leveraging Sun’s NFS protocol, and hiding a Sun workstation inside, Wizard of Oz style, the Auspex set the standard for everything we think of as “enterprise” in the open systems world. The company’s storage systems were truly ahead of their time, with ranks of redundant disks years before RAID became common.

Auspex was founded by Larry Boucher, father of SCSI and founder of Adaptec, and raked in sales while others struggled to figure out how to sell in the enterprise.  But their refusal to produce a smaller device would be their undoing. Today’s monolithic arrays owe as much to Auspex as they do to IBM, but the companies producing them could learn a lesson from the company’s demise.

1987 StorageTek 9310 PowderHorn

StorageTeks versatile and scalable 9310 PowderHorn defined backup

StorageTek was instrumental in many enterprise storage developments (see McData, for example), but one product literally transformed the datacenter: the 9310 “PowderHorn”. Consider the “glass house” datacenter tours that the largest companies would use to impress visitors: They would show off their mainframe, their Cray, or their PowderHorn.

This versatile system would accommodate every major tape cartridge format and system type and could scale to truly massive proportions. When disks were expressed in megabytes, PowderHorns held terabytes. 

1995 EMC Symmetrix 3000

EMC's Symmetrix 3000 family brought mainframe block storage technology to open systems

The 1990 introduction of the Symmetrix was the key turning point for the (now) giant of Hopkinton, but the first two generations were mainframe-only. In 1994, the company delivered data replication capability in the form of SRDF, moving key enterprise functionality to the storage array.

But it was the 1995 introduction of the third-generation Symmetrix 3000 that really changed the storage world. For the first time, open systems could connect to mainframe-class storage over the standard SCSI protocol and leverage features like SRDF and (in 1997′s Symm 4) TimeFinder.

One key ingredient often overlooked in the Symmetrix was in-box virtualization the likes of which hadn’t been seen before. It also featured RAID-like sub-disk data protection, though the Symmetrix line never did implement true RAID.

The Symmetrix was redesigned entirely in 2003 to become the DMX. Although it was as different from its predecessor as the New Beetle was from Volkswagen’s original, the DMX line continued many of the philosophical underpinnings set in 1990.

EMC also deserves credit for their original Celerra enterprise NAS system, which picked up where Auspex left off. The company followed these in 2002 with the Centera CAS system, which abandoned many traditional concepts of enterprise storage, like blocks- and filesystem-access and modular and monolithic architecture. The Centera is certainly innovative, but we have yet to see the impact it will have.

1994 Data General CLARiiON

Data General paired the CLARiiON (left) with their AViiON server

It’s easy to forget the world before RAID. But Data General was one of the first to market with a cached RAID system with their 1991 introduction of the HADA. Although this early system was tied to DG’s servers, it donated much of its architecture to a system that became more valuable than the rest of the company: CLARiiON.

Massively successful, and mighty impressive (PC Magazine called the first CLARiiON “amazing”), this modular block storage array set the template for over a decade. There were dozens of copycat arrays on the market within a few years of the introduction of the HADA. The CLARiiON gained Fibre Channel support, was sold to EMC, and remains a mainstay of corporate data centers, albeit with updates to every component. I have administered every generation of CLARiiON array, and can attest to their capability (when properly configured!)

1996 NetApp Multiprotocol Filer

NetApp's F330 Multiprotocol Filer was a huge hit in mid-size businesses and a major upgrade

NetApp (nee Network Appliance) was formed by a group of ex-Auspex engineers who wanted to create a more modular NAS server based on industry-standard hardware. They released their first product, the NFS-only FASServer, in 1995, but it was their August, 1996 introduction of the Windows-compatible Multiprotocol Filer software that really put them on the map.

Combined with their solid F220, F330, and F540 hardware, NetApp now had a serious challenger to Auspex, and their NAS systems blew away dedicated server-based solutions in terms of flexibility and manageability. NetApp’s unified NFS, CIFS/SMB, and HTTP access to content on their unique WAFL file system was impressive at the time, as was the quick setup and administration and the ease of adding drives to their RAID-4 sets. Plus, they brought a new level of friendliness to the data center with their bright colors and silly “toaster” nomenclature.

1999 McData Fibre Channel Director

McData has an interesting history, beginning with Storage Technology (StorageTek), intersecting with IBM, bought and spun off by EMC, and lately acquired by Brocade. Although the company spent a decade developing various peripherals for IBM mainframe systems, their late-1994 introduction of a switching director for ESCON traffic would change the storage world. We take large-scale SANs and LANs for granted today, but McData’s director was astonishing when it was introduced: There were no words to describe it or its function at the time, and period press articles are puzzling!

EMC scooped the company up a year later, but McData was spun out in 1997, adding Fibre Channel support that same year. In the ensuing years, McData became the enterprise SAN “arms dealer”, supplying IBM and EMC with the ED-5000 and ED-6064 directors. I recall commenting at the time that Fibre Channel connectivity built around the McData director was the first Storage Area Network worthy of the name. McData went IPO in 2000 and was purchased by rival Brocade in 2006. McData’s director architecture survives and thrives today against fierce competition from Brocade, Cisco, and others.

Others To Consider

What would you have included in this list? Here are some honorable mentions that I wish I had room for:

1. StorageTek “Aegis” L700 library
2. Brocade “LOOM” 2xxx switches
3. Adaptec SCSI HBAs
4. Copan MAID
5. EqualLogic iSCSI arrays
6. EMC Centera
7. HP EVA
8. HDS USP
9. Compellent Storage Center

Responses

Here’s a list of responses that others have posted:

• Marc Farley’s Top 10 List
• Alex MacDonald’s Bottom 10 List

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Top Ten Innovative Enterprise Storage Hardware Products
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.