The Four Horsemen of Storage System Performance: These four ugly gentlemen stand between you and your data.

Why do some data storage solutions perform better than others? What tradeoffs are made for economy and how do they affect the system as a whole? These questions can be puzzling, but there are core truths that are difficult to avoid. Mechanical disk drives can only move a certain amount of data. RAM caching can improve performance, but only until it runs out. I/O channels can be overwhelmed with data. And above all, a system must be smart to maximize the potential of these components. These are the four horsemen of storage system performance, and they cannot be denied.

The Nature of Disks

Hard disk drives are getting faster all the time, but they are mechanical objects subject to the laws of physics. They spin, their heads move to seek data, they heat up and are sensitive to shock. Storage industry insiders recognize the physicality of hard disk drives in the name we apply to them: Spindles. And there is no way to escape the bounds of a spindle.

The performance of a hard disk drive is constrained by both its physical limitations and how we use it. Physically, a hard disk drive must spin its platters under a moving arm with a read/write head at the tip. This arm slides across the media, creating a two-dimensional map of data across the disk. Hard disk drives spin at a constant speed, so data at the edge passes under the head quicker than data at the center, creating a distinctive curve of performance.

Although they are random-access devices, hard disk drives cannot access multiple locations at once. Although modern command queueing and processing allows the drive controller to optimize access, I/O operations are serialized before the drive can act on them. It takes a moment for the head to move (seek time) and the disk to spin (rotational latency) before data can be accessed, so sequential operations are much faster than random ones.

Most operating systems lay data out sequentially, beginning at the edge of the disk and moving inward. Although modern file systems try to keep individual files contiguous and optimize placement to keep similar data together, seeking is inevitable. This is the nature of physical hard disk drives.

Accelerating Disks

Disks can be made faster mechanically in one of three ways:

1. Data can be packed closer together, allowing more to be read in a given amount of time. This is a natural outgrowth of storage density improvements, and explains why today’s slowest hard disks have much better sequential access performance than the fastest enterprise drives of just a few years past.
2. Spindle speeds can be accelerated. This has a dual benefit of passing more data under the heads in the same amount of time and reducing the time it takes for the proper spot on the disk to pass the heads. Maximum spindle speed has remained constant at 15,000 rpm for decades, but the slowest drives have accelerated in recent years, with 5,400 now the minimum and 7,200 rpm now common even for portable devices.
3. Disk media can be made smaller, reducing the area the heads must travel across. Increasing media density allows for the viability of smaller disks outside volume-sensitive areas like laptops. Many high-speed enterprise drives use small-diameter media even where the outer shell remains at the standard 3.5″ form factor.

Each of these methods are limited by practicality, however. Doomsayers have been declaring that density is reaching the limits of technology for decades, and while the industry has always surpassed these expectations, it can only adapt so far. Spindle speeds beyond 15,000 rpm are impractical as well, especially since this limits the density of storage used. And geometry limits how small hard disk media can get: The spindle itself requires some space, leaving less area for data storage. In fact, it appears that today’s 15,000 rpm 2.5″ hard disk drives are as quick and small as practicality allows. Only increasing density will improve their native performance further.

Combining Spindles

Although they are quick, the mechanical limitations of hard disk drives makes them the first suspect in cases of poor storage performance. A single modern hard disk drive can easily read and write over 100 MB per second, with the fastest drives pushing twice that much data. But most applications do not make this sort of demand. Instead, they ask the drive to seek a certain piece of data, introducing latency and reducing average performance by orders of magnitude.

Then there is the I/O blender of multitasking operating systems and virtualization. Just as each application requests data spread across a disk, multitasking operating systems allow multiple applications and process threads to request their own data at once. File system development has lagged behind the advent of multi-core and multi-thread CPUs, leading to frustrating slowdowns while the operating system waits for the hard disk drive. Virtualization magnifies this, allowing multiple operating systems running multiple applications with multiple threads to access storage all at once.

The key innovation in enterprise storage, redundant arrays of independent disks or RAID, was designed to overcome the limits of disk spindles. In their seminal paper on RAID, Patterson, Gibson, and Katz focus on “the I/O crisis” caused by accelerating CPU and memory performance. They suggest five methods of combining spindles (now called RAID levels) to accelerate I/O performance to meet this challenge. Many of today’s storage system developments are outgrowths of this insight, allowing many more spindles to share the I/O load or optimizing it between different drive types.

The Rule of Spindles

This is the rule of spindles: Adding more disk spindles is generally more effective than using faster spindles. Today’s storage systems often spread I/O across dozens of hard disk drives using concepts of stacked RAID, large sets, subdisk RAID, and wide striping.

Faster spindles can certainly help performance, and this is evident when one examines the varying performance of midrange storage systems. Those that rely on large, slow drives are much slower than the same systems packed with smaller, quicker drives. But the rule of spindles cannot be ignored. Systems that spread data across more spindles, regardless of the capabilities of each individual disk, are bound to be quicker than those that use fewer drives.

Onward: Cache, I/O, and Smarts

The horseman of spindles is harsh, but he does not rule the day. There are many ways to overcome his limits and his three brothers often come into play. These are cache, which bypasses the spindle altogether; I/O, which can constrain even the fastest combination of disk and cache; and the intelligence of the whole system, which limits or accelerates all the rest. We will examine these horsemen in the future!

I’ve been meaning to write this up for a long time. Thanks for listening and commenting!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | The Four Horsemen of Storage System Performance: The Rule of Spindles
This post was categorized as Computer History, Enterprise storage, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It has been the core technology behind the storage industry since day one, but the sun is setting on traditional RAID technology. After two decades of refinement and fragmentation, we are abandoning the core concepts of disk-centric data protection as storage and servers go virtual. Next-generation storage products will feature refined and integrated capabilities based on pools of storage rather than combinations of disk drives, and we will all benefit from improved reliability and performance.

RAID Classic

Early storage systems were revolutionary, in physically removing storage from the CPU, in enabling sharing of storage between multiple CPUs, and especially in virtualizing disk drives using RAID. When Patterson, Gibson, and Katz proposed the creation of a redundant array of inexpensive disks (RAID) in 1987, they specified five numbered “levels”. Each level had its own features and benefits, but all centered on the idea that a static set of disk drives would be grouped together and presented to higher-level systems as a single drive. Storage devices, as a rule, mapped host data back to these integral disk sets, sometimes sharing a single RAID group among multiple “LUNs”, but never spreading data more broadly. Storage has remained stuck with small sets of drives ever since.

The core insight of the 1980s remains true: More spindles means better performance. Although additional overhead dulls the impact somewhat, the benefit of spreading data across multiple drives can be tremendous. A typical RAID set offers much better performance than the drives alone, and can handle a mechanical failure as a bonus.

Cracks are appearing in the RAID veneer, however. Double drive failures are much more common than one would expect, leading to the development of hot spare drives and dual-parity RAID 6. If four drives perform well, then forty drives perform much better, leading to the common practice of “stacking” one RAID set on others. Caches and specialized processors were introduced to overcome the performance issues related to parity calculation.

But traditional RAID cannot overcome today’s most critical storage issues. As drives have become larger, the tiny chance of an unrecoverable media error compounds, becoming a certainty. Even dual-parity will not be able to guarantee data protection on the massive disks predicted for the near future – statistics cannot be denied. The latest disks contain so much data, without commensurate improvements in throughput, that rebuild times have skyrocketed, resulting in hours or days of reduced data protection.

RAID is also ill-suited to the demands of virtualized systems, where predictable I/O patterns become fragmented. It cannot provide tiered storage or account for changing requirements over time. It cannot take advantage of the latest high-performance solid state storage technology. It cannot be used in cloud architectures, with massive numbers of small devices clustered together. It interferes with power-saving spin-down ideas. Most RAID implementations cannot even grow or shrink with the addition or removal of a disk. In short, traditional RAID cannot do what we now need storage to do.

RAID is Dead

Although most vendors still use the name, nearly every one has abandoned much of the classic RAID technology. EMC’s Symmetrix pioneered the idea of sub-disk RAID, pairing just a portion of each disk with others to reduce the impact of “hot spots”. HP’s AutoRAID added the ability to dynamically move data from one RAID type to another to balance performance. And NetApp paired disk management so closely with their filesystem that they were able to use RAID 4 and the flexibility it brings.

Today, a new generation of devices has even evolved beyond RAID’s concept of coherent disk sets. Compellent, Dell EqualLogic, 3PAR and others focus on blocks of data, moving portions of a LUN between RAID sets, disk drive types, and even inner or outer tracks based on access patterns. With these devices, a single LUN could encompass data on every drive in the storage array. And the latest clustered arrays can spread data across multiple storage nodes to scale performance and protection.

These innovative devices point the way to a future in which virtual storage is serviced and protected very differently than in the past. Perhaps software like Sun’s ZFS serves to illustrate this future best: It unifies storage as a single pool, intelligently protecting it and presenting flexible storage volumes to the operating system. Although Sun calls its data protection scheme “RAID-Z”, it has little in common with its namesake. Like NetApp’s WAFL, the copy-on-write ZFS filesystem is totally integrated with the layout of data on disk, allowing mobility and efficient use of storage. A single pool can include striping, single- or dual-parity, and mirroring, and disks can be added as needed. Importantly, ZFS also checksums all reads, detecting disk errors.

Long Live RAID

The post-RAID future will see these concepts spread across all enterprise storage devices. Disks will be pooled rather than segregated into RAID sets. Tight integration between layout and data protection will allow for much greater flexibility, integrating tiering and differing data protection strategies in a unified whole. Storage virtualization will allow mobility of data within these future storage arrays, and clustering will enable massive scalability.

Two things will likely remain to remind us of Patterson, Gibson, and Katz, however. First, the core principle that multiple drives working as one yields dividends in terms of performance and data protection. And second, that whatever we use should be called RAID, even though the definition of that term has changed beyond recognition in the last two decades.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Turning the Page on RAID
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.