A few years back, I wrote an immensely-popular series of blog posts outlining the four things that were holding storage system performance back, and the ways to fix them. At the time, I created some presentation content to go along with these posts, and even considered pulling them into a white paper, but nothing came of that. Now, however, I’m pleased to announce that my Four Horsemen are accompanying me to the stage November 10, 2015 at the DeltaWare Data Solutions Emerging Technology Summit in Edina, Minnesota.
Here’s the original “Four Horsemen” series:
When I wrote those four pieces, back in 2010, the industry was on the verge of a major shift: Hard disk drives had always been the critical inhibitor of storage system performance (see “The Rule of Spindles“), but widespread use of flash memory meant this was about to change. Many people don’t realize that RAID was originally designed primarily to deliver increased IOPS, not to protect data in the event of a drive failure. Yet even today, hard disk drives remain a critical gating factor to storage system performance. The advent of sequentialization, hybrid flash cache, and smarter storage systems has mitigated the problem, but most data still lands on a spinning disk eventually. Which means that we’re still stuck optimizing them.
NAND flash has also dented the second “horseman” (see “Never Enough Cache“), since it’s much cheaper on a per-capacity basis than DRAM. This is why virtually every storage system today has a large (by RAM standards) pool of NAND flash to serve as a temporary landing zone. And to stretch that metaphor, most systems also use NAND flash as a “pre-departure waiting area”, storing data that is likely to be read soon. And now that storage I/O has become quicker (see below), we’re starting to see optimization above NAND, with SLC caching MLC and DRAM above both.
Regardless of the performance of the storage capacity layer, however, there will always be bottlenecks in the I/O channel (see “I/O As a Chain of Bottlenecks“). Faster storage (thanks largely to flash) has disrupted the industry, with the “big SAN” model giving way to top-of-rack flash, converged infrastructure, and even re-internalized storage. Next-generation solid state storage is so fast, in fact, that it requires some extreme re-engineering to take advantage of all these IOPS. Otherwise we’re left harnessing a racehorse to a wagon.
The most exciting area of enterprise storage today is the emergence of intelligence and integration (see “Get Smart“). No matter what a storage device can do, we can’t maximize return on investment without optimizing data access. This is why today’s trend toward data-aware storage, integrated caches, server/storage integration protocols, and software-defined storage is exciting. If the array’s capabilities (and limitations) are exposed to an intelligent data management layer, we can extract maximum performance from the whole system. Even those pesky hard disk drives.
If you’re in Minnesota in two weeks, come on down to the DeltaWare Data Solutions Emerging Technology Summit! I’ll also try to film my presentation and post it here. And if you’re organizing a conference yourself, might I suggest that you reach out to bring me in as a speaker?