Waves of Storage Innovation

Before you read this, check out Today’s Storage: Same As It Ever Was for a bit of background…

For the past few decades, some of the best minds in computer science have been drawn to the question of storage. How can data be better protected, shared, and exploited? Given the price and performance profiles of various storage devices, how can we maximize benefit for a given application? What can we do to transcend inertia and make data available everywhere at once?

These fundamental questions have been tackled by a continuing stream of companies promoting waves of new technology. The limitations of block storage gave us CAS, object storage, and the cloud. RAID was invented because hard disk drives can not physically drive sufficient I/O operations, but wide striping, data dispersal, tiering, and caching are table stakes today. Data compression was pioneered as a transmission technology but has moved to array controllers and static data sets.

In each area, a predictable pattern emerged:

1. New algorithms and architectures are developed
2. Start-up companies are funded and launched, creating standalone products centered on them
3. These technologies become features as they are leveraged broadly across the industry, with the start-ups acquired or withering
4. Mainstream adoption is predictably slow and insider attention turns to the next wave

This last bit is pretty remarkable, and something I’ll be talking about in a follow-on post, but let’s focus on the early phases for now.

How many product, and even company, categories have become merely an obscure feature? I’ve watched this pattern repeat with SRM, object storage, data deduplication, thin provisioning, iSCSI, hybrid storage, and all-flash architectures (just off the top of my head). And now we’ve got RAM and flash caching, storage virtualization, and data-aware storage.

It is interesting too that each wave tends to build on those before it. Deduplication was once a marquee product but now is mainly a way to optimize the use of flash for hybrid or all-flash storage systems. SRM was the “next big thing” but is now an integrated feature of data-aware storage systems. Scale-out and virtual storage is easiest to do with an object store under the covers. And so on.

Each technology wave is accompanied by the same archetypal companies, too:

• Academics and technologists who can’t articulate a use case but usually have the best technology
• Re-packagers of outdated products trying to stay relevant by claiming they’re part of the current wave
• Marketing-first companies making noise with little or no IP of their own but capable of making lots of noise
• Super-funded leaders who hire and expand rapidly, are flush with cash, and become the standard-bearers for the wave
• Actual companies selling useful products (too few of these!) who have solid products and rely on real sales and use cases

It is rare that any of these companies lasts more than a few years. But their products tend to live on, occasionally in standalone companies (usually the last two) but more typically as a product line or feature inside one of the giant incumbents.

And so it goes. Waves of innovation and waves of companies, crash on the storage market, but the same incumbent leaders and product lines survive for decades. It’s actually pretty depressing. Are things changing? It’s hard to see sometimes, but real progress has been made. And perhaps all those waves have finally chipped away at the outdated storage paradigms I complained about last time to send it all toppling down.