Commodity Hardware Always Wins

Get used to it: Commodity hardware like this Super Micro SBB server will dominate enterprise storage

The history of technology moves in fits and starts, but one trend trumps all else: An inevitable shift from fine-tuned specialized gear to general-purpose commodity building blocks. We see it in both hardware and software, and at all levels of the industry, from chips and wafers to operating systems and networking devices. Take a step back and you’ll certainly agree: Commodity hardware always wins (eventually).

Doing the Impossible

Building something new often requires amazing feats of engineering. Steve Wozniak’s hardware wizardry with the original Apple computer is now legend, but similar tales are found everywhere.

It’s frightfully difficult to do the impossible without at least some wizardry. Most new technologies are therefore built on specialized hardware and hand-tuned software. Engineers at the forefront of technology must eke out every ounce of performance from trailing technologies.

This is why we see so many special-purpose processors in high-performance devices, and why companies employing custom ASICs often enjoy a performance advantage. In enterprise storage, we look to companies like HDS and BlueArc who packed their arrays with special-purpose hardware, pointing the way to the future. We also see impressive developments from Fusion IO and SandForce in the SSD space, leading to the next generation of storage.

Can You Keep Ahead Of Intel?

Just about every technology sector progresses from the impossible to the commonplace, and these changes are often very quick. High-performance storage systems were once exotic multi-million dollar devices but millions of IOPS are now available for under $100k from dozens of vendors.

This progression typically includes a move from special-purpose to commodity underpinnings. Exotic real-time operating systems have been pushed aside in favor of Linux, BSD, and even Windows, while ASICs and FPGAs give way to Intel’s CPU juggernaut. Even Apple computers are today almost entirely commodity PCs.

I love how Denton Gentry phrases this in his Monday blog entry about Intel and Achronix:

“Once you commit to a specialized hardware design, the clock starts ticking. There will come a day when a software implementation could meet the requirements, and at that point the FPGA becomes an expensive liability in the BOM cost. You have to make enough profit from the hardware offload product to pay for its own design, plus a redesign in software, or the whole exercise turns out to be a waste of money.”

In other words, specialized software on proprietary hardware will eventually be overtaken by general-purpose software on commodity hardware. The decision must include not just what one can do today but what that baggage will mean in the future. Designing a system around proprietary components might look good now, but the next-generation product will be put at risk by this decision.

Stephen’s Stance

My opinion is right there in the title of this piece: Commodity hardware always wins. No matter how great your ASIC is, industry-standard CPUs will out-perform it sooner or later. No matter how much effort you put into tuning your software, Linux-based systems will eventually do just as well.

The rise of commodity hardware is everywhere: EMC, HDS, IBM, Oracle, and HP have all embraced Intel CPUs and their hardware is looking more and more like Intel’s reference designs, too. Startups are increasingly relying on software rather than hardware for their differentiation, and we’re seeing Supermicro servers shipped with just about everyone’s name on them. The Storage Bridge Bay specification is looking better all the time, too. Commodity hardware is winning in storage, just like it always does.

For more on this topic, see these related posts by others:

  • DGentry

    Yup. I wasted four years of my career on a custom CPU which was a fire breathing monster when it debuted, but became old and busted before we could get its replacement out.

    The chip business is a marathon, not a sprint. Startups tire out after the first few miles.

  • Nick

    Totally Agree Steve. Commodity will always win and provides end-users with a base platform they know is being used by the masses.

    We regularly shift to the latest, best in class, commodity tin to ensure our customers get the best that is out there to run MatrixStore on. We decided from day one not to dedicate our limited resources to hardware development and that is now paying dividends.

  • Joseph Martins

    As I often say… the only true commodities are found in the periodic table.

    The word commodity aside…what’s your point?

    We all know products become less expensive and more widely available year over year. [Though at a ZDNet Storage Summit a couple years back I forecast that trend will eventually slow down, stop and perhaps even reverse in the not-too-distant future as real commodities become more scarce.]

    Early adopters will continue to pay a premium for an early competitive advantage and their behavior will continue to fuel more widespread adoption. By the time the tech goes mainstream, they’ve moved on to the next big thing. In many, but not all cases, what was once a low-volume high-margin market dominated by a handful of companies becomes a high-volume, pathetically-low-margin market overflowing with competition.

    From a Joe Average Consumer perspective it’s fantastic. From a business perspective (sellers and bleeding edge adopters) I’m not so sure I’d call that a win.

  • Pingback: Commodity hardware always loses | RayOnStorage Blog()

  • Sshottan

    While indeed capturing a clear trend in the storage business, Stephen Foskett

  • Pingback: The Scale Out Game « BlueViews()

  • Pingback: In Praise of Innovations « BlueViews()