• Skip to main content
  • Skip to primary sidebar
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Computer History / Thinking About Storage In a New Way, From Cloud to Flash, with Dropbox and Fusion-io

Thinking About Storage In a New Way, From Cloud to Flash, with Dropbox and Fusion-io

July 23, 2013 By Stephen 1 Comment

I’ve been a storage revolutionary for quite a while, looking for new ways of data storage rather than technologies that perpetuate the same old approaches. That’s why I’m excited about the implications of two very different API access methods announced by Dropbox at DBX and by Fusion-io today at OSCON.

We need smarter, more integrated storage.

Dropbox App Storage

On the consumer side, there’s Dropbox, who announced Sync and Datastore APIs at their DBX conference this month. These APIs are interesting in and of themselves, but more so when one takes a big picture look at them: Dropbox is challenging the whole concept of mobile device storage, suggesting that temporary caches of a cloud datastore are more relevant.

It would be foolish to argue that distributed sync and key/value datastores aren’t taking over the mobile world, but these things are devilishly difficult to get right. Witness Apple’s continuing stumbles with iCloud, for example. But Dropbox has done a fantastic job of making sync work in the real world, and now they’re opening those systems to developers.

But Dropbox is also explicitly challenging the whole notion of file-based storage. This is another huge, and overdue, revelation. Why bother with files when what today’s apps really need is key/value lookups and blob storage? Watch the keynote above and consider what they’re saying: It’s a new world, and this is what storage ought to look like.

Fusion-io NVM Access

After taking a week to digest the implications of Dropbox’s mobility-enhancing storage strategy, I was contacted by Fusion-io regarding their announcements at Open Source Convention (OSCON) 2013. After a discussion with Brent Compton, Sr. Director of Product Management, I am convinced that this seemingly-unrelated announcement actually has a lot in common!

Fusion-io is announcing three contributions to open source:

  1. A key-value interface to flash, NVMKV – This is an API spec and library along with source code enabling applications directly to store data in non-volatile memory (e.g. Fusion-io flash cards) using a key/value pair rather than conventional file or block I/O.
  2. A modification of the Linux VM subsystem enabling better use of demand paging from non-volatile memory. Paging was basically given up for dead among Linux server guys, but makes a whole lot more sense in a NVM world!
  3. API specs for the Fusion-io flash translation layer enabling atomic access to non-volatile memory, including vectored atomic writes. This has already been implemented in MariaDB 5.5.31 and Percona Server 5.5.31-30.

Let’s take a moment to consider the Fusion-io announcement in a larger context: They’re not talking about conventional storage paradigms anymore. Instead, Fusion-io is pushing specific enhancements to real-world applications to use storage in a new way. This is the Internet datacenter flipside of the Dropbox announcement!

Here’s an example: MySQL databases (with Innodb) use a double write technique to ensure consistency: They write all data twice, then perform an fsync(), then proceed knowing the data is safely written. This is necessary because block storage can result in partial writes: Perhaps the system crashes after one or two 4K blocks were written, losing the remaining blocks and leaving the data in an inconsistent state. But Fusion-io’s atomic write API pushes responsibility for data consistency to the storage device: Issue an atomic write and the ioMemory card will ensure the data has been written completely. So MySQL can skip the double write and just commit data directly.

Atomic writes are practical for non-volatile memory devices because this is how they already function internally: Flash memory is handled in this way by the controller, so it’s a simple matter to expose this to the application and enable atomic writes. This has never been possible before because spinning disks just don’t work this way!

Fusion-io can even do vectored I/O, enabling multiple disjoint buffers to be updated in one atomic operation. Known as scatter-gather, this is another new paradigm for data storage since disks simply could not do this.

Then there’s the key/value API. This has exciting implications for databases, sure, but it goes well beyond that. Just like mobile applications, today’s “big data” systems (yeah, I said it) need key/value access not just block storage. This has long been handled by intermediary applications (databases) and those will likely continue. But instead of managing data on block storage, databases can commit a shard to a NVM key/value interface like the one Fusion-io just offered.

Stephen’s Stance

It’s not that we need a new storage array or company. Rather, we need a whole new way of “doing storage” that reflects the changing reality of both clients (mobile devices, web applications, etc) and storage targets (flash vs. disk). We already live in this new world of key/value datastores but storage technology has been slow to adapt. That’s why I’m excited about Dropbox, Fusion-io, and so many other “new ideas” companies!

It’s important to recognize that the “old school” companies aren’t completely unaware of this shift. The SNIA NVM Programming working group includes lots of familiar names in storage (Dell, EMC, HP, IBM, Intel, NetApp, Oracle, etc), all looking beyond today’s spinning disks. Just like me, they recognize that non-volatile memory is the future.

You might also want to read these other posts...

  • Electric Car Over the Internet: My Experience Buying…
  • Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!
  • Ranting and Raving About the 2018 iPad Pro
  • GPS Time Rollover Failures Keep Happening (But…
  • Tortoise or Hare? Nvidia Jetson TK1

Filed Under: Computer History, Enterprise storage, Features, Terabyte home Tagged With: API, Atomic I/O, big data, database, Dropbox, Fusion-IO, Innodb, ioMemory, key/value, Linux, MariaDB, MySQL, NVM, OSCON, Percona, storage revolution, vectored I/O

Primary Sidebar

James T. Kirk: Mr. Scott. Have you always multiplied your repair estimates by a factor of four?
Montgomery Scott: Certainly, sir. How else can I keep my reputation as a miracle worker?

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!

May 29, 2020

What You See and What You Get When You Follow Me

May 28, 2019

GPS Time Rollover Failures Keep Happening (But They’re Almost Done)

April 6, 2019

Symbolic Links

    Featured Posts

    Ten Terrible Apple Products

    June 14, 2012

    Why You Should Never Again Utter The Word, “CIFS”

    February 16, 2012

    My 2012 Project: Improving Energy Efficiency

    January 3, 2012

    A Fairy Tale of Two Storage Protocols

    September 23, 2014

    Generation 3 drobo: Fall In Love All Over Again

    April 9, 2015

    Faster Ethernet Gets Weird

    June 19, 2015

    Deduplication Coming to Primary Storage

    September 16, 2008

    Thoughts on the Modern Miracle of 3D Printing

    July 28, 2015

    How Will Cisco Recover From The Consumer Strategy Blunder?

    January 2, 2013

    How To Sync Your iPad With Your Exchange Server

    April 3, 2010

    Copyright © 2021 · Log in