Single Instance Storage

It’s fairly simple for a system to eliminate duplicate data by storing only a single instance of multiple identical files. In other words, if you and I both upload “Presentation.pptx” and it’s bit-for-bit identical, it would be a simple matter to store just one copy.

Dropbox definitely does this. I proved it with a simple experiment:

1. Create a new 10 MB encrypted disk image in TrueCrypt (so it’ll be 100% unique, random data)
2. Move it to the Dropbox folder and wait a few minutes as it uploads
3. Copy the file with a new name to the folder and notice that it “uploads” instantly

Dropbox is at least single-instancing storage. This helps users, since it speeds uploads and reduces bandwidth usage. It helps Dropbox in the same way, but goes further since they still “charge” files against your account whether they’re single-instanced or not.

Note that this single-instancing works across users and geographies. I gave a file to a friend to upload to a different Dropbox account, and saw the same “acceleration effect.” This would be quite useful to users and the company for files like iTunes songs which are identical and widespread.

Clashing MD5 Hashes?

Three files with identical sizes and MD5 hashes but different names? Creepy!

A global single-instance storage system sounds great, but it opens the door to hash collision issues. Imagine if you and I both uploaded identical files. Both would have the same “fingerprint” and Dropbox would only store it once. Now imagine instead that, out of coincidence or malice, I uploaded a file with the same fingerprint as yours but different contents. This is not so far-fetched as it seems, and could lead to all sorts of security nightmares.

A common and compromised file checksum method is MD5, so I decided to test how Dropbox handles files of identical size, name, and MD5 hash using the “Nostradamus Attack” PDFs generated by Marc Stevens. My tests show that Dropbox correctly handled the files I tried, and no combination of uploading and naming could force it to incorrectly store the right file. So Dropbox either doesn’t use MD5 or uses a combination of hashing and other mechanisms. Testing other schemes is left as an exercise to the reader.

One more thought: The fact that de-duplication is mentioned in the “privacy” section of the Dropbox policies raises my eyebrows, since it indicates that they see this hash collisions as a matter of privacy rather than data corruption. This indicates that Dropbox is both aware of and susceptible to hash collision attacks generally, though obviously not as simply as creating a bogus MD5 match.

Note: Dropbox is well aware of this issue, having recently squashed an open-source exploit called Dropship!

Sub-File De-Duplication

Data de-duplication is like single-instancing, but it applies to some subset of data. Some enterprise storage systems de-duplicate at multi-megabyte levels, while others are far more granular.

To test whether Dropbox de-duplicates data, I devised a simple experiment:

1. Create a new local copy of my existing random TrueCrypt file
2. Add a single byte to the end using the “cat” command
3. Copy the resulting file to Dropbox
4. Watch as Dropbox takes just a few seconds to upload the new file

This test proves that Dropbox does indeed de-duplicate at the sub-file level. Since it took a bit longer to upload that would be expected for a single byte, we can see that Dropbox “chunks” files for hashing and uploading.

De-Duplication Granularity

The next question is just what size chunks or blocks Dropbox uses to de-duplicate data. To test this, I created various blocks of random data using TrueCrypt and experimented to see where the “stair-steps” were in terms of upload time.

My tests used four basic building blocks of 512 KB, 1024 KB, 2048 KB, and 4096 KB in size. Guessing that Dropbox used one of these sizes for their chunking system, I assumed these would quickly demonstrate the answer.

On my Mac, Dropbox clearly uses a 4 MB "chunk" size for deduplication

First, I uploaded each file individually and watched as Dropbox took about 30 seconds per MB. This will vary greatly, of course, but the absolute performance doesn’t matter. Only relative performance matters for demonstrating chunking.

Next, I concatenated each file with itself to create a new file twice as large. This would be ideally “chunkable” since it consists of exactly identical data with a nice, clean, evenly-divisible “border”. I uploaded each of these and noticed that the “4096 KB x 2″ file uploaded nearly instantly, while all others took the expected amount of time.

I repeated this with “x 3″, “x 4″, and “x 8″ files and noticed that the 4096 KB (4 MB) “barrier” was very obvious. Whenever a file contained 4096 KB or less of data Dropbox had seen before, it single-instanced it. Any time it saw a unique “block” smaller than this, it uploaded it fresh.

This proves, at least in the case of my own Mac OS X install of Dropbox, that a 4 MB chunk size is used for de-duplication.

 Stephen’s Stance

Dropbox is a very useful service, and I appreciate the technology they use to make it work. By single-instancing storage, the company is able to keep costs and transfer time in check and offer a basic service for free for many users. Despite the recent security issue, I continue to use Dropbox myself and would not hesitate to recommend it. But I do suggest using your own encryption for any sensitive data, as demonstrated in my recent post, Mac Users, Secure Your Stuff in Dropbox.

I remain somewhat concerned about the privacy and security implications of global de-duplication of shared random data. If they use SHA-1 hashes alone, which I suspect, there is a chance that an object will not be stored correctly once 2^80 (or perhaps 2^69 or even 2^52) objects are stored. This would lead to issues of data corruption or inadvertent disclosure. This is a very remote chance indeed, but “birthday problems” like this work against hashing systems. I would love to hear from Dropbox regarding how they prevent this from happening, including disclosure of their methods of hashing data. It’s nice to see the company taking responsibility by disclosing this in their privacy policy, though!

Update: Dropbox apparently does indeed use raw SHA256 hashes to “uniquely” identify data, and this can be exploited in a number of ways.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | How Does Dropbox Store Data?
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Apple's iCloud is not just MobileMe 2.0!

iCloud is a key enabler of the “post-PC” experience for iOS users. It supports wireless daily backup, storage of purchased music, apps, and books, and synchronization. It also has some interesting Mac OS X features, and it’s free. But the most-compelling feature of iCloud is what it means to future applications on iDevices, the Mac, and even Windows!

iCloud Is Much More Than MobileMe

It’s easy to look at iCloud as a simple evolution of MobileMe, and that’s how the majority of users will see it. After all, Apple mostly focuses on end user computing and the iCloud site focuses on familiar applications.

Photo Stream is a great example of the sort of application enabled by iCloud storage. It will push and cache photos to any supported device, including iPhone, iPad, Mac, and Windows PC. Think of it as a cloud-enabled shared photo frame. It’s much like Picasa or Flickr’s offerings, actually.

But scratch the surface, and iCloud is much more. The iCloud storage platform is available for developers of Mac OS X, iOS, and other operating systems. This is how Documents in the Cloud stores Pages, Numbers, Keynote, and third-party application content, and developers will eat it up.

The iCloud Storage API

See also From CAS to Cloud: Revolutionary Storage

The most important new feature is the iCloud Storage API, a real, honest-to-goodness cloud storage API. It supports revision control and push updates to devices, as well as storing “key/value data” in the cloud, opening the door to lots of creative uses.

But there are already many other cloud storage services with APIs. iCloud is different because the Apple ecosystem makes it so. This isn’t mere fanboy talk: The installed base of iOS devices and apps will spur adoption on the part of developers to be sure, but let’s not forget the users.

Apple already has the largest pool of e-commerce users short of Mastercard, and all of these users will transparently be able to use iCloud storage in many applications. Like Game Center, Facetime, and iTunes, it will just work. iCloud Storage is another front in Apple’s war to take over the computing client world.

Stephen’s Stance

At WWDC, Steve Jobs specifically talked about Apple spending 10 years trying to “get rid of file system storage”. Although you might think a “storage guy” like me would be upset by this, I reacted with joy. I will not shed a tear when the old “disk, folder, files” paradigm is destroyed, since I’ve seen the power of real API-driven cloud storage. The iCloud Storage API is a huge step in the right direction, and it’s especially important that Apple is taking it.

iCloud Storage won’t keep massive companies like EMC and NetApp awake at night. In fact, they’ll likely be cheering its success since they provide the equipment that supports the service. But end users are entering a new world, where local storage decreases in importance and cloud services like iCloud come to dominate daily interaction with apps. iCloud, like Apple’s computers, truly is designed “for the rest of us!”

© sfoskett for Stephen Foskett, Pack Rat, 2011. | How Apple iCloud Will Challenge the Storage Status Quo
This post was categorized as Apple, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Thoughts on Hitachi Data Systems, their products and market

I’ve been following the progress of Hitachi Data Systems (HDS) for well over a decade. When I participated in high-end enterprise storage system shootouts as an end-user, HDS routinely placed second against EMC, HP and NetApp in the, and the decision was always a close one. But the storage market has changed, with nimble startups innovating around established competitors and industry giants flexing their marketing muscles.

Where Does HDS Fit?

Concerned with the question of where HDS fits in this new market, I was pleased to be invited to join the company at their Sefton Park, UK briefing center to get some answers. After two days of intense briefing and discussion, I remain unsure of HDS’ future: The company impressed me as always with their technology, but it takes more than a few good people and products to compete effectively.

I always respected HDS and its products, but they remain stubbornly fixated on the high end of the market. The HDS USP/VSP may be the only really credible competitor to the EMC Symmetrix in the stratosphere of enterprise storage, but these products are analogous to the Airbus A380 and Boeing 747. No one expects them to go away any time soon, and they represent a nexus of innovation and profitability, but the market as a whole has moved on.

Ask an application developer about the infrastructure of their dreams and they’ll start talking scale-out platforms that sound awfully cloudy. The Symmetrix and USP/VSP are the mainframes of storage, and will be desired and required for certain use cases for decades to come. But CIOs are much more interested in the next wave of applications that their businesses will grow on in the future, and these applications need something totally different: A storage platform.

HCP: Positioned for the Next Generation

The best move HDS ever made was acquiring Archivas, developer of what is now known as the Hitachi Content Platform (HCP). A crossover between content-addressable storage (CAS) and the new generation of cloud storage systems, HCP is an excellent product for next-generation enterprise applications, with an HTTP/REST interface, object-level metadata-driven storage, and solid credentials for reliability.

The more I learn about HCP, the more I feel that EMC made a mistake by developing an entirely-new cloud storage platform (Atmos) rather than further developing their CAS system (Centera). HDS remained committed to HCP, and the result better matches the needs of enterprise applications than either of EMC’s products. I remain skeptical of HCP as a public cloud platform, but would not hesitate to recommend it for internal use cases.

HDS’ Data Ingestor (HDI) shows how the HCP platform can benefit an enterprise environment. As a NAS gateway, HDI pulls data from “the edge” (remote offices, file servers, and other applications) and stores it in a core HCP repository. This allows enterprise data to be better managed, protected, and manipulated than traditional distributed storage solutions. HDS is pitching HDI as an alternative for remote office backup, but it is really an example of the next-generation storage platforms enterprise CIOs wish they had today.

The Hitachi Clinical Repository vividly demonstrates what this future storage platform looks like. HDS surrounded the HCP with integration points for electronic health records applications and systems, allowing hospitals and other healthcare organizations to handle this rapidly-expanding data set. Departments as diverse as payroll and radiology can share the same platform, tracking and organizing data across applications and locations. The Clinical Repository at once demonstrates the strength of the HCP technology, the strategic value of integrated storage platforms, and the fact that HDS will not be left behind in this new world. Bravo!

Serious Challenges for HDS

But HDS’ future is not a slam dunk. The company faces serious obstacles in all directions, and it is not clear that they are ready to overcome them.

Let’s start with baggage. HDS is the exact opposite of a startup. This is an industry stalwart with a history of technology stronger than its sales execution; a private organization that must open up and change course without sinking; a company at once supported by and saddled with its link to “the mothership”, Hitachi Limited; and a group of talented people whose pride does not always serve their objectives.

The recent sale of Hitachi Global Storage Technologies (HGST), a key hard disk drive manufacturer, to Western Digital set some tongues wagging that HDS would be next. But HDS is nothing like HGST: Hitachi is first and foremost a developer and manufacturer of technological products, from televisions to nuclear reactors, and HDS is an integrator and distributor of their storage products. Hitachi could, in theory, decide to exit the storage space, and this would be the end of HDS. But this is highly unlikely. Hitachi is in the IT space for the long haul, with a slate of products that also includes servers and networking gear, and HDS is key to internationalizing this business.

But Hitachi is also an anchor, restricting the flexibility and creativity of HDS, and the curious case of Hitachi’s blade servers is a case in point. As noted, Hitachi is a major server vendor in Japan, and the company has named HDS the international custodian of these products. But HDS lacks the resources to sell into the highly-competitive server market. HDS has little choice but to put on a brave face and try to use these blade servers for their content platform, NAS, and unified “datacenter block” offerings, but there is no way they would have sought this product if not for Hitachi’s influence.

More pressing, however, is the challenge HDS faces with their sales force. Time and again one hears tales of clueless HDS sales reps pushing the same old SAN products to the same old customers rather than talking about HCP and the value of a strategic storage platform. One fellow event attendee remarked that his rep hadn’t ever mentioned HCP and HDI, even though it would have been a perfect fit for his large multi-national corporation. The rep either didn’t know about it or didn’t want to bother rocking the boat, and both are equally damning for HDS’ future growth.

One also senses a certain frustrated pride from some HDS employees, and this can be off-putting. This event was populated by some of the best and brightest representatives the company has to offer, but some continually resorted to a pointless “been there, done that, invented it” grouchiness. To hear them tell it, Hitachi is the secret force behind just about every enterprise storage innovation of the past two decades and their failure to dominate the market is a cruel injustice. But this is patently false: HDS is an innovator to be sure, but they play in a competitive market of equals. They must take responsibility for their own failings and work harder for success rather than whining about pretenders and upstarts.

Stephen’s Stance

As the baseball season begins, I can’t help but compare HDS to a big-budget team like the Chicago Cubs: They have all the tools and talent they need to succeed, but only time will tell how competitive they can be.
HDS should be commended for acquiring and developing the HCP technology, as well as spotting and fostering great new faces like Michael Heffernan, David Merrill, and Dave Wilson. But HDS must execute.

My prescription is simple but difficult: Lose the hubris and really dive into the battle for hearts and minds. Engage thought leaders and customers alike and show them how you can solve business problems rather than blathering on about unrecognized technical superiority. Educate the sales force and better incentivize them to “go strategic” instead of falling back on “speeds-and-feeds” SAN sales, and cut off anyone who isn’t on board. Let Heff and Wilson run free, find more like them, and bring Merrill into every CIO-level meeting.

HDS has to step up and be the healthy competitor for NetApp and EMC that the enterprise storage industry so desperately needs.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Concerning HDS
This post was categorized as Computer History, Enterprise storage, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

My new MacBook Pro was up and running with my data in less than an hour, thanks to Google and Dropbox!

I replaced my trusty MacBook Pro last week, the latest in a series of upgrades stretching back over 25 years. In the past, moving to a new computer is a time-consuming process of installing applications and moving data. But things were different this time: I still had the installs to do, but most of the data migrated on its own.

A Cloud of Data

Like many people, I’ve spent years unconsciously integrating my workflow with cloud applications and services. My email was first, and moving it to Google’s servers pulled my calendar and contacts along as well. This kind of data “wants” to live in the cloud, where it can be accessed on my phone, laptop, desktop, or any web browser I happen to be seated in front of.

Over the last few months, I have begin to use Dropbox to mirror my documents off-site. A clever trick allows this cloud-based repository to keep multiple computers in sync as well, and I recently set it up between my desktop iMac and the old MacBook Pro.

My only portable data set that still doesn’t live in the cloud is the collection of music and movies I like to take with me. The iMac maintains a massive iTunes repository on my Drobo, but I like to take a few thousand songs and a few movies and TV shows on my laptop as well. It seems ironic that this data is so stubbornly local, considering that, for the most part, it purchased and downloaded from a cloud service!

That’s It?

The majority of my data migrated automatically to my new computer

Just after buying my new 2011 13″ MacBook Pro, I sat down in a cafe next to the Apple Store to have a look. I was still in range of the Apple Store guest network, so the laptop was already online. I typed my information into Apple’s Mail, Contacts, and iCal applications and watched as Google re-populated them automatically.

But even I was surprised at the ease of moving the rest of my daily data. I downloaded the Dropbox client and entered my credentials. In moments, my entire Documents folder began filling up, and the astonishingly-quick Apple network made short work of a decade of content. By the time I finished my smoothie, I was up and running.

Stephen’s Stance

The ease of this migration is simply amazing: I was up and running quicker than I could have driven home! Google and Dropbox made short work of my “working set” of data, and I could have been literally anywhere on the planet. How times have changed!

Once I returned home, I fired up rsync to pull over the iTunes library and I was ready to retire the old machine. To be on the safe side, I pulled the 640 GB hard disk drive out of the old MacBook pro and installed it in an external enclosure. I’ll replace it with another drive and re-install Mac OS X before handing it down to a family member.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | How Google and Dropbox Revolutionized My Laptop Migration
This post was categorized as Apple, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing
  • Processing and Scheduling Thin Provisioning is another entry in my State of the Art Thin Provisioning series
  • I posted another napkin-tastic infographic: Real-World Port Throughput Relative To Light Peak
  • Wednesday saw the Thunderbolt leak, Will Apple Call Light Peak “Thunderbolt”?, but USB 3.0 For Mac Is Here (sort of)
  • Then there was the real reveal on Thursday: Thunderbolt = Light Peak = Mini DisplayPort + PCI Express
  • I had to dispel some silly speculation: Why the iPad 2 Won’t Include Thunderbolt
  • I didn’t write the post, but I did write the DCIG 2011 Small Enterprise Storage Array Buyer’s Guide!
• Other stuff
  • Good stuff from Nasuni: Blocks vs. Files: Which Approach is Better for Cloud Gateways?
  • IPv6 was a topic of discussion at Tech Field Day, so this one from Eric Siebert was interesting: IPv6 support in vSphere
  • Terry Slattery also has some IPv6 ideas: Are You Ready for IPv6?
  • Another great TFD post from Jeff Fry: IP Address Management (IPAM) and INFOBLOX
  • A response to my VAAI questions from IBM: XIV VAAI Support – whats the story?
  • I’ve been wondering about this. What’s your opinion on Gabrie van Zanten’s post? To USB or not to USB, how do you boot your ESXi host?
  • And now for something completely different: Pants Down on Madison: Salinger’s Slightly Salingeresque Boswell Wants to Be Left Alone

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, February 25, 2011
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I invited two excellent guests to join me for this conversation:

1. Josh Goldstein, VP of Marketing, Cirtas
2. Andres Rodriguez, Founder and CEO, Nasuni

It was a pretty broad discussion, all packed into less than half an hour. I urge you to check out the podcast (and subscribe in iTunes, rss, or email) but I wanted to share a few excerpts. You can also read the entire transcript over at Foskett Services: I’ll be posting it as a series of articles over the next week!

Andres Rodriguez: I think one of the things that hurt StorageNetworks at the time was the fact that they couldn’t deploy the equipment in an efficient, multi-tenant way. And so, if you look at the new cloud architectures, (places like Nirvanix, places like Amazon) those systems are designed from the get-go to be shared among many, many users, and make very efficient use of the equipment and the software running it across that user base.

…

Josh Goldstein: Today, the cloud is built on top of process that are very difficult for most companies to replicate on their on. So, the price you’re paying to your cloud provider includes not just storing your data but also keeping multiple replicas of that data spread across different geographical sites.

You’re highly protected against not only a disk drive failure, but also an entire array failure or even an entire site failure where your information’s still is survived those kinds of events and is remaining accessible to your when you need it.

That’s something that for most organizations to engineer that level of reliability is extremely expensive and difficult for them. The cloud providers have been able to do that at scale and still deliver the capacity to you with that type of protection at a price point that’s really pretty amazing.

…

Andres Rodriguez: I’ll start there and I said anything that traditional storage world, we have file based systems. We have block based systems and that is exactly where Cirtas and Nasuni stand. We are the equivalent of say a company named EMC and NetApp for the cloud world. The approaches are complimentary. And they are both trying to solve the same problem. I’ll start on the file side but Josh can take on the block. But on the file level is really, look, if you want to have something that behaves very much like a file server, say like a NetApp box. It means you are going to have a file system, you want to have a protocol to export it locally on so something like CIFS, you are now going to have access to directory integration so that you can have access control. This is what makes a file server useful in the datacenter.

…

Andres Rodriguez: One of the really interesting things about that is that Josh is essentially describing thin provisioning in the classic sense. But unlike doing thin provisioning and this is the cloud is automatically provisioned. So he is essentially promising the volume. But then the volume is growing gradually, smoothly without any administrative interference. Without you having to worry about it; the volume is filling in its data as it goes. And that is one of the things we talked about in the beginning.

The cloud really allows you to build a different type of storage system, because automatic provisioning is really thin provisioning should be but isn’t. When you’re talking about physical linked arrays that are limited by actual hard drives, you know, running, spindles running in your datacenter.

Again, you can watch the video of the whole conversation right now over at Foskett Services, or you can read the transcripts that will be posted there over the next few days.

Disclaimer: I worked at StorageNetworks, and Nasuni is a client of Foskett Services.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Talking Cloud Storage Gateways With Nasuni and Cirtas
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing
  • Cloud storage gateway eases transition to cloud storage services
  • Eleven Tech Trends To Watch In 2011
  • How To Write To Windows NTFS Drives In Mac OS X 10.6 “Snow Leopard”
  • Introducing “Raising the Floor”, Our Datacenter Podcast
• Enterprise tech
  • NetApp Snapshot: News On NetApp – Ep 4
  • December #SANchat – 11 Trends for 2011
  • Fujitsu Teams up with NetApp to Design a Storage Blade
  • The Scaling Limitations of Etherchannel -Or- Why 1+1 Does Not Equal 2
• Other geeky stuff:
  • The Curse of Beta Hardware
  • Full Body Scanners: What’s Next?
  • Useful applications for the PC-to-iPad convert
  • CradlePoint CTR-500 Review

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Back From the Pile: Interesting Links, December 3, 2010
This post was categorized as Enterprise storage, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing
  • 9 Ideas To Make Your Blog More Reader-Friendly
  • Creating Web-Friendly Collateral For Product Launches
  • The Best Black Friday Deals For Nerds Like Me (2010 Edition)
  • Do Not Buy Weird, Cheap, Off-Brand Android Tablets!
  • Overcoming The Limits Of Thin Provisioning With Automatic Provisioning!
  • What To Look For In A Laptop (Holiday 2010 Edition)
  • Are Hybrid Hard Drives A Good Alternative To An SSD?
  • Buy The Speedy Seagate Momentus XT 500 GB Hybrid Drive For Under $120!
• Enterprise tech
  • NetApp asks rival to cease and desist FUD, while EMC and IBM fight
  • Storage Caching vs Tiering Part 2
  • Saving our data from digital decay
  • All the Space You Need: The Advantage of Thin Provisioning With Cloud Storage
  • Competing With Cisco
  • Overland Offers 2-for-1 Storage Capacity for SnapServers
  • Storage biggies lining up to buy Seagate?
  • Novell Rides Off Into The Sunset
  • Cisco IOS Hints and Tricks: FCoE, QCN and Frame Relay analogies
• Cool geeky stuff
  • Binary marble adding machine
  • Worst gadget ever? Ars reviews a $99 Android tablet
  • The Muppets – Popcorn
  • How to Bypass Carriers Apple-Style
  • Review: Aerobie AeroPress coffee maker

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Back From the Pile: Interesting Links, November 26, 2010
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Most thin provisioning solutions are colossal hack jobs. How about real automatic provisioning instead?

I’ve never been a fan of thin provisioning as a storage management tool. Don’t get me wrong, I love having thin provisioning in my toolkit to overcome the limitations of conventional filesystems. Thin provisioning just gets under my skin when folks try to use it to solve business problems like long deployment time and slow purchasing cycles. If you attended any of the thin provisioning sessions I’ve presented at Storage Decisions, Interop, E-Storm, or elsewhere then you’ve heard my wistful dreaming of real automatic provisioning without the hackery of thin provisioning systems. But perhaps I didn’t mention that actual automatic provisioning actually exists today! It’s one of the many things I love about API-driven cloud storage!

Why Shouldn’t People Use Thin Provisioning?

I’ve gone into this many times, but I probably said it most-succinctly in my post, Use Process Solutions For Process Problems, Technical Solutions For Technical Ones:

I have a long-standing love/hate relationship with thin provisioning, one of the many proposed technical solutions to the utilization problem. Thin provisioning eliminates many technical challenges: It simplifies adding capacity to the Drobo that serves as my home office storage center; the ability to automatically grow VMware images makes virtualization practical in the tight confines of a laptop; and it contribute to the usefulness of advanced solid-state storage systems like the new Nimbus S-Class. But I have serious reservations about using thin provisioning to over-subscribe enterprise storage systems due to failures of capacity planning and IT-to-business communication. Thin provisioning will only make process issues worse.

This isn’t always the case, of course. Some people (including me) use thin provisioning to solve technical issues relating to inflexible filesystems. As Mr. Backup notes, you shouldn’t go around saying that everyone using thin provisioning is stupid. It’s just that some people (and vendors) over-rely on thin provisioning and use it to cover up a more-serious business problem.

So What’s Really Wrong With Thin Provisioning?

The biggest problem with most thin provisioning implementations is that they’re not really all that thin most of the time. I don’t blame the storage vendors: It’s really, really hard to “do” thin provisioning with conventional filesystems and block storage.

I spend lots of time talking about this in my “thin session” presentation, but I’ll sum it up here:

1. There’s a lack of information exchange between the application, file system, volume manager, and array controller so no one “knows” what to thin and when
2. De-allocating on delete is a pain because most filesystems don’t really delete data
3. There are two thin provisioning options, neither of which are simple or bulletproof:
   1. You can add smarts to the server, reporting back to the array when data is deleted
   2. Or you can add smarts to the array, snooping on the filesystem or reclaiming zeroes
4. Thin provisioning granularity has an impact on effectiveness, but not as much as you might think
5. Most arrays use lazy and ineffective thin reclamation; it takes real engineering to have in-line reclamation that doesn’t kill performance
6. Filesystems just weren’t designed to be thinned – they fragment, have alignment issues, etc…

It all boils down to a simple fact: Conventional systems expect to be stored “fat” on local disks, not thinned and virtualized and mangled. It’s possible to make it work, but takes so much engineering and processing power that you start wondering if it’s all really worth the bother.

We Need Automatic Provisioning Instead

I have a dream for automatic provisioning rather than reverse-engineering filesystem layouts, adding shims and semaphores, and hunting for zeroes. I want a “storage platform” that has a concept of data stored and deleted and allows applications to communicate needs beyond basic provisioning. In short, I want cloud storage.

Most people get so bound up thinking about the “cloud” part of cloud storage (service providers, REST, public/private, etc) that they overlook the obvious “storage” benefits! Cloud storage protocols enable applications to do amazing things with storage, decoupling them from the old assumptions about “my disk” and “my filesystem”. Yes, the Internet has a speed limit (both for throughput and latency). Yes, cloud storage is more expensive on a per-used-bit basis than on-site hardware. But these limits evaporate when one looks at total cost of ownership or deploys local equipment.

Real automatic provisioning is enabled by cloud storage access methods, made real by cloud gateways, and goes way beyond what any conventional thin provisioning system is capable of:

1. Imagine actually paying for exactly and only the storage capacity you are using! I use cloud storage services from Amazon and Nirvanix to host all of the images on this blog as well as the Field Day Roundtable Podcast video files. There is zero waste here: I’m paying only for used capacity and data transfer, and not a dime for empty space.
2. Imagine having unlimited scalability with no migrations! I don’t care where my service providers store my data and what equipment they use. They have an SLA to meet, and they’ve always met it. Internal or private clouds could do the same, liberated from the vendor support matrix lock-in game.
3. Imagine having your “bill” immediately reduced as soon as you delete data! I can temporarily use all the capacity I want, delete what I no longer need, and even switch providers with zero capacity “inertia” and clean-up. Public and private providers can immediately repurpose that capacity without any kind of reclamation process.

One of the dings against cloud storage is that it’s hard to start using. That is, since it uses a “proprietary protocol” it can’t be used by real-world applications. Well, more and more applications are supporting cloud providers directly. And cloud gateway products from companies like Nasuni, Cirtas, StorSimple, and others abound, allowing regular applications to go there, too. As Nasuni’s Rob Mason says, “there’s always more space available.”

Stephen’s Stance

We need to “get past” thin provisioning and the rest of the technical cruft that comes from using outdated concepts like block storage and filesystems. We need revolutionary storage that lets applications communicate in more-effective ways about capacity usage, retention requirements, replication, and similar needs. In short, we need cloud storage for a whole lot of reasons other than outsourcing of management. And it’s getting easier and easier to use cloud storage thanks to integration, cloud gateways, and the fact that the APIs are really simple and easy to use!

Watch my presentation, Extreme Tiered Storage: Flash, Disk, and Cloud for more detail on this topic!

Disclaimer: Although this is not a paid, sponsored, or for-hire article, Nasuni is a client of my consulting company, Foskett Services. I worked for Nirvanix and remain their customer. I am also a customer of Amazon and former customer or Rackspace. Symantec, maker of the Veritas storage products including the Thin API referenced here, is also a client and sponsor of Gestalt IT activities. I have lots of storage array companies as occasional clients and sponsors, including many like 3PAR, HP, Data Robotics, and EMC who make thin provisioning arrays.

Image Credit: “Push Pad to Open Automatic door. Right…” by @davestone

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Overcoming The Limits Of Thin Provisioning With Automatic Provisioning!
This post was categorized as Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

IT is crossing a chasm, and we may not be prepared for the new direction we are taking

I’m an IT revolutionary. I talk all the time about the quaint backwards “state of the art” in enterprise IT, what with its (many) decades old protocols, paradigms, and practices. What we call modern is really just a charade of faked-out old-fashioned open systems infrastructure: Pretend servers talking to fake disks over frankenstein networking technology.

What is Flexible IT?

I’ll be doing a “Twitterview” about Flexible IT today with NetApp. While I’m sure most companies would like to define flexible IT as “what we have in our product catalog,” I’m not going to play that game. IT has to change its ways or The Techie/Business Schismwill get us all.

I see the roadmap ahead as one journey over two paths:

1. Tactically, IT infrastructure groups must immediately begin implementing technologies that are responsive to user needs and focused on improving the performance of the business applications they support. This is what NetApp is calling Flexible IT.
2. Strategically, IT infrastructure must change entirely, implementing revolutionary platforms to support the next-generation apps that are being widely developed. Call this “cloud” if you must, but just about everything we take for granted today will lose significance in this new era.

Flexible IT (as it stands today) is tactical. IT infrastructure must be entirely virtualized to enable “right now” provisioning and reconfiguration. This means embracing enabling technologies like server-side hypervisors, blade computing, converged networking and I/O, and advanced storage technologies.

What Does it Mean to be a Service Provider?

Automobiles used to be raw and mechanical, but today the mechanical elements are obscured by a covering of user-friendliness and hands-off service

But real IT flexibility will mean higher-level abstraction. The next applications won’t want to run on “a server” – even a very flexible one. These apps will require a programming platform that abstracts away the entire concept of “server.” Although there will always be servers and disks and such, they will fade into the background, hidden under a layer of application services.

You know how some people like to think of themselves as “service providers”? They’re on the right track, though I’m not sure that they (yet) grasp the full significance of that concept.

A service provider has to provide the services required by consumers or it will fail. This means IT infrastructure has to prepare the environment demanded by IT application developers, rather than just shuffling around the chairs and tables and hoping customers will stream in.

What happens when the IT applications group asks the IT infrastructure manager to implement Microsoft’s Azure Services Platform? What if they want to run on App Engine or Force.com? The head-scratching in the VMware community about SpringSource is a sign that IT infrastructure just doesn’t comprehend what’s happening (yet).

Stephen’s Stance

This is a strategic shift, and we have to be ready. Being ready means changing not just what we do but also how we do it. The systems we employ in a decade might look familiar inside, but the process of delivering services on them will be entirely different. Making this shift requires real flexibility from IT, not just another round virtualization fake-outs.

Image credits: Highway construction Guizhou S0404 by sweart, VW engine by Grant Foskett

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Flexible IT and the Path to the Services Future
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.