You may not know Caringo, but you have probably heard of cloud storage, EMC Centera, and Dell DX. Read on to learn the link!

Now that the hype of “cloud everything” is subsiding, organizations are getting down to work deploying cloud storage to do actual useful tasks. The march from CAS to cloud to object storage has seen high-profile high-end flare-ups (think EMC Centera and Atmos) but the bulk of work is done by more pedestrian (think lower-cost) hardware and software. Through it all, Paul Carpentier has been at the forefront. Now his company, Caringo, is back in the news, delivering much-needed storage service features like multi-tenancy, named objects, and dynamic caching.

For essential background, check out my article, From CAS to Cloud: Revolutionary Storage

The Back-Story of Caringo

The “Caringo” company name refers to its three founders, CTO Paul Carpentier, President Jonathan Ring, and CEO Mark Goros. Carpentier is the man behind CAS pioneer FilePool, which EMC acquired and markets as Centera. The three formed Caringo and launched the CAStor product in 2006 as a software alternative to Centera.

Although you may not have heard of Caringo, you may have encountered their product in the form of the Dell DX object storage system. Many were puzzled when Dell, known for its EMC-powered storage offerings, embraced Caringo for object storage, but those familiar with the products weren’t surprised. Caringo’s approach is much more in line with Dell’s image of affordability, simplicity, and commodity products, and their relationship with EMC is increasingly shaky due to their recent acquisition strategy.

Caringo’s CAStor is a software product that transforms commodity servers into a scale-out object repository. It is all-inclusive, with compliance, tiering, spin-down, and replication part of the total package. Like most CAS and cloud storage solutions, CAStor uses a simple HTTP interface for client access, with “gateways” available for NAS along with some native support from applications.

What’s New in CAStor 5?

Caringo has set a course for the service provider market, adding essential features like multi-tenancy and flexible permissions to version 5 of CAStor. Although still pitched as an object store, CAStor 5 is close enough to be thought of as a cloud storage platform.

The ability to support and segregate multiple “tenants” is a holy grail for service provider storage systems and a key ingredient of cloud storage solutions. CAStor 5 can be segmented into multiple domains, each with its own security and authentication and each subdivided into “buckets” for different applications. This would be useful both for a public service provider and an internal-only solution, since segmenting applications is relevant in the enterprise as well.

CAStor 5 no longer clings to system-assigned names for objects, allowing users to assign their own names for public consumption. This is a huge advancement for CAS, and was one of the key differentiators of cloud solutions which often directly serve content to web clients. Another “ripped from the cloud” feature is dynamic caching, allowing high performance access to popular content, again useful for direct client access.

Stephen’s Stance

Caringo seems reluctant to wear the “cloud storage” mantle, but their product has been steadily moving in that direction. CAStor 5, with its multi-tenancy, segmented security and authentication, named objects, and caching, looks an awful lot like Amazon S3 and the rest. But the hype around “cloud storage” is dying away. Businesses looking for functionality rather than marketing labels will find a lot to like in CAStor.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Caringo Bulks Up CAStor For Cloud Services
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

We need to move beyond fake disks and deploy application-centric storage

Change is not a word normally associated with storage, and revolution is practically unheard of. Today’s modern enterprise storage systems and networks employ massive resources to do one simple thing: Emulate the basic hard disk drives used over three decades ago. But cracks are appearing in our mausoleum of fake disks: Application developers are discovering the value of object storage, and storage systems are appearing to support this need.

I also wrote about this two years ago, proclaiming that We Need a Storage Revolution and forecasting The Techie/Business Schism

The CAS Revolution

Caringo founder and CTO, Paul Carpentier, rose to prominence around 2000 at FilePool, one of the prime movers in the content-addressable storage (CAS) space. I recall a light going off in my head as Paul introduced me to FilePool’s CAS technology back then, imagining the possibilities of the concept. Files would be stored based on “what they were” rather than “where they were” and could be organized according to application needs rather than the conventional “extent of blocks” or tree heirarchy.

CAS discarded decades of filesystem and block storage baggage, introducing a new method for storing and retrieving data that better-matched the burgeoning web and enterprise applications of today. I had seen the failure of the first wave of storage service providers from inside StorageNetworks, and it was this desire for a real storage revolution that led me to dive into cloud storage at Nirvanix almost a decade later. Although I am now on my own, I remain convinced that the future belongs to storage systems that look nothing like today’s SAN and NAS.

Shortly after that 2001 meeting, EMC acquired FilePool and launched it as the Centera product line. But CAS systems quickly ran into a serious roadblock: Conventional applications cannot read and write to unconventional storage systems like Centera. EMC pushed key software vendors (especially in the archiving space) to create special Centera interfaces, and the industry bogged down developing the XAM standard. Other companies, like Seven Ten Storage Software, jumped in to help with the translation from proprietary CAS interfaces, but the transition from legacy files and blocks to object storage has been long and slow.

Cloud Storage: Another Dimension

Meanwhile, in an alternate dimension, web developers realized they had a serious problem. They were developing applications that scaled massively, spanning servers and exhausting conventional filesystems. Conventional systems just wouldn’t cut the mustard.

Since they were soaking in web applications, these developers applied the lessons of web services to storage: Why not just make an HTTP connection and ask for an object by a unique ID rather than walk a filesystem tree? Why not encapsulate the “state” of this request in the request itself rather than make a lasting connection and association between the client and server?

Thus was born cloud storage, and it was bookseller Amazon who opened the floodgates with their 2006 introduction of a “Simple Storage Service” or S3. They allowed anyone to store and retrieve objects from their massive web services infrastructure. S3 and similar services from Rackspace, Nirvanix, and others, are special-purpose web servers, and their simple interfaces are wonderfully attractive to web developers. For example, this WordPress-based blog uses cloud storage to serve images to your browser!

Similarities in CAS and Cloud

Although developed from vastly-differing starting points, CAS and cloud storage are essentially similar: Both reject conventional blocks and files in favor of object storage; both organize data with metadata databases; both multiply and scale out. There is one other major similarity between CAS and cloud storage: Both are attractive to service providers.

Imagine you operate a business that stores data for customers. You would want a flexible infrastructure that would scale with demand and segment each “tenant” from others for security and performance. As we learned at StorageNetworks, conventional SAN and NAS systems just weren’t meant to work in this kind of environment. Whether operating an internal service or a public cloud, service providers require something entirely different.

Cloud storage was designed from the start with service providers in mind, embedding per-object and per-”bucket” security, scalability, and abstraction between hardware and clients. Although quite complex to design, cloud storage is amazingly simple to use, provided an application can interface with it.

CAS wasn’t designed like this. Systems like EMC’s Centera were created for the needs of applications like enterprise archiving, but secure storage of content and extreme scalability are critical here as well. But early CAS systems didn’t need simple web-style interfaces or extreme hardware abstraction. These were enterprise systems, after all.

The CAS/Cloud Colission

CAS wasn’t exactly successful. Although object storage found a niche in enterprise archiving, the enterprise storage world has mostly continued with blocks and files. The major storage vendors all have some kind of object storage, but most are repurposed NAS rather than dedicated CAS like the Centera.

Although much skepticism has been raised about cloud storage in the enterprise, its impact on application development cannot be denied. Indeed, the majority of developers are now focused on programming platforms that abstract both compute and storage from conventional operating systems. The next generation of applications will run in “platform as a service” environments first, and cloud storage is a key component.

Storage vendors are rapidly moving to rework their conventional systems for cloud use. Although block and file systems from 3PAR, NetApp, Isilon, Symantec, HDS, HP, and others are useful in cloud environments, unconventional CAS becomes more valuable here. This is where EMC, Mezeo, and Caringo (with Dell) shine, and why HDS bought Parascale, NetApp bought Bycast, and what Overland could do with MaxiScale. In the mean time, “gateway” products from Nasuni, Cirtas, StorSimple, Twin Strata, and Asigra are awfully interesting.

Stephen’s Stance

The storage revolution is coming, whether we in the industry are ready or not. Developers are voting with their feet, targeting cloud storage and application platforms rather than conventional filesystems. Although the market for cloud storage products is slow to develop, the cloud storage concept will eventually dominate the landscape.

It seems most likely that this revolution will decimate the storage industry as we know it today. Unable to push high-margin storage arrays into the ballooning cloud space, product vendors will see their market share eroded by service providers with no use for these expensive systems. Monolithic file and block will soldier on in the new legacy applications, but the action will inevitably slip away.

The likely winners will be those who can leverage commodity hardware for scale-out cloud storage use. The proliferation of cloud platforms will settle down, with a few gaining traction and the rest discarded. Then we will see companies like HP, Dell, and Oracle rise to lead the storage sales charts with massive volume shipments to service providers.

Disclosure: I used to work for StorageNetworks (which is now defunct) and Nirvanix.

Image credit: Barcelona Graffiti by Aeioux

© sfoskett for Stephen Foskett, Pack Rat, 2010. | From CAS to Cloud: Revolutionary Storage
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

On the contrary, I suggest that the enterprise storage world we know was shaped by singular innovative products of the past. Without these, the IT world might look very different.

While you’re at it, check out my list of the coolest enterprise storage flops!

So let’s take a walk through history, identifying the ten most innovative and important enterprise storage hardware products. But let me note first that this list could be 100 items long, and we all have our favorites. Lots of the storage blogging world contributed their ideas, too!  So I’m setting some arbitrary rules to keep myself on track:

1. The products have to be reflected in the shape of modern enterprise storage for open systems. If their day came and went, they’re not listed here.
2. Hardware only – I’m working on another list for software!
3. Listed products have to have been successful in the market (or they’d probably fail rule number 1 anyway), so I’ll put together a list of cool flops later.
4. For product families, I tried to pick the most influential and innovative 2member.
5. This is an enterprise storage list – items have to be used in big companies, not little PCs.
6. Only one product per company (sorry, IBM and EMC!)

So without further ado, on with the list!

1964 IBM 2311

Let’s get one thing straight: IBM is responsible for much of modern enterprise computing, so it’s no surprise that when it comes to storage, they developed just about everything, including disk drives, floppies, tapes, arrays, and libraries. So why pick this particular piece of kit? Prior to 1964, computer components were developed as a set (the 350 storage system went with the 305 RAMAC, for example), but the 2311 changed that. It was a generic storage device, plug-compatible with a range of computers. If you claimed that the 2311 was the common ancestor of all modern enterprise storage, I would certainly agree with you!

Other notable IBM products worth mentioning:

• The original 1956 RAMAC 350 disk drive unit
• The 1961 301, which used a separate arm and head for each platter (like every disk drive produced since)
• 9-track tape, which dominated from 1964 until the 1/2-inch tape revolution in the 1990s
• The 1970 3330, which added error correction and remained in production for 13 years
• The 1971 introduction of the 23FD floppy disk drive
• The 1973 3430, whose “30/30″ code name caused people to refer to hard drives as “Winchesters”
• 1974′s 3830 “MSS” tape library
• The 1980 3380, which introduced film-head technology
• 2003′s SAN Volume Controller, the first successful SAN virtualization product (after many others failed!)

1976 Shugart SA-400 minifloppy

Alan Shugart’s floppy drive was a massive hit in the home computing market, but why include it in a list of enterprise storage technologies? Because its storage interface set the standard for plug- and protocol-compatible storage in the nascent microcomputer world.

Designing a general product for use in a multitude of systems was truly innovative, and many later computers were literally designed around both the concept and physical form factor of Shugart’s drive. Simply put, this fat floppy drive inspired computer designers to create computers that could accept standard peripherals, the very definition of open systems. The definition of “peripheral” would soon grow to include standard I/O devices like storage, terminals, printers, communications gear, and everything else we know today.

1980 Seagate ST-506

Shugart left his company in 1979, founding Seagate Technology. That company’s first product was the ST-506, a 5 MB hard disk drive that shared its physical shape with the SA-400 and used a similar interface.

Like the floppy, Shugart’s hard drive set the standard for microcomputers, eventually finding its way into enterprise systems, and catapulting Seagate to its current position of disk drive leadership. Higher-capacity derivatives of the ST-506 were fitted with interfaces using Larry Boucher of Adaptec’s SCSI protocol, which continues in use even today. Although the ST-506 wasn’t a “smart” drive, the ecosystem that developed around it was critical.

1984 DEC TK50

Digital Equipment’s introduction of the MicroVAX II in 1985 was accompanied by a new half-inch backup tape drive and cartridge called the TK50. Open-reel tapes had dominated non-disk storage before this, but cartridges quickly replaced reels, leading to the development of tape robotics and (more) reliable off-site storage.

If you picked up a TK50 tape today, you could be forgiven for mistaking it for a modern SDLT or LTO, because these use a similar cartridge and drive form factor, the same linear tape technology, and the same half-inch tape size. The TK50, DLT, and SDLT were the mainstays of open backup for decades.

1987 Auspex

This terrible photo is an actual Auspex press image!

What was the first open systems enterprise storage array? Leveraging Sun’s NFS protocol, and hiding a Sun workstation inside, Wizard of Oz style, the Auspex set the standard for everything we think of as “enterprise” in the open systems world. The company’s storage systems were truly ahead of their time, with ranks of redundant disks years before RAID became common.

Auspex was founded by Larry Boucher, father of SCSI and founder of Adaptec, and raked in sales while others struggled to figure out how to sell in the enterprise.  But their refusal to produce a smaller device would be their undoing. Today’s monolithic arrays owe as much to Auspex as they do to IBM, but the companies producing them could learn a lesson from the company’s demise.

1987 StorageTek 9310 PowderHorn

StorageTeks versatile and scalable 9310 PowderHorn defined backup

StorageTek was instrumental in many enterprise storage developments (see McData, for example), but one product literally transformed the datacenter: the 9310 “PowderHorn”. Consider the “glass house” datacenter tours that the largest companies would use to impress visitors: They would show off their mainframe, their Cray, or their PowderHorn.

This versatile system would accommodate every major tape cartridge format and system type and could scale to truly massive proportions. When disks were expressed in megabytes, PowderHorns held terabytes. 

1995 EMC Symmetrix 3000

EMC's Symmetrix 3000 family brought mainframe block storage technology to open systems

The 1990 introduction of the Symmetrix was the key turning point for the (now) giant of Hopkinton, but the first two generations were mainframe-only. In 1994, the company delivered data replication capability in the form of SRDF, moving key enterprise functionality to the storage array.

But it was the 1995 introduction of the third-generation Symmetrix 3000 that really changed the storage world. For the first time, open systems could connect to mainframe-class storage over the standard SCSI protocol and leverage features like SRDF and (in 1997′s Symm 4) TimeFinder.

One key ingredient often overlooked in the Symmetrix was in-box virtualization the likes of which hadn’t been seen before. It also featured RAID-like sub-disk data protection, though the Symmetrix line never did implement true RAID.

The Symmetrix was redesigned entirely in 2003 to become the DMX. Although it was as different from its predecessor as the New Beetle was from Volkswagen’s original, the DMX line continued many of the philosophical underpinnings set in 1990.

EMC also deserves credit for their original Celerra enterprise NAS system, which picked up where Auspex left off. The company followed these in 2002 with the Centera CAS system, which abandoned many traditional concepts of enterprise storage, like blocks- and filesystem-access and modular and monolithic architecture. The Centera is certainly innovative, but we have yet to see the impact it will have.

1994 Data General CLARiiON

Data General paired the CLARiiON (left) with their AViiON server

It’s easy to forget the world before RAID. But Data General was one of the first to market with a cached RAID system with their 1991 introduction of the HADA. Although this early system was tied to DG’s servers, it donated much of its architecture to a system that became more valuable than the rest of the company: CLARiiON.

Massively successful, and mighty impressive (PC Magazine called the first CLARiiON “amazing”), this modular block storage array set the template for over a decade. There were dozens of copycat arrays on the market within a few years of the introduction of the HADA. The CLARiiON gained Fibre Channel support, was sold to EMC, and remains a mainstay of corporate data centers, albeit with updates to every component. I have administered every generation of CLARiiON array, and can attest to their capability (when properly configured!)

1996 NetApp Multiprotocol Filer

NetApp's F330 Multiprotocol Filer was a huge hit in mid-size businesses and a major upgrade

NetApp (nee Network Appliance) was formed by a group of ex-Auspex engineers who wanted to create a more modular NAS server based on industry-standard hardware. They released their first product, the NFS-only FASServer, in 1995, but it was their August, 1996 introduction of the Windows-compatible Multiprotocol Filer software that really put them on the map.

Combined with their solid F220, F330, and F540 hardware, NetApp now had a serious challenger to Auspex, and their NAS systems blew away dedicated server-based solutions in terms of flexibility and manageability. NetApp’s unified NFS, CIFS/SMB, and HTTP access to content on their unique WAFL file system was impressive at the time, as was the quick setup and administration and the ease of adding drives to their RAID-4 sets. Plus, they brought a new level of friendliness to the data center with their bright colors and silly “toaster” nomenclature.

1999 McData Fibre Channel Director

McData has an interesting history, beginning with Storage Technology (StorageTek), intersecting with IBM, bought and spun off by EMC, and lately acquired by Brocade. Although the company spent a decade developing various peripherals for IBM mainframe systems, their late-1994 introduction of a switching director for ESCON traffic would change the storage world. We take large-scale SANs and LANs for granted today, but McData’s director was astonishing when it was introduced: There were no words to describe it or its function at the time, and period press articles are puzzling!

EMC scooped the company up a year later, but McData was spun out in 1997, adding Fibre Channel support that same year. In the ensuing years, McData became the enterprise SAN “arms dealer”, supplying IBM and EMC with the ED-5000 and ED-6064 directors. I recall commenting at the time that Fibre Channel connectivity built around the McData director was the first Storage Area Network worthy of the name. McData went IPO in 2000 and was purchased by rival Brocade in 2006. McData’s director architecture survives and thrives today against fierce competition from Brocade, Cisco, and others.

Others To Consider

What would you have included in this list? Here are some honorable mentions that I wish I had room for:

1. StorageTek “Aegis” L700 library
2. Brocade “LOOM” 2xxx switches
3. Adaptec SCSI HBAs
4. Copan MAID
5. EqualLogic iSCSI arrays
6. EMC Centera
7. HP EVA
8. HDS USP
9. Compellent Storage Center

Responses

Here’s a list of responses that others have posted:

• Marc Farley’s Top 10 List
• Alex MacDonald’s Bottom 10 List

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Top Ten Innovative Enterprise Storage Hardware Products
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Great! But I thought the new era for IT was a cloud with choice, mobility, and application support, as trumpeted by EMC’s VMware subsidiary! Wasn’t Cloud vServices from VDC-OS supposed to be the prototype cloud strategy for the datacenter?

What we have here is a simple clash of marketing amusingly taking place at (nearly) the same company. VMware figured out how to extend their server virtualization products outside the confines of the data center, and laid that technology out as a strategy with the trendy “cloud” name. Meanwhile, mother EMC is working on next-generation content storage software and decides to roll that out as a strategy and also jumps on the “cloud” meme. What’s an IT manager to do?

Defining Atmos

As predicted, EMC’s Atmos (code-name Maui) is a distributed software layer to handle the storage and management of data objects across geographically-dispersed storage devices. EMC’s Chuck Hollis demonstrates Atmos with a simple, practical example, perhaps making it sound too much like Akamai but generally getting the point across. You have a data object, write it to Atmos through REST/SOAP or CIFS/NFS, assign some metadata, and the software takes care of data placement for you. It’ll add local copies, replicate for availability and performance, compress or deduplicate, manage versions, and all sorts of goodies (if you ask it to).

But EMC already has a capable object storage platform, the Centera. We’ve just got used to the content-addressable storage (CAS) label for object storage (even though this name misses the point of object storage, in my opinion) and now EMC wants us to learn a new label for a somewhat-similar device? Steve Todd, EMC’s object guy extraordinaire, lays it out:

SAN Value = Centralized, secure multi-tenancy for blocks.

NAS Value = Centralized, secure multi-tenancy for files.

CAS Value = Centralized, secure multi-tenancy for objects (content + metadata).

COS Value = Globalized, secure multi-tenancy for content with rich policies.

Ok, so the defining capabilities of Atmos are its global scale and rich policies. And the fact that “objects” has become “content”, presumably since Atmos can handle traditional NAS (CIFS/NFS) chores as well.

Prayers Answered?

It sounds like EMC is answering my prayers for a storage revolution, delivering a highly-capable object storage platform that transcends the old limits of blocks, directories, and files. Steve Todd points out that Atmos handles five policy categories out of the box:

• Replication
• Compression
• Spin-down
• Object de-dup
• Versioning

So we write some data to Atmos, using either traditional NAS or webby dubby protocols like SOAP, and can then apply policies in any of these five categories to that data. One can also extend the Atmos to accept other policies, but the absence (out of the box) of concepts like encryption, secure deletion, retention, and access control are surprising.

I am quite puzzled about how practical these policy capabilities will be in the real world. How exactly would an application say “I want you to compress that file I wrote over NFS just now?” Hitachi’s HCAP platform, for example, also has policy capabilities and a NAS front end, and although archiving applications can communicate their policy needs, I don’t see lots of current general-purpose applications using it.

Strategic Storage?

This brings me to my puzzlement: The default Atmos policies are all general-purpose, production computing ideas, not the special-purpose, archiving and retention needs served by Centera, HCAP, and the rest. So the Atmos is clearly intended to be a production data storage system, not an archiving system to compete with Centera.

Since mainstream business applications currently don’t have any capability to specify policies like these when writing files, and since NAS protocols lack any means to communicate them even if the apps want to, we can conclude that EMC expects that Atmos users will write special applications to take advantage of it.

EMC certainly doesn’t expect that the NAS-capable Atmos will simply replace today’s distributed NAS solutions. NAS is a sideshow for Atmos. The real action will be in the REST/SOAP webby dubby applications that will be written with the platform in mind and will take full advantage of these capabilities.

If this is true, and I and others suspect that it is, then Atmos really isn’t a game-changing platform unless you change your game. If you write new applications to store data with SOAP, Atmos is a nice in-house alternative to Amazon S3 or Nirvanix, and offers a very compelling set of data management capabilities. And if you want to set up shop to compete with those service providers, Atmos is a dream come true with built-in multi-tenancy.

Datacenter Strategy

So EMC alone has two seemingly competitive datacenter strategies. And then there’s Microsoft, which announced its Azure cloud platform recently, and Amazon and the other cloud providers.

So let’s say you’re a CIO for a large corporation. Which of the following strategies is more compelling:

1. Use VMware VDC-OS to add capabilities and Cloud vServices extend your current virtual infrastructure geographically
2. Recompile and tweak your Windows applications to leverage Microsoft Azure
3. Develop new applications to take advantage of the impressive storage capabilities of an in-house EMC Atmos system
4. Point your new applications at a third-party cloud provider like Amazon or Nirvanix

IT people are practical. Although we love new technology, we tend to be cautious. We also hate massive software development efforts, and only sanction them when they’re absolutely necessary. Given these personality traits, I’d say VDC-OS and perhaps Cloud vServices still stands out as the most likely and practical scenario for the majority of applications and businesses.

This is not to say that EMC Atmos will be a flop. I’m impressed by the technology, and expect that Atmos will find buyers, just as Centera did. And Atmos might even replace Centera once EMC adds retention policies to it and scales it down as well as up and out. But Atmos will not redefine the datacenter. We’re stuck with blocks and files, and VMware’s practical strategy is a winner in that world.

Update: Marc Farley compares Atmos to WAFS, with ominous implications, and echos my recent question on what is and is not innovative.

Update 2: Chuck Hollis, Storagezilla, and Len Devanna have all come right out and said that this is only intended for certain customers with massive distributed storage needs, and is not intended as a new datacenter strategy. Even the “cloudfella” says “ciao”:

Update 3: More great information, including a reply regarding VDC-OS and Atmos from the one and only Chad Sakac, more great detail about the inner workings of Atmos from Steve Todd, and even more info from Dave Graham. Finally, although I think that Cloudfellas video is cute, I wouldn’t categorize it as viral. But those Mozy ads are awesome!

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | EMC Atmos Versus VMware VDC-OS: Will The Real Cloud Strategy Please Stand Up?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although deduplication of storage is nothing new, with Data Domain and other making hay with the technique for years, it has never been ready for prime time – reduction of active primary storage applications like email and databases. Instead, deduplication has been relegated to second- or third-tier status, deduplicating archives and backup data. But change is in the air, and deduplication vendors are starting to bustle towards the bright lights of primary storage.

Stone Knives and Bear Skins

We have all been here before, of course. Back at the dawn of the personal computer era, data compression was a hot topic of conversation. I recall being so impressed by an article in Byte (1986:5, p99) outlining Huffman coding that I tried cooking up an implementation in Atari BASIC. Lossless compression has a magical pull to the geek in many of us – redundant data just wants to be eliminated!

Stacker dominated the disk compression world - until Microsoft introduced DOS 6.0

Companies soon applied compression to primary storage, especially the limited storage in personal computers. Stacker was a hit after 1990, until Microsoft built a workalike, called DoubleSpace, into DOS 6.0 in 1993, leading to a historical lawsuit. I personally used the ADDSTOR disk compression built into DR-DOS 6.0 to stretch two more years out of the 20 MB MFM hard drive in my AT&T PC6300 at WPI.

But something funny happened in the late 1990s: Compression began to lose its luster. Compressing data always takes quite a bit of CPU power, but this was offset somewhat by the truncated data transfers and more-efficient file system layout afforded in early PCs. But as disks got larger and faster, using precious CPU time to save space seemed less and less compelling. Today, although nearly every operating system includes built-in compression of files, folders, or perhaps disks, these features are rarely used. And compression was never popular in the performance-sensitive enterprise space.

Deduplication Has a Nice Ring

Although traditional fine-grained compression has not been very successful in the enterprise, its lanky cousin, single-instance storage, has long found niche jobs. Applications from databases to email systems to file servers have long had the ability to recognize to requests to store the exact same file or record, and to store just a single instance in this case. Even file systems have the ability to do single instance storage through the use of links, though this is initiated by the user rather than in an automated fashion.

In the late 1990s, FilePool began developing a content-addressable storage device, which was acquired by EMC in 2001. This device, later known as the Centera, was one of a number of storage platforms targeted at the archiving market introduced this decade. At the same time, virtual tape libraries made the jump from the mainframe to open systems. Both devices, being outside the critical path of performance but offering massive capacity, were well-suited to implement advanced capacity optimization technologies that combined the concepts of compression with single-instance storage. Thus was created the modern world of data deduplication.

What we think of as deduplication is neither fish nor fowl: It assesses larger “chunks” of data than compression technologies, delivering greater capacity savings and potentially reducing performance impact, but is more flexible than single-instancing, recognizing the similarities within files or objects.

But it is still maddeningly difficult to scale deduplication while maintaining performance. Rather than fight to maintain reasonable write throughput, most deduplication products have switched to post-processing, deferring their work to quieter times.

It’s Not Just for Breakfast

Regardless of their methods or underlying technology, no deduplication vendor has stood up to support challenging low-latency or high-throughput production applications, however. NetApp was the first to raise the issue of support for production applications, but although they tout the technology for VMware, they haven’t exactly been shouting from the rooftops to get their A-SIS deduplication technology deployed in other high-I/O applications. And I haven’t seen Hifn’s card yet.

Yesterday, I mentioned that greenBytes was adding deduplication to their ZFS-based storage array for primary data. And now Riverbed has fired another shot over the bow, repurposing their (deduplicating) WAN accelerator product for primary (file) storage. They might be able to pull it off, too, since they have a long list of customers who are already enjoying the technology in production. It’s not a stretch to suggest that Riverbed’s appliances can scale to handle production data loads. Although it’s file-only, I can imagine quite a few scenarios where this tech could really yield benefits. Could we come full-circle, with deduplication finally reaching the enterprise storage world?

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Deduplication Coming to Primary Storage
This post was categorized as Computer History, Enterprise storage, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.