Join me in New York for Storage Decisions, September 19 & 20

Tomorrow, I will be in San Francisco for TechTarget’s Storage Decisions conference. This show does a good job on the editorial side, suggesting timely topics and bringing in folks like Dennis Martin, Mark Staimer, and Jon Toigo. I will have two presentations on data reduction and storage virtualization in the main conference track – both are updated from my New York sessions. Registration is free for qualified end-users, and I urge you to attend.

Reclaim Capacity with Data Reduction for Primary Storage

I have updated the session with additional information on thin provisioning and compression, as well as expanding the slides to reflect many of the questions and comments I received in New York. The end result remains the same: I’m not sold on data reduction for primary storage as a product and recommend tackling data growth if at all possible. If it’s completely impractical to delete data, there are a few products that work well.

Depending on which industry study you read, most companies are wasting anywhere from 30% to 50% of their installed disk capacity, which translates into thousands of dollars spent with no effective return on investment. Storage vendors are beginning to provide tools that can help storage managers make the most of the disk they have installed. For example, data reduction for primary storage borrows data deduplication technology developed for backup and classic compression algorithms to help squeeze the air out of nearline and primary data and reduce its footprint. This session’s topics will include an overview of data reduction technologies and where they will have the greatest impact, what key storage vendors are offering in data reduction and an update on the major players, and the consequences of using primary data dedupe along with dedupe for backups. We’ll also look at the potential for vendor lock-in and consider why we’re reducing data in the first place.

Topics include:

• Introducing data reduction technologies
  • Compression: How it works and where it’s found
  • Deduplication: From single-instancing to variable block
  • Application-specific: Cracking open files
• Overview of data reduction products
• Where to use them
  • The capacity conundrum: Store less and reduce utilization
  • Ideal applications: Justifying the cost of data reduction
  • Side effects: Considering the impact on backup, replication, I/O workload and vendor lock-in

Storage Virtualization: Who’s Doing It and Why

My storage virtualization session has been massively tweaked, including updated information on standalone virtualization products as well as a more in-depth discussion of successful and failed use cases. This session presents the conundrum of why server virtualization has been so successful while storage virtualization has failed for over a decade. I believe this is due to the problems these products try to solve. Consolidation of resources and reduction of administrator effort are noble goals, but not really compelling in the long term. Unless some real is this value can be extracted, storage virtualization will continue to be a failed product.

Storage virtualization has been around for decades and, although research indicates that 70% of companies have already virtualized at least some of their installed block or file storage, most remain unaware of this technology. Grandiose schemes for comprehensive virtual SANs have given way to more practical host- and array-based virtualization technologies, and server virtualization has created a new opportunity to create a pool of storage. This session will look at the current state of storage virtualization, how to quantify its benefits and describe which approaches are best for particular environments, and also cover how storage virtualization compares to private storage clouds.

Topics include:

• Defining storage virtualization: What it is and where to find it
  • Abstraction of storage resources
  • Tiered storage
  • Flexibility
• Popular approaches to storage virtualization
  • SAN controllers
  • File virtualization
  • Volume managers
• The pool, the hypervisor and the cloud
  • The impact of server virtualization
  • Is this a private cloud?

Registration

You can see the future from here!

To register for Storage Decisions San Francisco, just go to the TechTarget registration page.

Disclosure: TechTarget pays my expenses to attend and present at Storage Decisions, and has for many years. But they don’t pay me to present and I own the copyright on my session content. Happily, I license it all CC-by-NC-SA so I can give it out freely!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Storage Decisions San Francisco 2011: Optimization and Virtualization
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Data Reduction can be hazardous to your health!

I’m not a big fan of data reduction technology, yet I found myself talking compression and de-duplication all week. Between Storage Decisions and my recent posts over at SearchStorage and The Storage Community, I’ve had quite a bit to say on the subject. Funny enough, I’m not really a fan of data reduction technology for primary storage. Too often, data reduction is more expensive and difficult than just storing raw data.

You should also read Deduplication Coming to Primary Storage and Compression, Encryption, Deduplication, and Replication: Strange Bedfellows

Storage Decisions

My Storage Decisions presentation on data reduction was hilarious, if I do say so myself, even though turnout was poor at 8:30 AM on Tuesday morning. Maybe it was this “intimate” group, but I found myself really getting into the discussion. And the nods and hollers from the audience helped, too!

My basic thesis at Storage Decisions was the same as always: Don’t throw good money at technology that will have little ROI. Considering that disk capacity is incredibly cheap, and dropping all the time, data reduction doesn’t look like a great fit except in certain situations. Why spend money to reduce utilization? Why put in the effort when most primary storage data reduction technologies don’t do anything to address the “multiplier effect” of archiving, DR, and backup storage?

This is not to say that all data reduction technology is worthless. In fact, the free compression and de-duplication built into many SSDs and even some enterprise storage devices make perfect sense. I just don’t understand spending a bunch of money to address storage capacity when most applications are starved for storage performance.

You might like reading my two other posts on the subject from last week:

• Interest in data reduction methods needs to keep pace with data growth (SearchStorage.com)
• Has the Time Finally Come for Data Reduction? (The Storage Community, sponsored by IBM)

You’re Losing Me

On the other hand, I do see quite a bit of value in something many people would overlook out of hand: Lossy compression of office files. Every systems administrator knows that end-users do “stupid stuff” like embedding massive photos and videos in PowerPoint presentations and Word documents. But not everyone knows that there are technological means to address this “PEBKAC” issue.

Some office applications already automatically reduce the size of embedded content, and operating systems can do the same. One of my more popular blog posts, in fact, is a technique to create a filter to reduce the size of PDF files in Mac OS X Preview. And the Microsoft “X” Office file formats include lossless compression as well.

An application that recently caught my eye is the FILEminimizer Suite by Balesio. This inexpensive application reduces the size of Office and media files while leaving them in their native format. It re-compresses image files, reducing them to optimum size for use in presentations, documents, or printouts. A companion product, FILEminimizer Server, can be used on enterprise file servers to perform the same magic across the whole range of users.

Stephen’s Stance

Native Format Optimization (NFO) makes a lot of sense, since it addresses a common user error in a practical way, and allows capacity savings to “trickle-down” to backups, e-mail systems, and archives. But wholesale compression and the duplication of primary storage may not be worth much, especially since the cost of disk keeps dropping dramatically.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Data Reduction: the Condensed Version
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Join me in New York for Storage Decisions, September 19 & 20

Later this month, I will be heading to New York for TechTarget’s Storage Decisions conference. This show does a good job on the editorial side, suggesting timely topics and bringing in independent voices like Howard Marks. I will have two presentations on data reduction and storage virtualization in the main conference track. Registration is free for qualified end-users, and I urge you to attend on September 19 and 20, 2011.

Reclaim Capacity with Data Reduction for Primary Storage

Depending on which industry study you read, most companies are wasting anywhere from 30% to 50% of their installed disk capacity, which translates into thousands of dollars spent with no effective return on investment. Storage vendors are beginning to provide tools that can help storage managers make the most of the disk they have installed. For example, data reduction for primary storage borrows data deduplication technology developed for backup and classic compression algorithms to help squeeze the air out of nearline and primary data and reduce its footprint. This session’s topics will include an overview of data reduction technologies and where they will have the greatest impact, what key storage vendors are offering in data reduction and an update on the major players, and the consequences of using primary data dedupe along with dedupe for backups. We’ll also look at the potential for vendor lock-in and consider why we’re reducing data in the first place.

Topics include:

• Introducing data reduction technologies
  • Compression: How it works and where it’s found
  • Deduplication: From single-instancing to variable block
  • Application-specific: Cracking open files
• Overview of data reduction products
• Where to use them
  • The capacity conundrum: Store less and reduce utilization
  • Ideal applications: Justifying the cost of data reduction
  • Side effects: Considering the impact on backup, replication, I/O workload and vendor lock-in

Storage Virtualization: Who’s Doing It and Why

Storage virtualization has been around for decades and, although research indicates that 70% of companies have already virtualized at least some of their installed block or file storage, most remain unaware of this technology. Grandiose schemes for comprehensive virtual SANs have given way to more practical host- and array-based virtualization technologies, and server virtualization has created a new opportunity to create a pool of storage. This session will look at the current state of storage virtualization, how to quantify its benefits and describe which approaches are best for particular environments, and also cover how storage virtualization compares to private storage clouds.

Topics include:

• Defining storage virtualization: What it is and where to find it
  • Abstraction of storage resources
  • Tiered storage
  • Flexibility
• Popular approaches to storage virtualization
  • SAN controllers
  • File virtualization
  • Volume managers
• The pool, the hypervisor and the cloud
  • The impact of server virtualization
  • Is this a private cloud?

Registration

You can see the future from here!

To register for Storage Decisions New York, just go to the TechTarget registration page.

Disclosure: TechTarget pays my expenses to attend and present at Storage Decisions, and has for many years. But they don’t pay me to present and I own the copyright on my session content. Happily, I license it all CC-by-NC-SA so I can give it out freely!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Storage Decisions New York: Capacity Optimization
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Join me in Chicago for Storage Decisions, June 21

Next month, I will be heading to Chicago for TechTarget’s Storage Decisions conference. This show does a good job on the editorial side, suggesting timely topics and bringing in independent voices like Howard Marks. I will have three presentations to give: Sessions on data reduction and storage virtualization in the main conference track, as well as a dinner discussion focusing on controlling the growth of data. Registration is free for qualified end-users, and I urge you to attend on June 21, 2011.

Reclaim Capacity with Data Reduction for Primary Storage

Depending on which industry study you read, most companies are wasting anywhere from 30% to 50% of their installed disk capacity, which translates into thousands of dollars spent with no effective return on investment. Storage vendors are beginning to provide tools that can help storage managers make the most of the disk they have installed. For example, data reduction for primary storage borrows data deduplication technology developed for backup and classic compression algorithms to help squeeze the air out of nearline and primary data and reduce its footprint. This session’s topics will include an overview of data reduction technologies and where they will have the greatest impact, what key storage vendors are offering in data reduction and an update on the major players, and the consequences of using primary data dedupe along with dedupe for backups. We’ll also look at the potential for vendor lock-in and consider why we’re reducing data in the first place.

Topics include:

• Introducing data reduction technologies
  • Compression: How it works and where it’s found
  • Deduplication: From single-instancing to variable block
  • Application-specific: Cracking open files
• Overview of data reduction products
• Where to use them
  • The capacity conundrum: Store less and reduce utilization
  • Ideal applications: Justifying the cost of data reduction
  • Side effects: Considering the impact on backup, replication, I/O workload and vendor lock-in

Storage Virtualization: Who’s Doing It and Why

Storage virtualization has been around for decades and, although research indicates that 70% of companies have already virtualized at least some of their installed block or file storage, most remain unaware of this technology. Grandiose schemes for comprehensive virtual SANs have given way to more practical host- and array-based virtualization technologies, and server virtualization has created a new opportunity to create a pool of storage. This session will look at the current state of storage virtualization, how to quantify its benefits and describe which approaches are best for particular environments, and also cover how storage virtualization compares to private storage clouds.

Topics include:

• Defining storage virtualization: What it is and where to find it
  • Abstraction of storage resources
  • Tiered storage
  • Flexibility
• Popular approaches to storage virtualization
  • SAN controllers
  • File virtualization
  • Volume managers
• The pool, the hypervisor and the cloud
  • The impact of server virtualization
  • Is this a private cloud?

Cutting Off Data Growth at the Disk

In this special dinner presentation, Stephen Foskett will discuss how to apply key data management technologies to arrest the growth of data. You’ll learn how capacity optimization technologies such as data deduplication and compression can reduce the trajectory of data growth as well as how tiering can reduce the cost of storage. Finally, Stephen will explore why the time may have finally come for active archiving and will leave you with practical ways to help your corporation better manage its data.

Note that space is limited for the dinner, which is sponsored by my friends at Dell.

Registration

To register for Storage Decisions Chicago, just go to the TechTarget registration page. Dinner guests will apparently be selected from that same pool of attendees.

Disclosure: TechTarget pays my expenses to attend and present at Storage Decisions, and has for many years. I also get a speaker fee for the dinner session.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Storage Decisions Chicago: All About Capacity Optimization
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

SVC 6.1 + Disk Hardware = V7000

Let’s get one thing out of the way immediately: As I’ve said many times (including on stage at Storage Decisions last month), SVC is about the only IBM storage product I genuinely like. Its well-engineered, useful, and performs well. It’s just too bad its native habitat is a jungle of weird and expensive IBM gear.

SVC is really an enterprise storage array without any disks, just as HDS’ USP VSP is a storage virtualization engine with disks. It does all sorts of great things, from thin provisioning to replication to automatic tiered storage to painless migration (once you’re migrated to it, at least). Fibre Channel comes in, magic happens, and Fibre Channel comes out. And it runs on commodity servers, which surely gives IBM a healthy profit margin but doesn’t seem to translate into lower cost for customers.

Green = SVC 5; Pink = SVC 6.1. No Storwize.

The new Storwize V7000 is essentially the SVC software running on server hardware that includes both dual controllers and a bunch of internal hard disk drives. This can connect to up to nine “dumb” expansion storage enclosures. Hardware-wise, it’s very like the typical midrange modular storage systems sold by EMC (CLARiiON), HP (EVA), HDS (AMS), and NetApp.

Software-wise the V7000 is all SVC. Much of the software is directly derived from SVC 5.1 (green stuff in IBM’s diagram), while some new tech is mixed in, too. But pretty much everything (green, blue, pink) is shared with SVC 6.1 other than the hardware. It’s just incredible what advanced software running on commodity hardware can do, and IBM is right up there with folks like HP and EMC who are adopting this engineering model.

Where’s the Storwize?

Then there’s that name. This isn’t just the V7000, it’s the Storwize V7000. When I heard the name, I was expecting that it would include some data reduction/optimization/compression/whatever technology from Storwize, the company IBM acquired in July. This would match EMC’s acquisition of Data Domain, Dell’s buy of Ocarina, and HP’s rollout of their cool StorOnce software.

But there’s no Storwize in the V7000 apart from the name. This is a straight-ahead midrange storage system with no special bit-crunching powers apart from the thin provisioning already offered by SVC. I asked the IBM folks about this, and they confirmed that they needed a name and thought Storwize was fitting.

Right from the horse's mouth. No Storwize software here (yet).

Stephen’s Stance

With everyone and their brother (well, EMC, HP, Dell, and NetApp) rolling out primary storage deduplication, I expect this situation will change. Perhaps “Storwize” will become the IBM equivalent of “StorageWorks” – sprayed across every product. Or maybe it will become IBM’s midrange brand. But sooner or later I expect IBM will include their compression technology, too (I dare not call it “data reduction” or face The Wrath of Tony).

So the Storwize V7000 is a really nice midrange product built on proven software and ought to compete nicely with EMC, HP, and HDS. It’s maybe even a little better than the competing modular storage products. My interest would be piqued, however, by news of a larger scale-out cluster of V7000 systems. The SVC can already scale out like this, with 4-pair I/O groups.

But even without compression and scale-out, I could see myself recommending the V7000 to midrange storage buyers. Good work, IBM! Now, let’s talk about the rest of your storage products…

V7000 Diagram courtesy of IBM

© sfoskett for Stephen Foskett, Pack Rat, 2010. | IBM’s Storwize V7000: 100% SVC; 0% Storwize
This post was categorized as Enterprise storage, Everything, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

See, Snow Leopard’s version of HFS+ allows per-file compression using three very creative filesystem hacks. I’ll let John Siracusa from Ars Technica take the story from here, and I urge you to read John’s complete (and very, very long) Snow Leopard review!

In Snow Leopard, other kinds of files climb on board the compression bandwagon. To give just one example, ninety-seven percent of the executable files in Snow Leopard are compressed. How compressed? Let’s look:

% cd Applications/Mail.app/Contents/MacOS

% ls -l Mail

-rwxr-xr-x@ 1 root wheel 0 Jun 18 19:35 Mail

Boy, that’s, uh, pretty small, huh? Is this really an executable or what? Let’s check our assumptions.

% file Applications/Mail.app/Contents/MacOS/Mail

Applications/Mail.app/Contents/MacOS/Mail: empty

Yikes! What’s going on here? Well, what I didn’t tell you is that the commands shown above were run from a Leopard system looking at a Snow Leopard disk. In fact, all compressed Snow Leopard files appear to contain zero bytes when viewed from a pre-Snow Leopard version of Mac OS X. (They look and act perfectly normal when booted into Snow Leopard, of course.)

So, where’s the data? The little “@” at the end of the permissions string in the ls output above (a feature introduced in Leopard) provides a clue. Though the Mail executable has a zero file size, it does have some extended attributes:

% xattr -l Applications/Mail.app/Contents/MacOS/Mail

com.apple.ResourceFork:

0000 00 00 01 00 00 2C F5 F2 00 2C F4 F2 00 00 00 32 …..,…,…..2

0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….

(184,159 lines snipped)

2CF610 63 6D 70 66 00 00 00 0A 00 01 FF FF 00 00 00 00 cmpf…………

2CF620 00 00 00 00 ….

com.apple.decmpfs:

0000 66 70 6D 63 04 00 00 00 A0 82 72 00 00 00 00 00 fpmc……r…..

Ah, there’s all the data. But wait, it’s in the resource fork? Weren’t those deprecated about eight years ago? Indeed they were. What you’re witnessing here is yet another addition to Apple’s favorite file system hobbyhorse, HFS+.

At the dawn of Mac OS X, Apple added journaling, symbolic links, and hard links. In Tiger, extended attributes and access control lists were incorporated. In Leopard, HFS+ gained support for hard links to directories. In Snow Leopard, HFS+ learns another new trick: per-file compression.

The presence of the com.apple.decmpfs attribute is the first hint that this file is compressed. This attribute is actually hidden from the xattr command when booted into Snow Leopard. But from a Leopard system, which has no knowledge of its special significance, it shows up as plain as day.

Even more information is revealed with the help of Mac OS X Internals guru Amit Singh’s hfsdebug program, which has quietly been updated for Snow Leopard.

% hfsdebug /Applications/Mail.app/Contents/MacOS/Mail

…

compression magic = cmpf

compression type = 4 (resource fork has compressed data)

uncompressed size = 7500336 bytes

And sure enough, as we saw, the resource fork does indeed contain the compressed data. Still, why the resource fork? It’s all part of Apple’s usual, clever backward-compatibility gymnastics. A recent example is the way that hard links to directories show up—and function—as aliases when viewed from a pre-Leopard version of Mac OS X.

In the case of a HFS+ compression, Apple was (understandably) unable to make pre-Snow Leopard systems read and interpret the compressed data, which is stored in ways that did not exist at the time those earlier operating systems were written. But rather than letting applications (and users) running on pre-10.6 systems choke on—or worse, corrupt through modification—the unexpectedly compressed file contents, Apple has chosen to hide the compressed data instead.

And where can the complete contents of a potentially large file be hidden in such a way that pre-Snow Leopard systems can still copy that file without the loss of data? Why, in the resource fork, of course. The Finder has always correctly preserved Mac-specific metadata and both the resource and data forks when moving or duplicating files. In Leopard, even the lowly cp and rsync commands will do the same. So while it may be a little bit spooky to see all those “empty” 0 KB files when looking at a Snow Leopard disk from a pre-Snow Leopard OS, the chance of data loss is small, even if you move or copy one of the files.

The resource fork isn’t the only place where Apple has decided to smuggle compressed data. For smaller files, hfsdebug shows the following:

% hfsdebug /etc/asl.conf

…

compression magic = cmpf

compression type = 3 (xattr has compressed data)

uncompressed size = 860 bytes

Here, the data is small enough to be stored entirely within an extended attribute, albeit in compressed form. And then, the final frontier:

% hfsdebug /Volumes/Snow Time/Applications/Mail.app/Contents/PkgInfo

…

compression magic = cmpf

compression type = 3 (xattr has inline data)

uncompressed size = 8 bytes

That’s right, an entire file’s contents stored uncompressed in an extended attribute. In the case of a standard PkgInfo file like this one, those contents are the four-byte classic Mac OS type and creator codes.

% xattr -l Applications/Mail.app/Contents/PkgInfo

com.apple.decmpfs:

0000 66 70 6D 63 03 00 00 00 08 00 00 00 00 00 00 00 fpmc…………

0010 FF 41 50 50 4C 65 6D 61 6C .APPLemal

There’s still the same “fpmc…” preamble seen in all the earlier examples of the com.apple.decmpfs attribute, but at the end of the value, the expected data appears as plain as day: type code “APPL” (application) and creator code “emal” (for the Mail application—cute, as per classic Mac OS tradition).

You may be wondering, if this is all about data compression, how does storing eight uncompressed bytes plus a 17-byte preamble in an extended attribute save any disk space? The answer to that lies in how HFS+ allocates disk space. When storing information in a data or resource fork, HFS+ allocates space in multiples of the file system’s allocation block size (4 KB, by default). So those eight bytes will take up a minimum of 4,096 bytes if stored in the traditional way. When allocating disk space for extended attributes, however, the allocation block size is not a factor; the data is packed in much more tightly. In the end, the actual space saved by storing those 25 bytes of data in an extended attribute is over 4,000 bytes.

But compression isn’t just about saving disk space. It’s also a classic example of trading CPU cycles for decreased I/O latency and bandwidth. Over the past few decades, CPU performance has gotten better (and computing resources more plentiful—more on that later) at a much faster rate than disk performance has increased. Modern hard disk seek times and rotational delays are still measured in milliseconds. In one millisecond, a 2 GHz CPU goes through two million cycles. And then, of course, there’s still the actual data transfer time to consider.

Granted, several levels of caching throughout the OS and hardware work mightily to hide these delays. But those bits have to come off the disk at some point to fill those caches. Compression means that fewer bits have to be transferred. Given the almost comical glut of CPU resources on a modern multi-core Mac under normal use, the total time needed to transfer a compressed payload from the disk and use the CPU to decompress its contents into memory will still usually be far less than the time it’d take to transfer the data in uncompressed form.

That explains the potential performance benefits of transferring less data, but the use of extended attributes to store file contents can actually make things faster, as well. It all has to do with data locality.

If there’s one thing that slows down a hard disk more than transferring a large amount of data, it’s moving its heads from one part of the disk to another. Every move means time for the head to start moving, then stop, then ensure that it’s correctly positioned over the desired location, then wait for the spinning disk to put the desired bits beneath it. These are all real, physical, moving parts, and it’s amazing that they do their dance as quickly and efficiently as they do, but physics has its limits. These motions are the real performance killers for rotational storage like hard disks.

The HFS+ volume format stores all its information about files—metadata—in two primary locations on disk: the Catalog File, which stores file dates, permissions, ownership, and a host of other things, and the Attributes File, which stores “named forks.”

Extended attributes in HFS+ are implemented as named forks in the Attributes File. But unlike resource forks, which can be very large (up to the maximum file size supported by the file system), extended attributes in HFS+ are stored “inline” in the Attributes File. In practice, this means a limit of about 128 bytes per attribute. But it also means that the disk head doesn’t need to take a trip to another part of the disk to get the actual data.

As you can imagine, the disk blocks that make up the Catalog and Attributes files are frequently accessed, and therefore more likely than most to be in a cache somewhere. All of this conspires to make the complete storage of a file, including both its metadata in its data, within the B-tree-structured Catalog and Attributes files an overall performance win. Even an eight-byte payload that balloons to 25 bytes is not a concern, as long as it’s still less than the allocation block size for normal data storage, and as long as it all fits within a B-tree node in the Attributes File that the OS has to read in its entirety anyway.

There are other significant contributions to Snow Leopard’s reduced disk footprint (e.g., the removal of unnecessary localizations and “designable.nib” files) but HFS+ compression is by far the most technically interesting.

via Mac OS X 10.6 Snow Leopard: the Ars Technica review – Ars Technica.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Bizarre HFS+ Tricks in Mac OS X 10.6 Snow Leopard
This post was categorized as Apple, Computer History. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Does data encryption throw storage efficiency out the window? Not always!

One of the great ironies of storage technology is the inverse relationship between efficiency and security: Adding performance or reducing storage requirements almost always results in reducing the confidentiality, integrity, or availability of a system.

Many of the advances in capacity utilization put into production over the last few years rely on deduplication of data. This key technology has moved from basic compression tools to take on challenges in the fields of replication and archiving, and is even moving into primary storage. At the same time, interconnectedness and the digital revolution has made security a greater challenge, with focus and attention turning to encryption and authentication to prevent identity theft or worse crimes. The only problem is, most encryption schemes are incompatible with compression or deduplication of data!

Incompatibility of Encryption and Compression

Consider a basic lossless compression algorithm: We take an input file consisting of binary data and replace all repeating patterns with a unique code. If a file contained the sequence, “101110″ eight hundred times in a row, we could replace the whole 4800-bit sequence with a much smaller sequence that says “repeat this eight hundred times”. In fact, this is exactly what I did (using English) in the previous sentence! This basic concept, called run-length encoding, illustrates how most modern compression technology functions.

Replace the sequence of identical bits with a larger block of data or an entire file and you have deduplication and single-instance storage! In fact, as the compression technology gains access to the underlying data, it can become more and more efficient. The software from Ocarina, for example, actually decompresses jpg and pdf files before recompressing them, resulting in astonishing capacity gains!

Now let’s look at compression’s secretive cousin, encryption. It’s only a small intellectual leap to use similar ideas to hide the contents of a file, rather than just squashing it. But encryption algorithms are constantly under attack, so some very smart minds have come up with some incredibly clever methods to hide data. One of the most important advances was public-key cryptography, where two different keys are used: A public key used for writing, and a private key to read data. This same technique can be used to authenticate identity, since only the designated reader would (in theory) have the key required.

Cryptography has become exceedingly complicated lately in response to repeated attacks. Most compression and encryption algorithms are deterministic, meaning that identical input always yields the same output. This is unacceptable for strong encryption, since a known plaintext attack can be used with the public key to reveal the contents. Much work has focused on eliminating residues of the original data from the encrypted version, as illustrated brilliantly on Wikipedia with the classic Linux “tux” image. The goal is to make the encrypted data indistinguishable from random “noise”.

What happens when we mix these powerful technologies? Deduplication and encryption defeat each other! Deduplication must have access to repeating, deterministic data, and encryption must not allow this to happen. The most common solution (apart from skipping the encryption) is to place the deduplication technology first, allowing it access to the raw data before sending it on to be encrypted. But this leaves the data unprotected longer, and limits the possible locations where encryption technology can be applied. For example, an archive platform would have to encrypt data internally, since many now include deduplication as an integral component.

Why do we prefer compression to encryption? Simply because that’s where the money is! If we can cut down on storage space or WAN bandwidth, we see cost avoidance or even real cost savings! But if we “waste” space by encrypting data, we only save money in the case of a security breach.

A Glimmer of Hope

I had long thought this was an intractable problem, but a glimmer of hope recently presented itself. My hosting provider allows users to back up their files to a special repository using the rsync protocol. This is pretty handy, as you can imagine, but I was concerned about the security of this service. What happens if someone gains access to all of my data by hacking their servers?

At first, I only stored non-sensitive data on the backup site, but this limited its appeal. So I went looking for something that would allow me to encrypt my data before uploading it, and I discovered two interesting concepts: rsyncrypto and gzip-rsyncable.

rsync is a solid protocol, reducing network demands by only sending the changed blocks of a file. But, as noted, compression and encryption tools change the whole file even if only a tiny bit has been altered. A few years back, the folks behind rsync (who also happen to be the minds behind the Samba CIFS server) developed a patch for gzip which causes it to compress files in chunks rather than in their entirety. This patch, called gzip-rsyncable, hasn’t been added to the main source even after a dozen years, but yields amazing results in accelerating rsync performance.

The same technique was then applied to RSA and AES cryptography to create rsyncrypto. This open source encryption tool makes a simple tweak to the standard CBC encryption schema (reusing the initialization vector) to allow encrypted files to be sent more efficiently over rsync. In fact, it relies on gzip-rsyncable to work its magic. Of course, the resulting file is somewhat less secure, but it is probably more than enough to keep a casual snooper at bay.

Both of these tools are similar to modern deduplication techniques in that they chop files up into smaller, variable-sized blocks before working their magic. And the result is awesome: I modified a single word in a large word document that I had previously encrypted and stored at the backup site and was able to transfer just a single block of the new file in an instant rather than a few minutes. My only real issue is the lack of integration of all of these tools: I had to write a bash script to encrypt  my files to a temporary directory before rsyncing them. I wish they could be integrated with the main gzip and rsync sources!

If you are interested in trying out these tools for yourself, and if you use a Mac, you are in luck: Macports offers both tools as simple downloads! Just install macports, type “sudo port install gzip +rsyncable” to install gzip with the –rsyncable flag, then type “sudo port install rsyncrypto” and you’re done! I’ll post more details here if there is interest.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Compression, Encryption, Deduplication, and Replication: Strange Bedfellows
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Great! But I thought the new era for IT was a cloud with choice, mobility, and application support, as trumpeted by EMC’s VMware subsidiary! Wasn’t Cloud vServices from VDC-OS supposed to be the prototype cloud strategy for the datacenter?

What we have here is a simple clash of marketing amusingly taking place at (nearly) the same company. VMware figured out how to extend their server virtualization products outside the confines of the data center, and laid that technology out as a strategy with the trendy “cloud” name. Meanwhile, mother EMC is working on next-generation content storage software and decides to roll that out as a strategy and also jumps on the “cloud” meme. What’s an IT manager to do?

Defining Atmos

As predicted, EMC’s Atmos (code-name Maui) is a distributed software layer to handle the storage and management of data objects across geographically-dispersed storage devices. EMC’s Chuck Hollis demonstrates Atmos with a simple, practical example, perhaps making it sound too much like Akamai but generally getting the point across. You have a data object, write it to Atmos through REST/SOAP or CIFS/NFS, assign some metadata, and the software takes care of data placement for you. It’ll add local copies, replicate for availability and performance, compress or deduplicate, manage versions, and all sorts of goodies (if you ask it to).

But EMC already has a capable object storage platform, the Centera. We’ve just got used to the content-addressable storage (CAS) label for object storage (even though this name misses the point of object storage, in my opinion) and now EMC wants us to learn a new label for a somewhat-similar device? Steve Todd, EMC’s object guy extraordinaire, lays it out:

SAN Value = Centralized, secure multi-tenancy for blocks.

NAS Value = Centralized, secure multi-tenancy for files.

CAS Value = Centralized, secure multi-tenancy for objects (content + metadata).

COS Value = Globalized, secure multi-tenancy for content with rich policies.

Ok, so the defining capabilities of Atmos are its global scale and rich policies. And the fact that “objects” has become “content”, presumably since Atmos can handle traditional NAS (CIFS/NFS) chores as well.

Prayers Answered?

It sounds like EMC is answering my prayers for a storage revolution, delivering a highly-capable object storage platform that transcends the old limits of blocks, directories, and files. Steve Todd points out that Atmos handles five policy categories out of the box:

• Replication
• Compression
• Spin-down
• Object de-dup
• Versioning

So we write some data to Atmos, using either traditional NAS or webby dubby protocols like SOAP, and can then apply policies in any of these five categories to that data. One can also extend the Atmos to accept other policies, but the absence (out of the box) of concepts like encryption, secure deletion, retention, and access control are surprising.

I am quite puzzled about how practical these policy capabilities will be in the real world. How exactly would an application say “I want you to compress that file I wrote over NFS just now?” Hitachi’s HCAP platform, for example, also has policy capabilities and a NAS front end, and although archiving applications can communicate their policy needs, I don’t see lots of current general-purpose applications using it.

Strategic Storage?

This brings me to my puzzlement: The default Atmos policies are all general-purpose, production computing ideas, not the special-purpose, archiving and retention needs served by Centera, HCAP, and the rest. So the Atmos is clearly intended to be a production data storage system, not an archiving system to compete with Centera.

Since mainstream business applications currently don’t have any capability to specify policies like these when writing files, and since NAS protocols lack any means to communicate them even if the apps want to, we can conclude that EMC expects that Atmos users will write special applications to take advantage of it.

EMC certainly doesn’t expect that the NAS-capable Atmos will simply replace today’s distributed NAS solutions. NAS is a sideshow for Atmos. The real action will be in the REST/SOAP webby dubby applications that will be written with the platform in mind and will take full advantage of these capabilities.

If this is true, and I and others suspect that it is, then Atmos really isn’t a game-changing platform unless you change your game. If you write new applications to store data with SOAP, Atmos is a nice in-house alternative to Amazon S3 or Nirvanix, and offers a very compelling set of data management capabilities. And if you want to set up shop to compete with those service providers, Atmos is a dream come true with built-in multi-tenancy.

Datacenter Strategy

So EMC alone has two seemingly competitive datacenter strategies. And then there’s Microsoft, which announced its Azure cloud platform recently, and Amazon and the other cloud providers.

So let’s say you’re a CIO for a large corporation. Which of the following strategies is more compelling:

1. Use VMware VDC-OS to add capabilities and Cloud vServices extend your current virtual infrastructure geographically
2. Recompile and tweak your Windows applications to leverage Microsoft Azure
3. Develop new applications to take advantage of the impressive storage capabilities of an in-house EMC Atmos system
4. Point your new applications at a third-party cloud provider like Amazon or Nirvanix

IT people are practical. Although we love new technology, we tend to be cautious. We also hate massive software development efforts, and only sanction them when they’re absolutely necessary. Given these personality traits, I’d say VDC-OS and perhaps Cloud vServices still stands out as the most likely and practical scenario for the majority of applications and businesses.

This is not to say that EMC Atmos will be a flop. I’m impressed by the technology, and expect that Atmos will find buyers, just as Centera did. And Atmos might even replace Centera once EMC adds retention policies to it and scales it down as well as up and out. But Atmos will not redefine the datacenter. We’re stuck with blocks and files, and VMware’s practical strategy is a winner in that world.

Update: Marc Farley compares Atmos to WAFS, with ominous implications, and echos my recent question on what is and is not innovative.

Update 2: Chuck Hollis, Storagezilla, and Len Devanna have all come right out and said that this is only intended for certain customers with massive distributed storage needs, and is not intended as a new datacenter strategy. Even the “cloudfella” says “ciao”:

Update 3: More great information, including a reply regarding VDC-OS and Atmos from the one and only Chad Sakac, more great detail about the inner workings of Atmos from Steve Todd, and even more info from Dave Graham. Finally, although I think that Cloudfellas video is cute, I wouldn’t categorize it as viral. But those Mozy ads are awesome!

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | EMC Atmos Versus VMware VDC-OS: Will The Real Cloud Strategy Please Stand Up?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although deduplication of storage is nothing new, with Data Domain and other making hay with the technique for years, it has never been ready for prime time – reduction of active primary storage applications like email and databases. Instead, deduplication has been relegated to second- or third-tier status, deduplicating archives and backup data. But change is in the air, and deduplication vendors are starting to bustle towards the bright lights of primary storage.

Stone Knives and Bear Skins

We have all been here before, of course. Back at the dawn of the personal computer era, data compression was a hot topic of conversation. I recall being so impressed by an article in Byte (1986:5, p99) outlining Huffman coding that I tried cooking up an implementation in Atari BASIC. Lossless compression has a magical pull to the geek in many of us – redundant data just wants to be eliminated!

Stacker dominated the disk compression world - until Microsoft introduced DOS 6.0

Companies soon applied compression to primary storage, especially the limited storage in personal computers. Stacker was a hit after 1990, until Microsoft built a workalike, called DoubleSpace, into DOS 6.0 in 1993, leading to a historical lawsuit. I personally used the ADDSTOR disk compression built into DR-DOS 6.0 to stretch two more years out of the 20 MB MFM hard drive in my AT&T PC6300 at WPI.

But something funny happened in the late 1990s: Compression began to lose its luster. Compressing data always takes quite a bit of CPU power, but this was offset somewhat by the truncated data transfers and more-efficient file system layout afforded in early PCs. But as disks got larger and faster, using precious CPU time to save space seemed less and less compelling. Today, although nearly every operating system includes built-in compression of files, folders, or perhaps disks, these features are rarely used. And compression was never popular in the performance-sensitive enterprise space.

Deduplication Has a Nice Ring

Although traditional fine-grained compression has not been very successful in the enterprise, its lanky cousin, single-instance storage, has long found niche jobs. Applications from databases to email systems to file servers have long had the ability to recognize to requests to store the exact same file or record, and to store just a single instance in this case. Even file systems have the ability to do single instance storage through the use of links, though this is initiated by the user rather than in an automated fashion.

In the late 1990s, FilePool began developing a content-addressable storage device, which was acquired by EMC in 2001. This device, later known as the Centera, was one of a number of storage platforms targeted at the archiving market introduced this decade. At the same time, virtual tape libraries made the jump from the mainframe to open systems. Both devices, being outside the critical path of performance but offering massive capacity, were well-suited to implement advanced capacity optimization technologies that combined the concepts of compression with single-instance storage. Thus was created the modern world of data deduplication.

What we think of as deduplication is neither fish nor fowl: It assesses larger “chunks” of data than compression technologies, delivering greater capacity savings and potentially reducing performance impact, but is more flexible than single-instancing, recognizing the similarities within files or objects.

But it is still maddeningly difficult to scale deduplication while maintaining performance. Rather than fight to maintain reasonable write throughput, most deduplication products have switched to post-processing, deferring their work to quieter times.

It’s Not Just for Breakfast

Regardless of their methods or underlying technology, no deduplication vendor has stood up to support challenging low-latency or high-throughput production applications, however. NetApp was the first to raise the issue of support for production applications, but although they tout the technology for VMware, they haven’t exactly been shouting from the rooftops to get their A-SIS deduplication technology deployed in other high-I/O applications. And I haven’t seen Hifn’s card yet.

Yesterday, I mentioned that greenBytes was adding deduplication to their ZFS-based storage array for primary data. And now Riverbed has fired another shot over the bow, repurposing their (deduplicating) WAN accelerator product for primary (file) storage. They might be able to pull it off, too, since they have a long list of customers who are already enjoying the technology in production. It’s not a stretch to suggest that Riverbed’s appliances can scale to handle production data loads. Although it’s file-only, I can imagine quite a few scenarios where this tech could really yield benefits. Could we come full-circle, with deduplication finally reaching the enterprise storage world?

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Deduplication Coming to Primary Storage
This post was categorized as Computer History, Enterprise storage, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.