Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
This post was categorized as Enterprise storage, Everything, Gestalt IT, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Data Reduction can be hazardous to your health!

I’m not a big fan of data reduction technology, yet I found myself talking compression and de-duplication all week. Between Storage Decisions and my recent posts over at SearchStorage and The Storage Community, I’ve had quite a bit to say on the subject. Funny enough, I’m not really a fan of data reduction technology for primary storage. Too often, data reduction is more expensive and difficult than just storing raw data.

You should also read Deduplication Coming to Primary Storage and Compression, Encryption, Deduplication, and Replication: Strange Bedfellows

Storage Decisions

My Storage Decisions presentation on data reduction was hilarious, if I do say so myself, even though turnout was poor at 8:30 AM on Tuesday morning. Maybe it was this “intimate” group, but I found myself really getting into the discussion. And the nods and hollers from the audience helped, too!

My basic thesis at Storage Decisions was the same as always: Don’t throw good money at technology that will have little ROI. Considering that disk capacity is incredibly cheap, and dropping all the time, data reduction doesn’t look like a great fit except in certain situations. Why spend money to reduce utilization? Why put in the effort when most primary storage data reduction technologies don’t do anything to address the “multiplier effect” of archiving, DR, and backup storage?

This is not to say that all data reduction technology is worthless. In fact, the free compression and de-duplication built into many SSDs and even some enterprise storage devices make perfect sense. I just don’t understand spending a bunch of money to address storage capacity when most applications are starved for storage performance.

You might like reading my two other posts on the subject from last week:

• Interest in data reduction methods needs to keep pace with data growth (SearchStorage.com)
• Has the Time Finally Come for Data Reduction? (The Storage Community, sponsored by IBM)

You’re Losing Me

On the other hand, I do see quite a bit of value in something many people would overlook out of hand: Lossy compression of office files. Every systems administrator knows that end-users do “stupid stuff” like embedding massive photos and videos in PowerPoint presentations and Word documents. But not everyone knows that there are technological means to address this “PEBKAC” issue.

Some office applications already automatically reduce the size of embedded content, and operating systems can do the same. One of my more popular blog posts, in fact, is a technique to create a filter to reduce the size of PDF files in Mac OS X Preview. And the Microsoft “X” Office file formats include lossless compression as well.

An application that recently caught my eye is the FILEminimizer Suite by Balesio. This inexpensive application reduces the size of Office and media files while leaving them in their native format. It re-compresses image files, reducing them to optimum size for use in presentations, documents, or printouts. A companion product, FILEminimizer Server, can be used on enterprise file servers to perform the same magic across the whole range of users.

Stephen’s Stance

Native Format Optimization (NFO) makes a lot of sense, since it addresses a common user error in a practical way, and allows capacity savings to “trickle-down” to backups, e-mail systems, and archives. But wholesale compression and the duplication of primary storage may not be worth much, especially since the cost of disk keeps dropping dramatically.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Data Reduction: the Condensed Version
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Join me in New York for Storage Decisions, September 19 & 20

Later this month, I will be heading to New York for TechTarget’s Storage Decisions conference. This show does a good job on the editorial side, suggesting timely topics and bringing in independent voices like Howard Marks. I will have two presentations on data reduction and storage virtualization in the main conference track. Registration is free for qualified end-users, and I urge you to attend on September 19 and 20, 2011.

Reclaim Capacity with Data Reduction for Primary Storage

Depending on which industry study you read, most companies are wasting anywhere from 30% to 50% of their installed disk capacity, which translates into thousands of dollars spent with no effective return on investment. Storage vendors are beginning to provide tools that can help storage managers make the most of the disk they have installed. For example, data reduction for primary storage borrows data deduplication technology developed for backup and classic compression algorithms to help squeeze the air out of nearline and primary data and reduce its footprint. This session’s topics will include an overview of data reduction technologies and where they will have the greatest impact, what key storage vendors are offering in data reduction and an update on the major players, and the consequences of using primary data dedupe along with dedupe for backups. We’ll also look at the potential for vendor lock-in and consider why we’re reducing data in the first place.

Topics include:

• Introducing data reduction technologies
  • Compression: How it works and where it’s found
  • Deduplication: From single-instancing to variable block
  • Application-specific: Cracking open files
• Overview of data reduction products
• Where to use them
  • The capacity conundrum: Store less and reduce utilization
  • Ideal applications: Justifying the cost of data reduction
  • Side effects: Considering the impact on backup, replication, I/O workload and vendor lock-in

Storage Virtualization: Who’s Doing It and Why

Storage virtualization has been around for decades and, although research indicates that 70% of companies have already virtualized at least some of their installed block or file storage, most remain unaware of this technology. Grandiose schemes for comprehensive virtual SANs have given way to more practical host- and array-based virtualization technologies, and server virtualization has created a new opportunity to create a pool of storage. This session will look at the current state of storage virtualization, how to quantify its benefits and describe which approaches are best for particular environments, and also cover how storage virtualization compares to private storage clouds.

Topics include:

• Defining storage virtualization: What it is and where to find it
  • Abstraction of storage resources
  • Tiered storage
  • Flexibility
• Popular approaches to storage virtualization
  • SAN controllers
  • File virtualization
  • Volume managers
• The pool, the hypervisor and the cloud
  • The impact of server virtualization
  • Is this a private cloud?

Registration

You can see the future from here!

To register for Storage Decisions New York, just go to the TechTarget registration page.

Disclosure: TechTarget pays my expenses to attend and present at Storage Decisions, and has for many years. But they don’t pay me to present and I own the copyright on my session content. Happily, I license it all CC-by-NC-SA so I can give it out freely!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Storage Decisions New York: Capacity Optimization
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

When is a gigabyte not a gigabyte? When you're not buying gigabytes!

I stepped into a hornet nest this week when I posted a write-up about a new flash storage array from Pure Storage. The controversy had nothing to do with the underlying technology, which seems quite sound. Rather, it was all about pricing, with Pure’s competitors calling foul on their price comparisons.

You’re Not Buying Gigabytes

In a world of 3 TB drives, storage capacity is almost irrelevant. Capacity is what people think they are getting when they buy enterprise storage devices, but capacity is only one aspect of the purchase, and it’s not a very significant one in most cases.

So what are enterprise storage buyers buying?

• Performance, especially I/O operations (IOPS), is much more critical than capacity in most cases, and it takes lots of spindles or expensive flash chips to deliver it.
• Data protection features like snapshots are increasingly important, and often cost extra.
• Compatibility is paramount, as is long-term supportability from all vendors involved.
• Integration and management features are often the deciding factor in purchases, especially when it comes to server virtualization applications.
• High availability and product support are “must-haves” that can multiply the cost of a solution.
• Power, cooling, and floor space can be very important for some applications and entirely inconsequential in others.
• Capacity is sometimes important, but many applications require just a few TB or less and thin provisioning, data deduplication, and compression are really blurring the lines here.

So although a typical customer will say “I need 200 GB for this application” they likely need nothing of the sort. They really need 100 IOPS, snapshots, a line on the HCL, VAAI and vCenter plugins, and redundant everything. Even the capacity number is questionable: Most applications grow over time, and few need much capacity really.

Since you can’t buy a 1 GB storage array and can’t fill a 10 TB device to 100%, pricing per GB is complete nonsense. Plain old storage space just sort of tags along for the ride once you build a system capable of meeting all these other needs.

Data Reduction or Pricing Obfuscation?

Utilization of storage capacity has always been terrible, but improving capacity efficiency is worthless. The best you can do is over-tax your array or put all your data “eggs” in too few drive “baskets”. Achieving impressive capacity utilization just concentrates I/O, and this is the last thing you want to do with spinning hard disk drives.

This is why I suggest redirecting the conversation away from capacity requirements. The amount of GB to be used and the efficiency of that storage doesn’t matter all that much except for certain massive and rare applications. Once the array is big enough to handle the data, everything else is a wash.

This is also why I’m skeptical of data reduction technologies. Most applications would be better off optimizing for performance not reducing capacity used. And data reduction techniques like compression and deduplication quickly lead down the “your mileage may vary” rat hole.

Comparing Apples to Apples

Also read Grapples and Tangelos: Why it’s Impossible to Compare Fairly

There is only one way to do a real fair comparison between different storage devices: Specify all the requirements and let each vendor put forward whatever they have that meets all of them. Who really cares if vendor A’s disk-based solution is 10% utilized while vendor B’s flash array needs 1/5 the capacity? As long as you have a place to put it (and enough power to feed it) it’ll still work fine.

One serious challenge in enterprise storage is the rise of flash memory as a storage medium. Flash chips are expensive on a data capacity basis but amazingly cheap in terms of performance and environmental efficiency. Put another way, an SSD can’t storage as much data as a hard disk, but it delivers massive I/O capability in a tiny, rugged, low-power footprint.

Since most enterprise applications need only a few hundred GB of capacity, a few SSDs can be a compelling alternative to a “refrigerator” full of disks. It can be hard to convince the boss, but you really can fit a whole datacenter’s worth of storage I/O into a few rack units!

Pure and Nimbus

This is the issue facing flashy solid state devices from many companies, and the root of my headaches this week. Pure Storage hasn’t finalized pricing yet, but are claiming that their new device costs $5 per usable gigabyte. This is incredibly cheap for an array that can blow the doors off most enterprise gear!

Nimbus Data, on the other hand, sells their all-flash enterprise storage array for about $10 per GB. But this is not the end of the story, and Pure might even be more expensive than Nimbus! Or maybe not. It all depends on what you’re comparing.

Pure claims that their cost is half the price of most comparable flash storage array competitors, but this is where the questions start to appear. Is that $5 gigabyte usable or raw? Does it include high availability? And can I really store any old gigabyte of data there or is that a compressed/deduplicated gigabyte?

It turns out that the real cost of Pure Storage capacity is $20 per GB including RAID and an extra mirrored array for high availability. But since every byte written to the array is thin provisioned, deduplicated, and compressed, many customers will pay much less for actual data stored. And since it’s an all-SSD array, it’ll perform way better than a disk-based system, too.

Muddying the Waters

So why not just call it $5 per GB and be done with it? It’s confusing, that’s why, and your mileage will vary widely.  Pure’s own slides show some applications getting 4:1 data reduction and others all the way up to 17:1. So these applications would be paying as low as $1.18 per GB or as high as $5.

But you can’t buy just 1 GB of storage from Pure. Their smallest array (which includes one controller and one shelf of SSDs) provides 5.5 TB of raw capacity, presumably using 24 256 GB SSDs. A high-availability configuration would include two controllers and two shelves of SSDs for 11 TB of raw storage. That’s going to cost almost a quarter of a million dollars according to my calculator. That’s one expensive gigabyte!

Of course no one would buy this array to store just a thousand megabytes. They would buy it to support a bunch of applications that need capacity and performance and efficiency and integration and everything else. And they can buy a mirrored pair of arrays from Pure Storage or Nimbus or Violin Memory or Texas Memory Systems or others at a variety of price points.

The only way to really compare these products is to spec them out on equal footing and see what the price tag looks like. These comparisons would include data reduction, but they would also have to bring in high availability and every other requirement of the applications they will support.

Stephen’s Stance

It’s way too difficult for me to do the pricing math for these systems, so I’m throwing in the towel. I’m thrilled to see all-flash arrays made available to IT buyers. This wouldn’t be possible without clever use of thin provisioning and data reduction, as well as smart software to overcome the limits of SSD.

I’m going to guess that Pure and Nimbus will cost about the same for similar configurations, though I’ll bet each believes they’re cheaper. Rather than get in the middle, I invite each company to post a comment below stating their case. I’ll even embed their responses into a future blog post on the subject if they get too long. Just don’t ask me to be the referee.

Update: Pure Storage responds with an outline of their pricing:

• How Pure Storage Delivers All-Flash Storage at Below the Price of Spinning Disk

Image credit: Rotten Apple by Wappas

© sfoskett for Stephen Foskett, Pack Rat, 2011. | When Pricing Gets Squishy Competition Heats Up
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

A Better Way to Dedupe Data: Dedupe Everywhere

You might also want to read When Marketing Becomes Pointless and The Epidemiology of Viral Videos

Stephen’s Stance

If you’ve been following the IT industry and my blog for a while, you probably recall the Mini Cooper stunts EMC pulled earlier in 2011: They packed a bunch of contortionists into a Mini at their New York event, while simultaneously parking a bunch of logo covered cars in front of NetApp headquarters. This was intended to demonstrate something about storage, but the exact point escaped me. In the end, the whole thing seems mean-spirited and pointless.

The Symantec video included here is an entirely different animal. It’s exactly the sort of thing I approve of: A lighthearted look at a serious technical topic with only a gentle poke at the “opposition”. And I bet producing this video was a lot less expensive than hiring an acrobat school! Let’s hear it for social media!

Disclaimer: Symantec is a frequent presenter at Tech Field Day and I have worked with them often on other projects. But this post is my own idea and I’m getting no compensation related to this video. I just liked it and wanted to spread the word!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | How Many Social Media Marketing Campaigns Fit Into a Mini Cooper?
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Single Instance Storage

It’s fairly simple for a system to eliminate duplicate data by storing only a single instance of multiple identical files. In other words, if you and I both upload “Presentation.pptx” and it’s bit-for-bit identical, it would be a simple matter to store just one copy.

Dropbox definitely does this. I proved it with a simple experiment:

1. Create a new 10 MB encrypted disk image in TrueCrypt (so it’ll be 100% unique, random data)
2. Move it to the Dropbox folder and wait a few minutes as it uploads
3. Copy the file with a new name to the folder and notice that it “uploads” instantly

Dropbox is at least single-instancing storage. This helps users, since it speeds uploads and reduces bandwidth usage. It helps Dropbox in the same way, but goes further since they still “charge” files against your account whether they’re single-instanced or not.

Note that this single-instancing works across users and geographies. I gave a file to a friend to upload to a different Dropbox account, and saw the same “acceleration effect.” This would be quite useful to users and the company for files like iTunes songs which are identical and widespread.

Clashing MD5 Hashes?

Three files with identical sizes and MD5 hashes but different names? Creepy!

A global single-instance storage system sounds great, but it opens the door to hash collision issues. Imagine if you and I both uploaded identical files. Both would have the same “fingerprint” and Dropbox would only store it once. Now imagine instead that, out of coincidence or malice, I uploaded a file with the same fingerprint as yours but different contents. This is not so far-fetched as it seems, and could lead to all sorts of security nightmares.

A common and compromised file checksum method is MD5, so I decided to test how Dropbox handles files of identical size, name, and MD5 hash using the “Nostradamus Attack” PDFs generated by Marc Stevens. My tests show that Dropbox correctly handled the files I tried, and no combination of uploading and naming could force it to incorrectly store the right file. So Dropbox either doesn’t use MD5 or uses a combination of hashing and other mechanisms. Testing other schemes is left as an exercise to the reader.

One more thought: The fact that de-duplication is mentioned in the “privacy” section of the Dropbox policies raises my eyebrows, since it indicates that they see this hash collisions as a matter of privacy rather than data corruption. This indicates that Dropbox is both aware of and susceptible to hash collision attacks generally, though obviously not as simply as creating a bogus MD5 match.

Note: Dropbox is well aware of this issue, having recently squashed an open-source exploit called Dropship!

Sub-File De-Duplication

Data de-duplication is like single-instancing, but it applies to some subset of data. Some enterprise storage systems de-duplicate at multi-megabyte levels, while others are far more granular.

To test whether Dropbox de-duplicates data, I devised a simple experiment:

1. Create a new local copy of my existing random TrueCrypt file
2. Add a single byte to the end using the “cat” command
3. Copy the resulting file to Dropbox
4. Watch as Dropbox takes just a few seconds to upload the new file

This test proves that Dropbox does indeed de-duplicate at the sub-file level. Since it took a bit longer to upload that would be expected for a single byte, we can see that Dropbox “chunks” files for hashing and uploading.

De-Duplication Granularity

The next question is just what size chunks or blocks Dropbox uses to de-duplicate data. To test this, I created various blocks of random data using TrueCrypt and experimented to see where the “stair-steps” were in terms of upload time.

My tests used four basic building blocks of 512 KB, 1024 KB, 2048 KB, and 4096 KB in size. Guessing that Dropbox used one of these sizes for their chunking system, I assumed these would quickly demonstrate the answer.

On my Mac, Dropbox clearly uses a 4 MB "chunk" size for deduplication

First, I uploaded each file individually and watched as Dropbox took about 30 seconds per MB. This will vary greatly, of course, but the absolute performance doesn’t matter. Only relative performance matters for demonstrating chunking.

Next, I concatenated each file with itself to create a new file twice as large. This would be ideally “chunkable” since it consists of exactly identical data with a nice, clean, evenly-divisible “border”. I uploaded each of these and noticed that the “4096 KB x 2″ file uploaded nearly instantly, while all others took the expected amount of time.

I repeated this with “x 3″, “x 4″, and “x 8″ files and noticed that the 4096 KB (4 MB) “barrier” was very obvious. Whenever a file contained 4096 KB or less of data Dropbox had seen before, it single-instanced it. Any time it saw a unique “block” smaller than this, it uploaded it fresh.

This proves, at least in the case of my own Mac OS X install of Dropbox, that a 4 MB chunk size is used for de-duplication.

 Stephen’s Stance

Dropbox is a very useful service, and I appreciate the technology they use to make it work. By single-instancing storage, the company is able to keep costs and transfer time in check and offer a basic service for free for many users. Despite the recent security issue, I continue to use Dropbox myself and would not hesitate to recommend it. But I do suggest using your own encryption for any sensitive data, as demonstrated in my recent post, Mac Users, Secure Your Stuff in Dropbox.

I remain somewhat concerned about the privacy and security implications of global de-duplication of shared random data. If they use SHA-1 hashes alone, which I suspect, there is a chance that an object will not be stored correctly once 2^80 (or perhaps 2^69 or even 2^52) objects are stored. This would lead to issues of data corruption or inadvertent disclosure. This is a very remote chance indeed, but “birthday problems” like this work against hashing systems. I would love to hear from Dropbox regarding how they prevent this from happening, including disclosure of their methods of hashing data. It’s nice to see the company taking responsibility by disclosing this in their privacy policy, though!

Update: Dropbox apparently does indeed use raw SHA256 hashes to “uniquely” identify data, and this can be exploited in a number of ways.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | How Does Dropbox Store Data?
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Join me in Chicago for Storage Decisions, June 21

Next month, I will be heading to Chicago for TechTarget’s Storage Decisions conference. This show does a good job on the editorial side, suggesting timely topics and bringing in independent voices like Howard Marks. I will have three presentations to give: Sessions on data reduction and storage virtualization in the main conference track, as well as a dinner discussion focusing on controlling the growth of data. Registration is free for qualified end-users, and I urge you to attend on June 21, 2011.

Reclaim Capacity with Data Reduction for Primary Storage

Depending on which industry study you read, most companies are wasting anywhere from 30% to 50% of their installed disk capacity, which translates into thousands of dollars spent with no effective return on investment. Storage vendors are beginning to provide tools that can help storage managers make the most of the disk they have installed. For example, data reduction for primary storage borrows data deduplication technology developed for backup and classic compression algorithms to help squeeze the air out of nearline and primary data and reduce its footprint. This session’s topics will include an overview of data reduction technologies and where they will have the greatest impact, what key storage vendors are offering in data reduction and an update on the major players, and the consequences of using primary data dedupe along with dedupe for backups. We’ll also look at the potential for vendor lock-in and consider why we’re reducing data in the first place.

Topics include:

• Introducing data reduction technologies
  • Compression: How it works and where it’s found
  • Deduplication: From single-instancing to variable block
  • Application-specific: Cracking open files
• Overview of data reduction products
• Where to use them
  • The capacity conundrum: Store less and reduce utilization
  • Ideal applications: Justifying the cost of data reduction
  • Side effects: Considering the impact on backup, replication, I/O workload and vendor lock-in

Storage Virtualization: Who’s Doing It and Why

Storage virtualization has been around for decades and, although research indicates that 70% of companies have already virtualized at least some of their installed block or file storage, most remain unaware of this technology. Grandiose schemes for comprehensive virtual SANs have given way to more practical host- and array-based virtualization technologies, and server virtualization has created a new opportunity to create a pool of storage. This session will look at the current state of storage virtualization, how to quantify its benefits and describe which approaches are best for particular environments, and also cover how storage virtualization compares to private storage clouds.

Topics include:

• Defining storage virtualization: What it is and where to find it
  • Abstraction of storage resources
  • Tiered storage
  • Flexibility
• Popular approaches to storage virtualization
  • SAN controllers
  • File virtualization
  • Volume managers
• The pool, the hypervisor and the cloud
  • The impact of server virtualization
  • Is this a private cloud?

Cutting Off Data Growth at the Disk

In this special dinner presentation, Stephen Foskett will discuss how to apply key data management technologies to arrest the growth of data. You’ll learn how capacity optimization technologies such as data deduplication and compression can reduce the trajectory of data growth as well as how tiering can reduce the cost of storage. Finally, Stephen will explore why the time may have finally come for active archiving and will leave you with practical ways to help your corporation better manage its data.

Note that space is limited for the dinner, which is sponsored by my friends at Dell.

Registration

To register for Storage Decisions Chicago, just go to the TechTarget registration page. Dinner guests will apparently be selected from that same pool of attendees.

Disclosure: TechTarget pays my expenses to attend and present at Storage Decisions, and has for many years. I also get a speaker fee for the dinner session.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Storage Decisions Chicago: All About Capacity Optimization
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Introducing InSync

Druva InSync is not a general-purpose enterprise backup application, though it is targeted at the business market. Instead of going head-to-head with successful enterprise backup applications like Symantec backup exec and net backup, EMC legato, IBM TSM, CommVault, and the like, Druva wisely found a different niche.

Everyone hates their remote backup application, and this is where Druva decided to focus. End-user laptop and desktop backup clients have a reputation for slamming performance and failing to protect data for machines on the go. A few years back, my work laptop at one of the leading remote backup solutions installed by corporate IT. I hated every time a backup kicked off, since I couldn’t get any work done until it was finished. So I secretly enjoyed the fact that it regularly failed to start. I guess like most end-users, I would rather get my work done than protect my data.

Druva InSync tackles these problems by focusing on simple installation and low resource requirements. The client install is only about 20 MB, and is configured to leverage the duplication and WAN acceleration technologies for quick backups. It also throttles CPU utilization and doesn’t rely on a fixed schedule like many solutions. The client can be remotely administered an updated once it is installed, keeping everything running smoothly, but users have some flexibility to add additional backups according to system policy.

The InSync server is remarkably tiny, weighing in at under 50 MB and installing in just a few minutes. While most backup applications require installing additional software, including Microsoft’s heavy-duty SQL Server, Druva InSync is completely self-contained. But it includes many advanced features, including SSD support and memory caching to improve de-duplication performance.

Druva InSync Installation and Configuration

I recently worked with Druva to produce a series of videos documenting the installation and configuration of InSync. As part of this process, I went through the entire roll out myself using virtual machines and real data. The result was eye-opening: InSync really does install in under 10 minutes!

My demonstration deployment included a Windows Server 2008 machine  and a Windows 7 client. Both ran under VMware Fusion but were configured with realistic CPU and memory footprints to approximate a real world environment. In my first video, I download and install the Druva InSync server and client software, and get everything up and running.

My next video includes some more advanced configuration topics. I create separate users for administration of accounts and profiles, discuss remote access and adding WAN network ports, and do some advanced storage configuration. I also discuss some of the best practice recommendations that the Druva folks told me about.

Finally, my third video includes a discussion of reporting features as well as some troubleshooting ideas.

Stephen’s Stance

Installation and configuration of Druva InSync is impressive, but what it does once it’s installed is what really matters. In my tests, I was impressed by the performance of the de-duplicated in accelerated data transfer from the client to the server. According to the Druva InSync dashboard, I was getting almost 2 to one reduction in data transfer on the very first use, and this looks to get much better over time. Druva’s internal videos show massive data reduction for clients that have been running for a while.

This got me thinking about the possibilities of using a product like InSync to back up web servers at remote hosting providers, and application I’m eager to try out. Druva recently introduced Phoenix, a server backup product to do just this. The company also has recently introduced a data loss prevention component for InSync called SafePoint. It looks like I will get a chance to try this product out very soon!

Disclaimer: My company, Foskett Services, was hired by Druva to produce these videos, and Druva sponsored Tech Field Day, an event I organize.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | You Really Can Have a Complete Backup System Up and Running in 10 Minutes!
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although the core issues with thin provisioning revolve around communication, it presents unique challenges to the storage array as well. We talked about granularity of pages, and the comments for that piece were extremely enlightening. Now let’s consider another key factor: Scheduling.

Note that the “provisioning” part is relatively easy to do on the fly: An array just has to allocate additional capacity as writes come in, which is something it does anyway. It’s the thin reclamation that poses a challenge, since this involves zero detection across a whole page of data in many cases.

Just like de-duplication, thin provisioning challenges the resources of the storage array to do background number crunching. And just like dedupe, the array engineers have a choice of when to do the reclamation processing: Well after writing or “in-line”. The extreme ends of this spectrum fall into two equally disappointing categories: Wholly ineffective or ridiculously intensive.

Let’s start with the “intensive” side: You could have the controller do thin provisioning automatically; that’s kind of what IBM does with SVC, for example, and 3PAR claims to do this too. The trouble is that the controller has to literally watch everything, and it’s got to reassemble whole pages, perhaps 42 MB or even one GB in cache. If it didn’t have all that data, it would have to go fetch it, put it into cache, look at it, make sure it was all zeros, then get rid of it. It’s really, really difficult to do automatic, in-line, thin provisioning. It’s a good thing to do, but it’s a hard thing to do.

So most vendors schedule thinning for later. In the “10 terabytes of zeros” example, they’re actually going to write 10 terabytes to disk, or at least through to cache. Then, at some point in the future, they’ll go back and reclaim that space. Some are pretty aggressive and reclaim capacity very frequently. Others are fairly lazy: The Drobo seems to reclaim only once or twice a day. A lot of people who have them are surprised when the thing springs to life and starts going, “Bada-bada-bada-bada-bada-bada.” Apparently it’s reclaiming storage at that time.

Some thin provisioning systems are even manually-initiated, and this is really pretty ineffective. The storage administrator has better things to do than reclaim storage all the time, so they are probably going to set a cron job to do it regularly at a specified time. If the system only does it on demand, that means that it doesn’t have the horsepower to do it automatically. Ergo, it’s sometimes going to conflict with “real work” and cause a problem.

I would look for a system that was fairly aggressive with thin reclamation. I was talking to the guys at Nimbus Data, for example, and they claim to do thin provisioning in-line all the time. I hope that we see more storage arrays that are doing that, and less that are doing it manually, on demand, because that’s just not as useful.

But considering that thin provisioning used to be almost useless, the fact that it’s now at least somewhat useful is gratifying.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Processing and Scheduling Thin Provisioning
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

W. Curtis Preston launched his own series of Backup Central Live! seminars for 2011

Last week, after the Exec Event in Palo Alto, I joined my friend W. Curtis Preston for his first Backup Central Live! event. Curtis has spent years educating IT pros about data protection, this was the first week of a new series of self-produced events. And let me tell you, although I’ve seen him present dozens of times, Curtis was really in his element here. He held the packed room enthralled, and the vendor sponsors I talked to were very pleased about the event!

Introducing Backup Central Live!

The Backup Central Live! series are day-long seminars across the USA in 2011. Each event includes over 3 hours of content from “Mr. Backup”, W. Curtis Preston, as well as presentations from Jacob Farmer and the sponsoring vendors. The seminars are free for qualified end-users, which includes most of the readers of this blog!

Curtis and company will cover the challenges of backing up and recovering data in a variety of settings:

• Virtualized servers (e.g. VMware, Hyper-V, Xen)
• Very large servers and data centers
• Remote offices and laptops
• Data retained for multiple years

The session also includes technical detail about key products and technologies:

• Cloud Backup Services
• Deduplication
• Continuous data protection (CDP) and near-CDP
• Archive software
• Tape and its proper role

Attendees even get free breakfast and lunch, which was of a good hotel caterer quality in my opinion.

Stephen’s Stance

The Backup Central Live! crew does a great job putting together a professional event

I knew Curtis could put together quality backup content, but the crew deserves credit for such a professional and successful event. They attracted some great sponsors, too, including AppAsure, Aptare, FalconStor, NEC, Quantum, Spectra Logic, and Cirtas. And Jacob Farmer’s involvement was a pleasant surprise, too: I’ve always enjoyed the deep technical conversations I’ve had with him!

If you enjoyed my own backup, archiving, and storage seminars in the past, I know you’ll love this event. The next Backup Central Live! cities are as follows. If you’ll be around, you really ought to attend!

• Orlando, FL Feb 1 Register here
• Houston, TX Feb 8 Register here
• Chicago, IL Feb 22 Register here

My only suggestion for the crew is that they get a bigger room next time!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | See W. Curtis Preston’s Backup Central Live!
This post was categorized as Enterprise storage, Gestalt IT, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.