Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
This post was categorized as Enterprise storage, Everything, Gestalt IT, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

When is a gigabyte not a gigabyte? When you're not buying gigabytes!

I stepped into a hornet nest this week when I posted a write-up about a new flash storage array from Pure Storage. The controversy had nothing to do with the underlying technology, which seems quite sound. Rather, it was all about pricing, with Pure’s competitors calling foul on their price comparisons.

You’re Not Buying Gigabytes

In a world of 3 TB drives, storage capacity is almost irrelevant. Capacity is what people think they are getting when they buy enterprise storage devices, but capacity is only one aspect of the purchase, and it’s not a very significant one in most cases.

So what are enterprise storage buyers buying?

• Performance, especially I/O operations (IOPS), is much more critical than capacity in most cases, and it takes lots of spindles or expensive flash chips to deliver it.
• Data protection features like snapshots are increasingly important, and often cost extra.
• Compatibility is paramount, as is long-term supportability from all vendors involved.
• Integration and management features are often the deciding factor in purchases, especially when it comes to server virtualization applications.
• High availability and product support are “must-haves” that can multiply the cost of a solution.
• Power, cooling, and floor space can be very important for some applications and entirely inconsequential in others.
• Capacity is sometimes important, but many applications require just a few TB or less and thin provisioning, data deduplication, and compression are really blurring the lines here.

So although a typical customer will say “I need 200 GB for this application” they likely need nothing of the sort. They really need 100 IOPS, snapshots, a line on the HCL, VAAI and vCenter plugins, and redundant everything. Even the capacity number is questionable: Most applications grow over time, and few need much capacity really.

Since you can’t buy a 1 GB storage array and can’t fill a 10 TB device to 100%, pricing per GB is complete nonsense. Plain old storage space just sort of tags along for the ride once you build a system capable of meeting all these other needs.

Data Reduction or Pricing Obfuscation?

Utilization of storage capacity has always been terrible, but improving capacity efficiency is worthless. The best you can do is over-tax your array or put all your data “eggs” in too few drive “baskets”. Achieving impressive capacity utilization just concentrates I/O, and this is the last thing you want to do with spinning hard disk drives.

This is why I suggest redirecting the conversation away from capacity requirements. The amount of GB to be used and the efficiency of that storage doesn’t matter all that much except for certain massive and rare applications. Once the array is big enough to handle the data, everything else is a wash.

This is also why I’m skeptical of data reduction technologies. Most applications would be better off optimizing for performance not reducing capacity used. And data reduction techniques like compression and deduplication quickly lead down the “your mileage may vary” rat hole.

Comparing Apples to Apples

Also read Grapples and Tangelos: Why it’s Impossible to Compare Fairly

There is only one way to do a real fair comparison between different storage devices: Specify all the requirements and let each vendor put forward whatever they have that meets all of them. Who really cares if vendor A’s disk-based solution is 10% utilized while vendor B’s flash array needs 1/5 the capacity? As long as you have a place to put it (and enough power to feed it) it’ll still work fine.

One serious challenge in enterprise storage is the rise of flash memory as a storage medium. Flash chips are expensive on a data capacity basis but amazingly cheap in terms of performance and environmental efficiency. Put another way, an SSD can’t storage as much data as a hard disk, but it delivers massive I/O capability in a tiny, rugged, low-power footprint.

Since most enterprise applications need only a few hundred GB of capacity, a few SSDs can be a compelling alternative to a “refrigerator” full of disks. It can be hard to convince the boss, but you really can fit a whole datacenter’s worth of storage I/O into a few rack units!

Pure and Nimbus

This is the issue facing flashy solid state devices from many companies, and the root of my headaches this week. Pure Storage hasn’t finalized pricing yet, but are claiming that their new device costs $5 per usable gigabyte. This is incredibly cheap for an array that can blow the doors off most enterprise gear!

Nimbus Data, on the other hand, sells their all-flash enterprise storage array for about $10 per GB. But this is not the end of the story, and Pure might even be more expensive than Nimbus! Or maybe not. It all depends on what you’re comparing.

Pure claims that their cost is half the price of most comparable flash storage array competitors, but this is where the questions start to appear. Is that $5 gigabyte usable or raw? Does it include high availability? And can I really store any old gigabyte of data there or is that a compressed/deduplicated gigabyte?

It turns out that the real cost of Pure Storage capacity is $20 per GB including RAID and an extra mirrored array for high availability. But since every byte written to the array is thin provisioned, deduplicated, and compressed, many customers will pay much less for actual data stored. And since it’s an all-SSD array, it’ll perform way better than a disk-based system, too.

Muddying the Waters

So why not just call it $5 per GB and be done with it? It’s confusing, that’s why, and your mileage will vary widely.  Pure’s own slides show some applications getting 4:1 data reduction and others all the way up to 17:1. So these applications would be paying as low as $1.18 per GB or as high as $5.

But you can’t buy just 1 GB of storage from Pure. Their smallest array (which includes one controller and one shelf of SSDs) provides 5.5 TB of raw capacity, presumably using 24 256 GB SSDs. A high-availability configuration would include two controllers and two shelves of SSDs for 11 TB of raw storage. That’s going to cost almost a quarter of a million dollars according to my calculator. That’s one expensive gigabyte!

Of course no one would buy this array to store just a thousand megabytes. They would buy it to support a bunch of applications that need capacity and performance and efficiency and integration and everything else. And they can buy a mirrored pair of arrays from Pure Storage or Nimbus or Violin Memory or Texas Memory Systems or others at a variety of price points.

The only way to really compare these products is to spec them out on equal footing and see what the price tag looks like. These comparisons would include data reduction, but they would also have to bring in high availability and every other requirement of the applications they will support.

Stephen’s Stance

It’s way too difficult for me to do the pricing math for these systems, so I’m throwing in the towel. I’m thrilled to see all-flash arrays made available to IT buyers. This wouldn’t be possible without clever use of thin provisioning and data reduction, as well as smart software to overcome the limits of SSD.

I’m going to guess that Pure and Nimbus will cost about the same for similar configurations, though I’ll bet each believes they’re cheaper. Rather than get in the middle, I invite each company to post a comment below stating their case. I’ll even embed their responses into a future blog post on the subject if they get too long. Just don’t ask me to be the referee.

Update: Pure Storage responds with an outline of their pricing:

• How Pure Storage Delivers All-Flash Storage at Below the Price of Spinning Disk

Image credit: Rotten Apple by Wappas

© sfoskett for Stephen Foskett, Pack Rat, 2011. | When Pricing Gets Squishy Competition Heats Up
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• Hop By Hop TCP
• What is a Switch Network Fabric ?
• Deal: 1800 mAh iPhone backup battery for a measly $13
• Web-based jailbreak returns, supports iPad 2 and any other iOS device
• Rumor: Apple soldering MacBook Air SSD to motherboard (and why it’s a bad idea) (updated 2x)
• Zerto: Or What I Learned at Tech Field Day #6! | vNinja.net
• The Secret is out and it’s called Zerto.
• 10 quick and easy ways to boost storage performance
• Ladies, hide your porn! « storagemistress
• The Secret 10 Step Guide to Giving Good Social
• Wireless CCIE, here I come!: The History of Wireless Part One
• Wireless CCIE, here I come!: Switching from a Blackberry 9630 to an iPhone 4
• Efficiency, Performance, and Locality
• Tech Field Day – June, Boston

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, July 8, 2011
This post was categorized as Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Eye-Fi X2 card packs a 200 MHz ARM CPU and limited Marvell 802.11b/g/n Wi-Fi chipset

My experience with the Eye-Fi connected SD camera card has been frustrating, but I wasn’t sure how much blame the company deserved. Uploading massive 14 megapixel photos is going to be slow over a 3G connection using any device, after all. But how fast could the Eye-Fi go on a really great wireless LAN? The company is skimpy with technical specs, but I did manage to figure out just what the Eye-Fi X2 series of cards are capable of in terms of CPU and Wi-Fi performance.

Eye-Fi claims that the X2 card line is capable of “Class 6” read and write speed inside a camera. Class 6 means 6 MB/s, and one might think this is the maximum throughput for the card. Considering that my camera can shoot seven frames per second at 14 megapixels (each image being about 6 MB in size), the Eye-Fi could not handle heavy-duty use when set to transfer all images immediately even if this was the real-world performance one could expect.

This only gets worse when one considers the Eye-Fi Pro X2 card with its RAW image compatibility. My Sony NEX-5 shoots 15 MB RAW images, about three times the size of a “fine” JPEG. This means it would take three times longer to transfer each image, a truly frustrating experience even with the fastest network.

What’s Inside?

Eye-Fi (the company) would rather that we focus on the capabilities of their card rather than its technical components. But any self-respecting geek is going to want to know what makes it tick! I’d rather not cut open my card to get a peek at the chips inside, but Eye-Fi released some official details about the components used in the X2 series of cards, and a quick Google search revealed all that I needed to know.

An official company blog post includes a cutaway image of the inside of the card as well as details about the new X2 chipset. According to Eye-Fi themselves, the X2 line includes a new engine called “Arcturus” which includes a 200 MHz ARM926 processor core. The 926 is part of the 32-bit ARM9 RISC family and includes a digital signal processor, Java acceleration, and local cache. This is not a bad chip, considering the ultra small form factor of the Eye-Fi card.

So the card has enough CPU juice to handle reasonable performance requirements, but what about the Wi-Fi chipset? Eye-Fi doesn’t say too much about the capabilities of the X2 card line, except to boast of their newfound 802.11n compatibility. But the markings on the Wi-Fi chip are clearly visible in Eye-Fi’s official photographs, and a quick search reveals very limited capabilities.

Eye-Fi Wi-Fi

You might want to refer to my 802.11n Overview

The Eye-Fi card uses a Marvell 88W8786 integrated system-on-chip WLAN controller. This is an early 802.11n chip with few features:

• The radio is capable of 2.4 GHz transmissions only, so it is incompatible with 5 GHz 802.11n networks
• Like many portable devices, a single transmit and receive antenna is used so MIMO performance gains are restricted
• The datasheet lists 20/40 MHz coexistence, so it must support 40 MHz channels on 2.4 GHz, a feature that is highly unlikely to be used given the limited number of channels there

In short, the Eye-Fi X2 card is “802.11n in name only” and does not support most of the advanced performance features users might expect. Theoretical data rate with 40 MHz channels is limited to 150 Mb/s, and throughput with 20 MB/s channels tops out at 75 Mb/s, with much less in the real world.

Stephen’s Stance

My own experience shows that the Eye-Fi X2 card takes 3 to 5 seconds to transfer a 6 MB image to my laptop using direct mode. This translates into roughly 12 Mb/s, and represents a best case scenario for image transfer. This drastically reduces the value of the Eye-Fi card when used with high-resolution cameras. Which are exactly the kind of cameras that people might have who are willing to spend more than twice as much for a connected SD card!

Then there’s compatibility. The Eye-Fi card does not support 5 GHz-only 802.11n networks. This isn’t unique – neither does the iPhone 4! But it’s bound to disappoint and frustrate some customers. Products like this are the reason I decided to set my AirPort Express (an either/or base station in terms of radio bands) to use 2.4 GHz even though it is “N-only”.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | What Are The True Eye-Fi X2 802.11n Wi-Fi Capabilities?
This post was categorized as Apple, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The new MacBook Pro impresses with its performance

As I mentioned in my previous article, I decided to buy the 13″ Core i5 (base model) MacBook Pro. It meets my needs as a travel workstation, but how does it perform? I decided to benchmark it against my other Macs to see how it stands up.

Benchmark Details

The following benchmarks were performed using Xbench and Geekbench, running on a freshly-booted machine. Xbench is outdated but solid and universally-accepted as the standard Mac benchmark. Plus, I had historical data from all of my machines. Geekbench is a great cross-platform CPU test, but it doesn’t measure as wide a variety of system parameters as Xbench.

Each Xbench test was normalized against the new MacBook Pro, which always shows as “100%” in these charts. This should help get a feel for how much slower or faster it is than the other machines.

The test subjects are as follows:

This probably seems like an odd and motley assortment, but they all have one thing in common: I own them. Although everyone’s performance baseline will be different, I was interested in how the new MacBook Pro compares to my other machines, and my 15″ machine in particular. So there you have it!

The CPU, graphics, and memory tests were performed running the latest version of Mac OS X “Snow Leopard”, version 10.6.6.

The disk tests, however, were run under either 10.6.6 (in the case of the 13″ MacBook Pro and iMac) or the version of Mac OS X that came with the machine originally (in the case of the 15″ Santa Rosa MacBook Pro and the Mac Mini). Since I am a storage guy, I have long-since upgraded the hard disk drives in both machines, and felt it was unfair to compare the OEM drive in the new MacBook Pro to these upgraded drives. So I used the original Xbench tests I performed when the machines were new.

Performance Overview

The new 13" MacBook Pro performance admirably

As others have reported, overall performance is solid. Even though it is the absolute base model in the line, the new MacBook Pro matches or bests my old machine in every respect. It clobbers the Santa Rosa MacBook pro in CPU, Thread, Memory, Quartz, UI, and disk tests, and ties in OpenGL performance. It even matches the high-end desktop in most tests, only falling behind when it comes to Disk, Thread and OpenGL graphics performance.

The fact that it achieves all this with a base price $600 less than my old MacBook Pro and runs for almost 7 hours on a charge is simply amazing. Moore’s law ought to allow new machines to outperform old ones, but one is still surprised to see it so flamboyantly displayed.

Performance Details

Now let’s dig a little deeper into these performance numbers!

CPU Performance Details

The base-model 13″ MacBook Pro uses a 2.3 GHz dual-core CPU, which hardly sounds better than the 2.2 GHz Core 2 Duo chip used in my old MacBook Pro. And it shouldn’t hold a candle to the mighty 2.66 GHz quad-core “Nehalem” Core i5 in my iMac.

The "Sandy Bridge" Core i5 performs very well - it's nearly twice as fast as the old Core 2 Duo!

The detailed CPU tests tell a different tale, however. The architectural improvements made between Merom/Penryn and Sandy Bridge are evident, with the new chip almost doubling the old in floating point math and (thanks to hyperthreading) solidly throttling it in thread tests.

The most impressive feat is its performance relative to the quad-core Nehalem Core i5 in the iMac, however. Looking past the thread tests, which are greatly helped by two more full cores, we see nearly equal performance between laptop and desktop. This suggests that the Sandy Bridge architecture does a solid job of reducing electrical demands without sacrificing performance. The quad-core desktop CPUs in this family are shockingly strong, as we will see in a moment!

Geekbench Results

Since it is a cross-platform benchmark, I was able to add a ringer to the Geekbench test: My new lab workstation. Built around a Sandy Bridge Core i5-2400, Asus P8H67-M PRO motherboard, and speedy OCZ memory, this is a seriously-fast desktop for very little money. All in, we’re talking about under $500 for this guy!

The Core i5 and i7 processors walk all over the old Core 2 Duos

The two Sandy Bridge Core i5 CPUs are neck and neck in most of the tests, which is really shocking given that the desktop has two more cores, can ramp to 95 Watts, and runs at 3.1 GHz. It is also impressive that the older Nehalem Core i5-750 can keep up in Geekbench tests. Unsurprisingly, the old Core 2 Duo machines aren’t in the same league, not managing even half the performance of these three.

I previously talked about the performance-per-dollar ratio of the various MacBook Pro machines, noting that the base models were much more attractive by this metric. Although it’s definitely not an apples-to-apples comparison (for starters, my desktop is harder to use on a plane…) I will admit that one can get roughly three times the Geekbench score by building a Sandy Bridge system at home. With a score of 7350, my desktop delivers almost 15 Geekbench points per dollar, compared to 3.5 to 5 points from the new MacBook Pro line.

Memory Performance Details

Both the MacBook pros sport 4 GB of RAM, while the Mini still has just 2 GB and the iMac has been upgraded with 8 GB. The old MacBook Pro uses 667 MHz PC-5300 RAM, while the Mac Mini and iMac use 1066 MHz PC-8500 SODIMMs. The new machine steps up to 1333 MHz PC-10600 memory.

The "Sandy Bridge" platform really shines when it comes to memory performance

The Nehalem series integrated the memory controller with the CPU, and this is continued in Sandy Bridge. This allows really amazing memory performance across the board – the old CPUs are stuck with 30% to 40% of the new machine’s memory access capabilities.

The new chipset even manages to beat the iMac in many tests, with only the thread-sensitive System Copy test showing a real loss.

Graphics Performance Details

Graphics performance is the one area I was most concerned about. All three older machines use discrete graphics controllers of various sorts, from the wimpy Nvidia GeForce 9400M in the Mac Mini to the Nvidia 8600M GT in the Santa Rosa MacBook Pro to the more-impressive ATI Radeon HD 4850 in the iMac.

Intel's integrated HD 3000 graphics do a fine job, generally keeping up with the discrete GPUs found in the other machines

As expected, the discrete graphics cards are much more competitive with the new HD 3000 engine integrated into the Sandy Bridge Core i5. Yet once again, the new machine is able to match or beat the old machines in nearly every test.

OpenGL performance seems to be an issue for Intel’s new chip. Perhaps a driver update might improve the situation? But it’s still solid – matching the older machines and only throttled by the big desktop. Most of the Quartz graphics tests show the iMac and new MacBook Pro tied with the old machines trailing far behind. Perhaps they are CPU-bound?

Hard Disk Performance Details

Finally we turn our attention to the question of storage. Hard disk drive performance depends on many factors, and Apple’s machines have historically varied quite a bit. Every one of the new MacBook Pros come standard with a mundane 5400 rpm Hitachi hard disk drive, so one cannot expect it to match the performance of the full-size 7200 rpm desktop drive in the iMac.

Although it's nothing to write home about, the 320 GB Hitachi drive performs adequately

Density improvements should give the new MacBook Pro a leg up on the old Mini and ‘Pro and, our tests bear this out. None of the disks are really all that impressive (sequential reads and writes in the 65 MB/s range aren’t impressive) but it’s not bad.

Stephen’s Stance

The Sandy Bridge MacBook pro really shines in terms of performance. It soundly beats my old laptop in nearly every test, and even gives the desktop a run in some tests. In all, I’d say the hard disk drive ought to be the first thing to get an upgrade. Throw in a speedy SSD and we’ll be looking at some really earth-shattering performance and battery life. And yet, we’d still be looking at a sub-$2000 machine!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Benchmarking the 2011 13″ Core i5 MacBook Pro
This post was categorized as Apple, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

CalDigit USB 3.0 PCI Express Card Review

I recommend the CalDigit PCI Express card for Mac Pro users with a need for (storage) speed!

No, Apple didn’t send me a sneak peak at some new hardware. My USB 3.0 experience comes courtesy of CalDigit, who sent me their Mac OS X-compatible USB 3.0 PCI Express card for evaluation. I’m pleased as punch with the card and software drivers they delivered, and it makes me crazy that this isn’t included by default in Mac Pro desktops, let alone the new MacBook Pros to be introduced tomorrow.

The CalDigit USB 3.0 adapter is a PCI Express card intended for use in a Mac Pro. A graphic artist friend of mine was kind enough to let me use his very-expensive Apple workstation for testing, and was so impressed that he turned around and ordered a CalDigit card for himself. Installation is a snap – just open the Mac Pro, unscrew the retainer above the PCI Express slots, slide the card in place, and screw everything back together. (Side note: I love Apple’s tower case design!)

I didn’t see a 4-pin mini floppy power connector inside the Mac Pro case, but the CalDigit card does include a compatible header. This would give the ports more power than they can draw from the PCI Express bus itself, but I didn’t see the need. Using only the PCI Express bus, I was able to spin up and use every USB bus-powered drive I tried with no issues.

Installing the software was a snap, too. CalDigit’s driver appears to be locked to only their card (I tried it with a variety of other USB 3.0 cards with no success) but it did support every USB 3.0 drive I tried. This is in stark contrast to the LaCie USB 3.0 driver which only talks to LaCie drives! Specifically, I connected two different Seagate GoFlex USB 3.0 drives, a generic USB 3.0-to-SATA adapter, and Iomega’s screaming-fast USB 3.0 SSD. Every one was instantly available to Mac OS X and visibly out-performed FireWire and USB 2.0.

Real-World Tests

Since this was not my own machine, I was not able to perform my usual benchmarks. But I did test some copy operations, experimenting with USB 2.0 and FireWire 400 and 800 connections. The 1 TB Seagate GoFlex drive pushed over 100 MB/s when using the CalDigit USB 3.0 card, according to my iPhone stopwatch, but were limited to about 45 MB/s and 30 MB/s when using FireWire 400 and USB 2.0, respectively. I had previously tested this drive using eSATA on my iMac and found it topped out at about 110 MB/s, so the drive itself appears to be the bottleneck when using USB 3.0.

Swapping in the Iomega USB 3.0 SSD was eye-opening. This drive proved to be blazing fast in my tests earlier in the week, topping 200 MB/s in both read and write operations when connected to my Asus Cougar Point motherboard running Windows 7. I wasn’t able to perform adequate benchmarks with the Iomega, but my stopwatch showed it accelerating past the GoFlex and easily pushing 150 MB/s or more. I wouldn’t doubt that the CalDigit card is capable of 200 MB/s with an appropriate storage device.

The CalDigit controller lagged in writes, but performance was still impressive

The story was somewhat different under Windows. My instrumented tests (using Atto Disk Benchmark in Windows 7) showed a curious slowdown in write operations compared to the ASMedia USB 3.0 controller selected by Asus for my P8H67-M Pro motherboard. The CalDigit card and drivers matched the ASMedia at over 200 MB/s in read operations to the Iomega SSD, but lagged behind at 150 MB/s when it came to writes. I wonder if perhaps Mac-oriented CalDigit did not optimize their Windows 7 drivers for this card. Of course, 150 MB/s is still more than four times faster than USB 2.0, and I would never have noticed this if I was only using a hard disk drive!

Stephen’s Stance

If you own a Mac Pro, there is no need to wait for Apple to release USB 3.0 hardware and software. I can unreservedly recommend the CalDigit USB 3.0 PCI Express card for Mac Pro owners. The performance and ease of use is well worth the $79 MSRP. With so many external storage vendors rapidly switching to USB 3.0, the days of FireWire 800 being top dog in Mac performance are over. I’d love to connect the new USB 3.0-equipped Drobo S to this card!

CalDigit promised to send me an ExpressCard USB 3.0 adapter to try in my MacBook Pro as soon as they refresh their stock. I’m eager to try it out, since I’ve noted less-thrilling performance in the other USB 3.0 ExpressCard adapters I have tried. Those maxed out at around 110 MB/s in my Dell XPS/Windows 7 laptop, suggesting serious performance limits for the ExpressCard form factor. I am curious to see how the MacBook Pro performs in comparison.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | USB 3.0 For Mac Is Here!
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Once something is in place, it's hard to get it to move again

In Philosophiæ Naturalis, Sir Isaac Newton defined inertia as follows:

The vis insita, or innate force of matter, is a power of resisting by which every body, as much as in it lies, endeavours to preserve its present state, whether it be of rest or of moving uniformly forward in a straight line.

Although Newton was referring to physical objects, the power of inertia affects companies, markets, and relationships in the same manner.  Humans are creatures of habit, and change is challenging.  When faced with a choice of continuing along the same road or branching off in a new direction, most will choose familiarity.

Inertia in IT Architecture

Consider the impact of inertia on IT architecture: once the solution is in place, it tends to remain there for a very long time.  This rule applies to practices, architectures, solutions, and hardware and software.  It explains the continued presence of Token Ring, MS-DOS, Mac OS 9, and Palm organizers in so many companies.  It also explains the curious devotion IT pros field toward solutions that are backward compatible: Ethernet, Windows, Intel x86 architecture, and so on.

Once, while visiting the data center of a midsize financial institution, I spotted a stack of old IBM PC desktop computers in the corner.  The company had purchased a company, which itself had purchased a bank many years ago.  The loans from that long ago and far off institution were still serviced by this archaic hardware and software.  The company’s IT staff had squirreled away half a dozen replacement computers so they could migrate the application to new old hardware in the event of a failure.  If this isn’t inertia, I don’t know what it is.

Overcoming Inertia

An external force is required to overcome inertia, and one must desire to initiate a change.  New products and solutions must not merely be attractive, it must also be compelling enough to overcome this inertia.  In my experience, there are three reasons that companies change direction when it comes to IT architecture:

1. A noticeable irrefutable return on investment (ROI)
2. A tangible and necessary performance benefit
3. A unique and desirable function

Many new technologies show promise in all three areas, including 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE), server virtualization, and data deduplication.  But these potential benefits are not necessarily compelling in all IT environments.

A company with a substantial investment in Fibre Channel SAN hardware may find that upgrading to 8 Gb Fibre Channel is more compelling than a switch to converged networking and FCoE.  Many companies have found it hard to justify the additional cost of data compression or deduplication technology when compared with the decreasing cost of capacity or the benefits of improved utilization through better storage management.  The growth of server virtualization has been steady, but the hold-outs indicate that many companies find it hard to justify the technology.

Stephen’s Stance

Contrary to our nerd dreams, mere technical superiority does not guarantee the success of a new product or solution.  It must be better, faster, and cheaper to achieve widespread success.  In short, it must demonstrate a compelling case, or inertia will set in and derail its progress.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Three Requirements To Overcome Inertia
This post was categorized as Computer History, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

In the previous post, I talked about how the Drobo uses metadata monitoring to solve the telephone game and make de-allocation possible. But that approach is challenging in complex enterprise environments. Instead, most enterprise arrays use a complex chain of semaphores to interpret signals from the connected hosts about the capacity that can be un-provisioned.

On the storage side, arrays can only use the information they have to de-allocate: The data that’s stored on them. They don’t know what application is using it, what file system it is. They don’t know anything at all.

But, somewhere along the line, someone had a big idea and said, “wait a second, what if we look for pages that are all zeros?” We’ll talk about pages a bit later, but for now, let’s talk about zeros. A zero is kind of a smoke signal coming up from over the hills that says, “there’s nothing valuable here.”

So the storage array watches for pages that are all zero and reclaims them. As protection against making a stupid mistake (what if you actually wanted to write all zeros?), anybody who asks for a page that has been reclaimed just gets all zeros back.

Most of the major vendors support this kind of zero page reclaim. This is good stuff. I don’t want to sound too critical of them because I appreciate them implementing at least this.

The problem is that there’s not a lot of ability to actually have those zeros be written. Almost no operating system writes zeros to deleted space. If they actually wrote pages of zeros, thin provisioning would work great.

So what do the storage vendors do? They come up with utilities that write zeros!

NetApp has SnapDrive, which zeros out empty space so that the Filer can go and recover that space. You run it whenever you want to run it. Eventually the storage array notices that you’ve zeroed out that space and it recovers it. Compellent and Symantec’s Veritas Storage Foundation have something like that, too. You can also force it using the SDelete command, and you can configure it using VMware ESX.

Zero page reclaim is pretty straightforward. It doesn’t take a lot of computing power – It’s not like you’re watching the file system for changes or anything. All you’re doing is occasionally going through and deleting pages full of zeros. So, you can post-process it, kind of like de-duplication.

There are quite a few issues with zero page reclaim, though:

• Things aren’t writing zeros
• Most of these implementations are page-based, which looks like a problem
• Theoretically, this drives more IO through the system, not less

This last is the biggest problem, really. In most cases IO performance is a bigger issue than capacity in enterprise storage. If I could give you all the capacity you could possibly want or all the performance you could possibly want, most people would pick performance. It used to be capacity, but now it’s all about performance. If infrastructure folks could get one for free and had to pay for the other, they would definitely pay for performance.

And zero page reclaim, the way that it’s implemented with SDelete or with eagerzeroedthick, is driving tons of IO. Basically, a delete is the same as a write because you have to write all these zeros over the bus. But there’s a way around that, too. And that’s the topic for the next piece in this series.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Zero Page Reclaim: Savior of Thin Provisioning?
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Get used to it: Commodity hardware like this Super Micro SBB server will dominate enterprise storage

The history of technology moves in fits and starts, but one trend trumps all else: An inevitable shift from fine-tuned specialized gear to general-purpose commodity building blocks. We see it in both hardware and software, and at all levels of the industry, from chips and wafers to operating systems and networking devices. Take a step back and you’ll certainly agree: Commodity hardware always wins (eventually).

Doing the Impossible

Building something new often requires amazing feats of engineering. Steve Wozniak’s hardware wizardry with the original Apple computer is now legend, but similar tales are found everywhere.

It’s frightfully difficult to do the impossible without at least some wizardry. Most new technologies are therefore built on specialized hardware and hand-tuned software. Engineers at the forefront of technology must eke out every ounce of performance from trailing technologies.

This is why we see so many special-purpose processors in high-performance devices, and why companies employing custom ASICs often enjoy a performance advantage. In enterprise storage, we look to companies like HDS and BlueArc who packed their arrays with special-purpose hardware, pointing the way to the future. We also see impressive developments from Fusion IO and SandForce in the SSD space, leading to the next generation of storage.

Can You Keep Ahead Of Intel?

Just about every technology sector progresses from the impossible to the commonplace, and these changes are often very quick. High-performance storage systems were once exotic multi-million dollar devices but millions of IOPS are now available for under $100k from dozens of vendors.

This progression typically includes a move from special-purpose to commodity underpinnings. Exotic real-time operating systems have been pushed aside in favor of Linux, BSD, and even Windows, while ASICs and FPGAs give way to Intel’s CPU juggernaut. Even Apple computers are today almost entirely commodity PCs.

I love how Denton Gentry phrases this in his Monday blog entry about Intel and Achronix:

“Once you commit to a specialized hardware design, the clock starts ticking. There will come a day when a software implementation could meet the requirements, and at that point the FPGA becomes an expensive liability in the BOM cost. You have to make enough profit from the hardware offload product to pay for its own design, plus a redesign in software, or the whole exercise turns out to be a waste of money.”

In other words, specialized software on proprietary hardware will eventually be overtaken by general-purpose software on commodity hardware. The decision must include not just what one can do today but what that baggage will mean in the future. Designing a system around proprietary components might look good now, but the next-generation product will be put at risk by this decision.

Stephen’s Stance

My opinion is right there in the title of this piece: Commodity hardware always wins. No matter how great your ASIC is, industry-standard CPUs will out-perform it sooner or later. No matter how much effort you put into tuning your software, Linux-based systems will eventually do just as well.

The rise of commodity hardware is everywhere: EMC, HDS, IBM, Oracle, and HP have all embraced Intel CPUs and their hardware is looking more and more like Intel’s reference designs, too. Startups are increasingly relying on software rather than hardware for their differentiation, and we’re seeing Supermicro servers shipped with just about everyone’s name on them. The Storage Bridge Bay specification is looking better all the time, too. Commodity hardware is winning in storage, just like it always does.

For more on this topic, see these related posts by others:

• The World Has Changed – Is Hardware Getting Softer?
• Commodity hardware always loses
• Storage is software
• Better storage through hardware

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Commodity Hardware Always Wins
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Four Horsemen of Storage System Performance: These four ugly gentlemen stand between you and your data.

Why do some data storage solutions perform better than others? What tradeoffs are made for economy and how do they affect the system as a whole? These questions can be puzzling, but there are core truths that are difficult to avoid. Mechanical disk drives can only move a certain amount of data. RAM caching can improve performance, but only until it runs out. I/O channels can be overwhelmed with data. And above all, a system must be smart to maximize the potential of these components. These are the four horsemen of storage system performance, and they cannot be denied.

The Chain of Command

It is tempting to think of storage as a game of hard disk drives, and consider only The Rule of Spindles. But RAM cache can compensate for the mechanical limitations of hard disk drives, and Moore’s Law continues to allow for ever-greater RAM-based storage, including cache, DRAM, and flash. But storage does not exist in a vacuum. All that data must go somewhere, and this is the job of the I/O channel.

To be useful, storage capacity must connect to some sort of endpoint. This could be the CPU in a personal computer or an embedded processor in an industrial device. Indeed, there are endpoints and I/O channels throughout modern systems, with potential bottlenecks, caches, and smarts at each point. “Storage people” like me tend to think too small – imagining that the I/O channel ends at the disk drive, the “front end” of the array, or the storage network. But data must travel further, all the way to its final useful point in the core of the CPU.

Once we consider I/O as a long chain of interconnected endpoints, we begin to see the fact that I/O constraints at any point can strangle overall system performance. This is not merely an academic exercise: Optimizing the I/O channel is a consuming passion for most practitioners of enterprise IT, including architects, engineers, and system developers. And, like a good game of Whack-a-Mole, increasing the speed of one link causes another chokepoint to rear its head.

Parallel and Serial I/O

Imagine you had a warehouse full of boxes to move across the country as fast as possible. There are a few options available to you:

1. A fast truck can zip back and forth with just a few boxes
2. A train is slower, but its many cars can haul a huge quantity

But there are realistic limits to both capacity and speed: The train has to fit on the tracks, and the truck can’t move at the speed of light. Plus, one must consider the time taken to load and unload the chosen vehicle.

We continually shift between parallel and serial I/O paradigms

The same trade-offs are true of computer busses: Serial channels can be optimized to zip individual bits back and forth, or parallel busses can be designed to carry whole bytes (or more) at a time. The simplicity of serial communications is tempting, but designers continue to resort to parallelization for added throughput.

Note: Most serial protocols actually feature two links, making them “full duplex”: One for transmit and another for receive.

Serial storage interconnects are dominant currently, with fraternal twins SAS and SATA coming to dominate the disk interface landscape. SAS and SATA share the same 1.5, 3, and now 6 gigabit per second serial physical interconnect, offering more than enough throughput for conventional hard disk drives and edging out older serial (Fibre Channel, SSA) and parallel (ATA and SCSI) alternatives.

Networks (Ethernet, Fibre Channel, and InfiniBand) are predominately serial as well, as are lower-end interconnects like USB and FireWire. Serial communication also dominates in the system bus world, with serial PCI Express toppling parallel PCI.

But parallel variants are often offered for increased throughput: Multi-lane PCI Express and bonded multi-link InfiniBand make up a fair portion of the installed base, while load balancing MPIO drivers are common in Fibre Channel storage. And let’s not forget that the “X4″ variants of Ethernet use multiple bonded links as well.

The Definition of Bottle Neck

Most English speakers have encountered the French term, “cul de sac”, meaning “bottom of the bag” or dead end. But hard disk drives have plenty of “bottom end”, or storage capacity. When it comes to disks, the issue is usually at the neck of the bag: Data just can’t be pulled out of a hard disk drive fast enough.

Emptying a barrel of wine through a spigot takes hours, but pry the end off and the floor is covered in a moment!

The density of modern hard disk drives (the capacity of our barrel) has been growing much more rapidly than the I/O channels serving them (the spigot). Where once a hard disk drive could be filled or emptied in an hour or two, modern drives take days or weeks!

I once called this “flush time“, but I think the wine metaphor is much more appetizing!

This “bottle neck” has serious implications beyond basic storage performance. Data protection is impacted, since ever-larger storage systems can no longer be backed up by dumping their content; system reliability is reduced, since week-long RAID rebuilds increase the risk of multiple drive failures; and cost containment efforts are also impacted, since adding spindles drives up prices.

Nowhere is this bottleneck more evident than in portable devices. Modern drives (like the 1 TB Seagate USB drive I recently reviewed) have massive capacity and pathetic performance. The USB 2.0 interface just can’t keep up, and this creates a limit to the expansion of capacity. It would take half a day to fill that drive under perfect conditions at 25 MB/s, reducing its value as a massive data movement peripheral. The emerging USB 3.0 standard promises to alleviate this performance issue for now, as illustrated with Iomega’s new external SSD.

Cache and solid state storage can help, but they have their own bottlenecks. Storage arrays typically use Fibre Channel or SAS SSDs, and their front-end interface remains the same. The best-performing SSDs use the PCI Express bus directly rather than emulating hard disk drives over SCSI interfaces. And even PCI Express might not be enough to handle the massive I/O of NAND flash or DRAM. In each case, the bottleneck moves down the chain.

A Chain of Bottlenecks

Let’s follow a typical I/O operation from the disk to the CPU core and count the I/O channels:

1. A read head senses the state of a bit of magnetic material on the surface of a disk
2. The head transmits this signal to a buffer on the disk controller board
3. The data is picked up by the disk controller CPU and transmitted over a SATA or SAS connection
4. The storage array or RAID controller receives the data and moves it over an internal bus to another buffer or cache
5. The data is picked up by another CPU in the array controller and sent out another interface using Fibre Channel or Ethernet
6. The data is buffered and retransmitted by one or more switches in the storage network
7. The host bus adapter (HBA) on the server side receives the data and buffers it again before sending it over a local PCI Express bus to system memory
8. The server memory controller pulls the data out of system memory and sends it via a local bus to the CPU core

There are actually many more steps than this, but the picture should be clear by now. There are many, many I/O channels to consider when it comes to storage, and the drive interface is just one potential bottleneck.

We constantly move bottlenecks around - as one link is improved, another choke-point appears

Optimizing Storage I/O

Tactical steps to improve storage performance typically focus at one link in the chain: Drive vendors move from 1.5 Gb to 3 Gb SATA, or SAN buyers upgrade from 4 Gb to 8 Gb Fibre Channel. But the basic architecture of enterprise storage has remained constant for over a decade, and the reliance on block SCSI commands endures. This is all about to change.

One critical bit of I/O optimization exists at the point of connection between the various chipsets inside the server. AMD pulled the memory controller off of the “northbridge” with their Athlon line. Intel did the same with their Nehalem and is eliminating the northbridge entirely with the Lynnfield/Jasper Forest CPU lines. This gives serious bandwidth to the crucial PCI Express-to-CPU-core link, moving the bottleneck downstream.

We are in the midst of a massive upgrade of the storage network as well. Between 8 Gb Fibre Channel and iSCSI and Fibre Channel over 10 Gb Ethernet, not to mention persistent interest in InfiniBand, storage network throughput is rapidly expanding. As with the internal PC connections, the expansion of network bandwidth has pushed the bottleneck to the storage array interface for the time being.

Microsoft and Intel recently pushed over a gigabyte per second over 10 GbE using iSCSI, but they needed multiple storage targets to feed that connection. It isn’t that modern storage systems couldn’t push that kind of I/O (indeed, arrays are tens to hundreds of times faster internally thanks to their spindles and cache), but that the conventional storage protocols are tightly linked to a single “front-end” interface. The current state of the art for storage array design is moving to distributed models, exemplified by pNFS and scale-out NAS concepts like MaxiScale (now acquired by Overland).

Once the array interfaces can pump out massive I/O, attention will turn once again to the disk interfaces themselves. Although 6 Gb/s SAS and SATA is now a reality, this interface is inappropriate for future high-performance SSDs. Arrays designed around flash or DRAM are likely to switch to PCI Express as their internal connection of choice for performance and to optimize data placement on these new devices. Companies like Nimbus and NetApp are already moving in this direction.

Time To Get Smart

Hard disk drive spindles make up the bulk of storage capacity, but small amounts of cache make them far more effective. But both of these horsemen must operate within the constraints of the I/O channels they pass through. This brings us to the final horseman of performance: Smarts. Clever designers have created clever controlling mechanisms to overcome the limits of spindles, cache, and I/O channels.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | The Four Horsemen of Storage System Performance: I/O As a Chain of Bottlenecks
This post was categorized as Computer History, Enterprise storage, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.