It’s that time again, when everyone who thinks they’re a pundit (that would be everyone with a blog or Twitter account) has to make predictions for the coming year. But predictions are perilous: Get it right and you look like a mere trend-watcher; get it wrong and you look like a fool. It’s such a hassle! So I’m doing something different this year: I’m going to make predictions for 2009 now that it’s over, and reflect on just how smart I am (not) to have made them. Or something.

What I Would Have Gotten Right

I definitely could have predicted a lot of what happened in 2009. I mean, these were slam dunks!

1. Twitter rocks the world – I wasn’t early to Twitter, but I spent the early part of 2009 evangelizing its benefits to companies and co-workers alike. Considering how common Twitter is today, it’s hard to believe how roundly criticized and misunderstood it was this time last year. Yet here we are, on the verge of 2010, and Twitter has seeped onto our business cards, presentation templates, and web sites. I might not have predicted how stable (!) Twitter got by the end of the year, though.
2. Apple’s Macs and iPhones rule – I switched to the iPhone and the Mac in 2007 and 2008, respectively, but it looks like I wasn’t much of an iconoclast after all: By November, half of the Tech Field Day delegates were using MacBooks, and the Windows and Blackberry holdouts have started vocally defending their operating system choice. Pretty much like Mac folks used to do way back in 2008.
3. The recession is a serious pain – Companies put the brakes on spending and hiring, many even shifting both into reverse in 2009. This came as no surprise to humans capable of thought. The impact on enterprise IT companies was similarly predictable: Although most were able to survive, the impact of 2009 will continue to be felt for years. I might have predicted it would be worse, though I’m glad to say I would have been wrong.
4. EMC, NetApp, HDS, HP, and IBM continue to quibble – Surprise: Big company bloggers spend way too much time criticizing the products and actions of each other and way to little time talking about the true value of their own products.

Non-IT slam-dunk predictions: Obama was reviled by the right; the war in Afghanistan continues; people do stupid stuff in the name of reality shows.

What I Probably Could Have Predicted

Although some details would likely have been missed, I think I would have seen these coming.

1. Cloud compute and storage hits the enterprise – I was a believer in the cloud this time last year, and I bet my future on it by taking a position at enterprise cloud storage provider, Nirvanix, in March. I would have predicted that enterprise buyers would be putting serious thought to buying cloud products, but the scope has surprised me. We’re talking enough petabytes that the non-cloud players felt compelled to strike back with the private cloud pitch. Awesome!
2. Sun and Data Domain were acquired – My money would have been on Dell, IBM, or HP as buyers for this pair, but EMC wouldn’t have been outside my guesses. Still, Oracle buying Sun and vocally committing to keep it going, SPARC and all, would never have come to mind. But I wouldn’t have guessed against it either, so I’ll give myself a point here!
3. Cisco and EMC buddy up – I’ve long thought an outright merger of these two was in the cards, but even the recession couldn’t make the financials work. A partnership would have been on the list, and Acadia came as no surprise to anyone.
4. Cloud outages and data loss – I definitely could have predicted that high-profile cloud services would fall over throughout the year, and that some would lose data. Not all are enterprise-grade, after all. But the outages at Google, Rackspace, and Amazon, and Microsoft’s Danger data loss, surprised me. Don’t those guys have their acts together?
5. IT conferences falter – I spoke at Interop in 2009, but it lacked the 20,000-strong crowd it once had. Storage Decisions and Storage Networking World managed to fill their halls, but the old-school IT conference has lost its luster. Although VMworld remains strong, attendance was definitely off.
6. FCoE and SSD are still starting – I’ve been lukewarm on Fibre Channel over Ethernet and Solid State Drives, but I’m a bit surprised that storage vendors didn’t push them harder in 2009. I might have guessed there would have been more customer uptake to match the buzz.
7. SMB storage is hot – There’s a hole in the storage market between $1,000 and $20,000, and companies like Drobo and Iomega are rushing in to fill it. Now that ESX has solid iSCSI support, I expect a world of innovation here. (Oops, that sounds kind of like a 2010 prediction!)

Also in the predictable category: Goldman Sachs and Bank of America thrived while others fell; Ford is the strongest of the remaining US automakers; Boeing finally got the 787 off the ground.

What I Never Would Have Guessed

I’m not perfect, even in retrospect. Some of the Tech news from 2009 was just completely off the wall.

1. Microsoft Bing: This time for sure! – Seriously, Microsoft should stick to in-house thinking instead of trying to copy its rivals. Yet somehow, miraculously, Bing appeared and did not suck. In fact, I’m hearing regular (non-techie) folks around town talking about using the search engine. I’ve even used it! Could they actually have a winner?
2. Windows 7 rocks – Really? Seriously? Could Microsoft have come up with a solid replacement for Windows XP?
3. Ship it! – It’s not even 2010, and enterprise storage buyers can go out and purchase NetApp’s OnTap 8, EMC’s FAST, EMC Atmos Compute, and unicorn tears. Well, maybe not unicorn tears.
4. Still no GDrive – Seemingly every company has a cloud storage platform, from Amazon to Rackspace, Nirvanix to EMC, so why not Google? Could GDrive join Duke Nukem Forever as the most famous vaporware of the decade?
5. The executive shuffle – Dave Donatelli was supposed to lead EMC, not HP. Alan Atkinson was supposed to launch another startup, not take over Xiotech. At least NetApp was gentle.
6. Mac OS X (still) lacks iSCSI and ZFS – Come on, Cupertino, what’s wrong with you guys? I’ve been hyping ZFS for years, and iSCSI is commonplace. Yet Snow Leopard is stingy with both. Makes me want to hiss like one of those blue folks in Avatar.
7. Gestalt IT is a success – On a personal note, Gestalt IT didn’t even exist this time last year, and now we have a successful IT infrastructure blog and social media event. Amazing!

Other total shockers: Everyone loves Michael Jackson again; digital Beatles tunes are available everywhere but iTunes; Obama’s Nobel Peace Prize arrives 10 years early.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | My 2009 IT Industry Predictions
This post was categorized as Apple, Computer History, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although deduplication of storage is nothing new, with Data Domain and other making hay with the technique for years, it has never been ready for prime time – reduction of active primary storage applications like email and databases. Instead, deduplication has been relegated to second- or third-tier status, deduplicating archives and backup data. But change is in the air, and deduplication vendors are starting to bustle towards the bright lights of primary storage.

Stone Knives and Bear Skins

We have all been here before, of course. Back at the dawn of the personal computer era, data compression was a hot topic of conversation. I recall being so impressed by an article in Byte (1986:5, p99) outlining Huffman coding that I tried cooking up an implementation in Atari BASIC. Lossless compression has a magical pull to the geek in many of us – redundant data just wants to be eliminated!

Stacker dominated the disk compression world - until Microsoft introduced DOS 6.0

Companies soon applied compression to primary storage, especially the limited storage in personal computers. Stacker was a hit after 1990, until Microsoft built a workalike, called DoubleSpace, into DOS 6.0 in 1993, leading to a historical lawsuit. I personally used the ADDSTOR disk compression built into DR-DOS 6.0 to stretch two more years out of the 20 MB MFM hard drive in my AT&T PC6300 at WPI.

But something funny happened in the late 1990s: Compression began to lose its luster. Compressing data always takes quite a bit of CPU power, but this was offset somewhat by the truncated data transfers and more-efficient file system layout afforded in early PCs. But as disks got larger and faster, using precious CPU time to save space seemed less and less compelling. Today, although nearly every operating system includes built-in compression of files, folders, or perhaps disks, these features are rarely used. And compression was never popular in the performance-sensitive enterprise space.

Deduplication Has a Nice Ring

Although traditional fine-grained compression has not been very successful in the enterprise, its lanky cousin, single-instance storage, has long found niche jobs. Applications from databases to email systems to file servers have long had the ability to recognize to requests to store the exact same file or record, and to store just a single instance in this case. Even file systems have the ability to do single instance storage through the use of links, though this is initiated by the user rather than in an automated fashion.

In the late 1990s, FilePool began developing a content-addressable storage device, which was acquired by EMC in 2001. This device, later known as the Centera, was one of a number of storage platforms targeted at the archiving market introduced this decade. At the same time, virtual tape libraries made the jump from the mainframe to open systems. Both devices, being outside the critical path of performance but offering massive capacity, were well-suited to implement advanced capacity optimization technologies that combined the concepts of compression with single-instance storage. Thus was created the modern world of data deduplication.

What we think of as deduplication is neither fish nor fowl: It assesses larger “chunks” of data than compression technologies, delivering greater capacity savings and potentially reducing performance impact, but is more flexible than single-instancing, recognizing the similarities within files or objects.

But it is still maddeningly difficult to scale deduplication while maintaining performance. Rather than fight to maintain reasonable write throughput, most deduplication products have switched to post-processing, deferring their work to quieter times.

It’s Not Just for Breakfast

Regardless of their methods or underlying technology, no deduplication vendor has stood up to support challenging low-latency or high-throughput production applications, however. NetApp was the first to raise the issue of support for production applications, but although they tout the technology for VMware, they haven’t exactly been shouting from the rooftops to get their A-SIS deduplication technology deployed in other high-I/O applications. And I haven’t seen Hifn’s card yet.

Yesterday, I mentioned that greenBytes was adding deduplication to their ZFS-based storage array for primary data. And now Riverbed has fired another shot over the bow, repurposing their (deduplicating) WAN accelerator product for primary (file) storage. They might be able to pull it off, too, since they have a long list of customers who are already enjoying the technology in production. It’s not a stretch to suggest that Riverbed’s appliances can scale to handle production data loads. Although it’s file-only, I can imagine quite a few scenarios where this tech could really yield benefits. Could we come full-circle, with deduplication finally reaching the enterprise storage world?

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Deduplication Coming to Primary Storage
This post was categorized as Computer History, Enterprise storage, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Spun Down

Although ZFS’ universal storage pool with non-RAID is a great concept, it stands in the way of at least one (sometimes) desirable storage technique: disk spin-down. Put simply, since every disk contains metadata, all disks must always be spinning. This issue is by no means a ZFS-only problem, though – certain vendors tout the (laughable) greenness of their storage systems, while hoping that the average user won’t notice the truth: That a disk simply cannot spin down while any part of it is in use. This means that tacking spin-down onto a regular storage array is like painting it a different color: There is no benefit whatsoever to the average user. Sure, a few non-provisioned drives might spin down, but what are you doing buying a lot of non-provisioned drives anyway?

The solution has always been right in front of everyone: Develop a new type of non-RAID with enough intelligence to allow drives to spin down when not used. This is what COPAN Systems did with their MAID technology: Invent an entirely new storage array, with integrated data protection and management techniques that allow alive but not active drives to spin down. Spin-down is not MAID any more than a bicycle is a Ducati.

Let’s make one thing clear: It’s really hard to reduce the power demands of storage devices. Disks guzzle watts like few other data center devices, and enterprise storage uses lots of disks. Lots of vendors are looking to hop onto the green storage bandwagon, and they all seem to realize that bringing some intelligence to power management by enabling spin-down is an open door. But it’s awfully hard to maintain performance and data protection when disks are spinning up and down all the time.

One element of the greenByte story is the way in which they have tweaked ZFS to allow disks to spin down. They limit the metadata updates to just a few disks, so the others can be idled when no access to them is made. The company suggests scheduling this for off hours to minimize latency as drives are brought back online, an approach that is less than optimal from an energy perspective but demonstrates that they understand just how difficult this problem is to crack. The core is there, however: They have integrated the data protection and storage management elements to enable spin-down to be practical.

Compressed

Another major storage industry theme of the last few years is deduplication of data. An advanced (or devolved, depending on your perspective) form of compression, deduplication allows a storage array to store duplicate data more efficiently, reducing the amount of capacity required for some applications. Data Domain is top-of-mind in this space, but just about everyone now offers some form of deduplication technology.

One major roadblock on the way to deduplication (or compression) nirvana is performance. Simply put, it’s really really hard to process data on the fly without affecting performance, especially as data scales up to the multi-terabyte range or as systems scale out to include multiple devices. One approach to tackling this issue is post-processing dedupe, which accepts incoming data in the normal way but goes back and processes it later to remove duplicates. This is the method NetApp uses, and they have leveraged it to become the first vendor to support deduplication of production applications.

Predictably, deduplication is another technology integrated into greenBytes’ “ZFS+” technology. They claim that they can handle inline compression at wire speed, and also claim deduplication inline. It’s not yet clear exactly what the difference between compression and deduplication is to the company, or just what kind of performance their inline technology will yield, but it’s certainly nice to see this tech integrated with ZFS!

Thin is In (the House!)

greenBytes gets closer to enterprise storage bingo by adding thin provisioning to the mix. Actually, as the company’s CTO was quick to point out, they had to offer virtual or thin provisioning to enable the rest of the system to function. When your storage is sliced and diced by their Cypress array, the only way to present storage is with a wink and a promise of capacity to spare. Thankfully this is not the core of their pitch, however.

The company also promises snapshots and CDP replication, all leveraging ZFS at the core. All they need to add is tier-0 solid state storage to get five chips in a row without even using the free space! Although greenBytes is using Sun’s Thumper chassis currently for their Cypress array, their core technology is the ZFS+ software, and I expect we might see this mixed quite differently in the future. This is a software company, not an array vendor.

All considered, greenBytes has thoroughly broken the link between physical and logical storage, and I applaud them for it. This is exactly the kind of storage revolution the industry needs right now.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | greenBytes Embraces and Extends ZFS
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.