I Promise Not To Quote That Old Annoying Dell PC Slogan

I’ve been buying Dell computers for decades, but not really because I loved them. Sure, my XPS laptop was awesome, but it burned out its motherboard and I never really touched the RMA replacement, having bought a MacBook Pro in the meantime. Enterprise buyers seem to have the same ambivalence about Dell. They buy it, but I’m not sure they really “buy” the company as an IT partner.

I’m sure you’ve heard the same comments as me: “Dell just assembles off-the-shelf components and sells them in volume” or “Dell’s a follower, not an innovator.” There seems to be a great deal of respect for Dell’s ability to produce competitive products and sell them at reasonable cost. Truly, most of their competitors would love to have this kind of reputation. But most of their competitors also have a reputation for partnership, innovation, and solution selling.

Dell Is Making An Effort

It seems clear that Dell would like to change this attitude, and they are investing serious resources to make it happen. While acquisitions like Compellent and Force10 raised eyebrows in storage and networking, it is the activity I see behind the scenes that paints the clearest picture. Dell isn’t just buying into new markets, they’re investing to change the company.

When Dell acquired EqualLogic in 2008, many assumed it was a tactical investment to increase margins over the (resold) EMC storage equipment the company was then pushing. Pundits were similarly dismissive of the acquisition of Perot Systems in 2009, calling it a “me too” effort after HP acquired rival EDS. Regardless of the motivations, however, Dell was becoming more of a serious challenger to HP and IBM every day.

After failing to acquire 3PAR in 2010, then picking up Compllent shortly after, accusations that Dell was “mini me” to HP were rampant. But HP stumbled mightily in 2011, and many in IT quickly lost confidence in that company’s management. All the while, Dell moved forward, increasing in-house IP and expanding enterprise offerings.

What Is The Result?

Today, one sees a very different landscape than just last year. Dell’s acquisitions focused on some of the ripest spots in storage and networking, and no one would disagree that the company has the ability strongly to push these products. Compellent and Force10 went from interesting startups to serious contenders overnight.

Dell really pulled out all the stops to tell us they are "Fluid by Design"

More importantly, Dell has retained much of the innovation these companies offered, from employees to support programs. Last week, I attended the Dell Storage Forum in London, an event initiated by Compellent prior to the acquisition. At the event, I talked to many Dell employees who came to the company through acquisition but had now been given power to challenge the status quo in their respective areas.

If Dell really intended only to push product, why retain marketing personnel? Why invest in the Dell Storage Forum? Why continue Compellent’s beloved Co-Pilot support program?

Then there are the products. Dell leveraged its investment in Ocarina Networks to create a deduplicating backup appliance, the new DR4000. They salvaged file system startup ExaNet and are beginning to bring scale out technology to market. The latest revision of the Compellent software finally brings it to parity in terms of VMware support. And Dell is really working to sell their DX Object Store.

This is the sort of activity one would expect from a contender, not a “box pusher”.

Stephen’s Stance

In the words of Malcolm Reynolds, my days of not taking Dell seriously are certainly coming to a middle. Dell is investing in product IP, innovative marketing and PR events, customer support, and personnel. This does not mean that Dell is instantly a player in the enterprise storage and networking markets, or that all this work will pay off. But I don’t laugh when I hear Dell boast that they intend to be a “top three” enterprise storage company in a few years. It could happen.

Disclaimer: Dell sponsored two Tech Field Day events in 2011, paid me as a speaker at two DX events, and paid for my trip to Dell Storage Forum in London. But no one can buy a post on this site, and I did similar business with IBM, HP, Cisco, and many other companies. This is my opinion.

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Why Should Anyone Take Dell Seriously in Enterprise Storage?
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The players are lining up for the biggest acquisition game enterprise IT has witnessed in a while

Today is the (a?) day of reckoning in the 3Par saga, with Dell widely expected to make a counter-offer higher than HP’s bid. But this mega deal, like the Data Domain war before it, sends a strong signal to the enterprise IT world: It’s open season on data storage companies! But the rising superpowers are also likely looking at networking as an area of expansion. The game is afoot!

The Competitors

The simple fact is, enterprise IT prefers to buy from large, established names like HP and IBM rather than smaller and less-familiar companies with names like Ocarina, EqualLogic, or even 3Par. The acceleration of sales by enterprise providers is what makes these big acquisitions so successful and why others involving less-powerful players often fail to deliver the same results.

Full-line “superpowers” like IBM, HP and now Dell and the new Oracle can influence purchasing decisions across a broad spectrum of hardware and software. Many large organizations are tightly coupled to one of these vendors, and will give their products stronger consideration even when they are new or unfamiliar. HP is already flexing their muscle selling their broadened network lineup, and Dell found that it could do this in data storage.

Software vendors like Microsoft, Citrix, and Oracle adamantly maintained a neutral stance toward hardware, but Ellison and company seem serious about changing this. Oracle’s success in selling Sun hardware will likely dictate further software acquisitions for Dell and HP, though IBM already has strength there. Then there is Intel, the wild card who just got wilder with their acquisition of McAfee.

Companies with narrower focus like Cisco, EMC, NetApp, Juniper, and Brocade have the same power within their sphere of influence but cannot pull in wholly-distinct products. Cisco is in the midst of this fight with their technically-excellent UCS blade servers: Although they were certainly a strategic CIO-level vendor in the largest organizations, “blades from a networking company” isn’t the as easy a decision as “networking gear from a full-line company.”

The Game

Then one must consider the market in contention. The enterprise IT space is not expanding, especially in the United States. This is very close to a zero-sum game, with Oracle’s or Dell’s wins being HP’s or IBM’s losses and vice-versa. There is massive money in play, and flexibility enough for it to swing between the competitors, but growth is not unlimited.

Enterprise storage and data center and campus networking are two areas where smaller companies retain enviably-large slices of the pie, explaining the interest in these spaces among the superpowers. These verticals still have room for sales to grow without displacing a fierce full-line foe, and the superpowers have lately been weak here. Storage and networking are enticing opportunities, but each slice is similarly dominated by “vertical superpowers” and partners EMC and Cisco.

So this is the game: Four full-line enterprise superpowers battling each other for datacenter dominance and coveting the extra profits of a few verticals. HP clearly believes they can chip away at EMC and Cisco in storage and networking; Dell and IBM have so far focused mainly on storage; and Oracle hasn’t made a move in either direction, instead challenging the other three in the core server and software space.

Pawns or Knights?

So where does this leave the smaller players? Are they merely pawns in the game, waiting to be sacrificed, or are they knights who can wield power across the field? The largest (Cisco and EMC especially) appear to have ambitions of their own as well as the financial and technical strength to shake up the game. They are unlikely to be acquired by the superpowers. Brocade, too, has broad strength in storage and networking but maintains relationships across the board that make an acquisition difficult.

Strong vertical players like Juniper, NetApp, Riverbed, and Compellent are ripe for acquisition, as were Foundry, 3Com, Data Domain, and 3Par. IBM, Dell, and Oracle are all likely buyers of the networking players, though HP may consider filling in where 3Com was found lacking. All four will likely take a strong look at the remaining storage players as well, with the loser in the battle for 3Par likely to be hungry indeed.

One should also consider the potential impact of smaller acquisitions. Although they would not immediately “move the needle” for a massive superpower, there are many excellent technology companies that could be bought low and pushed strongly. The enterprise-class technology at Sepaton, Pillar Data, Xiotech, BlueArc, Extreme Networks, Force10, Blue Coat, Isilon, CommVault, FalconStor, and many others should not be overlooked. If a superpower can drive a larger acquisition to succeed, imagine what they can do with strong but inexpensive technology from one of these!

Stephen’s Stance

This game is nowhere near finished. The 3Par acquisition will not only generate revenue, it will shake up the ranking of data storage array dominance. This is very likely to kick off additional acquisition in the data storage space, spurring either Dell or HP to pick up additional technology and perhaps causing IBM or Oracle to engage as well. With no easy alternatives to 3Par, I expect Compellent, Xiotech, and Pillar to get closer looks, but Sepaton and BlueArc are just as ripe. NetApp may be too expensive at this point, but would be a nice match for Oracle’s strategy.

HP’s acquisition of 3Com could also signal a race to integrate datacenter and campus networking technology into the stack. Many are suggesting a Brocade acquisition, and it would be much cheaper than Juniper, but OEM ties make it a difficult purchase for any of the superpowers. Extreme and Force10 would be excellent and less-expensive alternatives.

We should also keep our eyes outside the superpower space. Intel showed that they can make big moves, and Microsoft might consider a diversification into hardware as well. One should also look to the East, where Huawei could try to enter the Western market with a merger or joint-venture to cast off the China stigma. Although I would love to see a rebel alliance rise (imagine Juniper, NetApp, and Symantec joining forces!) this is not a likely scenario.

Image credit: Chess Board by mosdave

© sfoskett for Stephen Foskett, Pack Rat, 2010. | The Enterprise IT Acquisition Game
This post was categorized as Computer History, Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Who will power the enterprise? The smart money is betting on a few superpowers taking over.

After years spent focusing on personal technology, businesses are increasingly turning back to the enterprise. There are many reasons for this, but the biggest one is the poor economy. Individuals simply have less free cash to spend on gadgets and software, and the meagre profits are increasingly going into the pockets of a single company: Apple.

The corporate IT market is much more dynamic and competitive, with a few very large “superpower” companies discovering their power to drive purchasing decisions. If a supplier can create an integrated “stack” of hardware and software, they can push product purchases that might otherwise be overlooked or postponed. This is the main reason that enterprise IT acquisitions work so well: Where a small company must fight to sell their product, a large one can hitch it to a much more strategic sale and have it pulled along.

The old IBM model is the prototype, with that company once selling everything from office equipment to datacenter gear as well as the consulting and integration services to make it all work.

HP has spent almost two decades bulking up to become the new IBM, buying their way into open systems laptops, desktops, and servers (Compaq), networking (3Com), services (EDS), and storage (Compaq, LeftHand, Ibrix, and perhaps 3Par). HP has been remarkably proficient at executing on this enterprise plan: In talking to enterprise IT folks, I often hear IBM-esque sentiments regarding the new HP. They tell me they’re willing to give HP the benefit of the doubt when it comes to new technologies and products, buying on basis of the company’s reputation and ability to make everything work. This bodes well for the company’s post-Hurd future, and HP has the most-complete “enterprise stack” in the business.

But HP has a target on its back, pinned there by Dell. The folks from Round Rock believe they can be more efficient (and thus profitable) than HP in the same markets, and have been making moves to fortify their enterprise offerings. Dell was always more of a manufacturing than R&D business, but they have shown a desire to broaden their focus. Intrigued by the high-margin mid-enterprise storage business built from their EqualLogic acquisition and their success selling EMC storage, Dell is moving into the enterprise. They matched HP/EDS by purchasing Perot and have made smaller buys in storage (Ocarina, Exanet) as well as the big move for 3Par.

The next big emerging stack player is Oracle. The acquisition of Sun gave Oracle a strong hardware base to complement their command of enterprise software, and many expect further acquisitions. But Oracle is playing a different game than HP and Dell, focusing on the high-margin enterprise space and ignoring more competitive outlying areas. Many suspect the company might make a play in the network space (Brocade, Juniper, and F5 have been mentioned) but storage is possible as well. CEO Larry Ellison is a major investor in Pillar Data Systems, so many expect a spin-in here. But Oracle has the appetite for something much bigger, even EMC or NetApp.

Then there is Cisco, who have attempted to parlay their data center networking strength into a broader position. But Cisco’s halting moves into storage (Fibre Channel switching and SAN extension) did not displace the market leaders, and their server products (UCS) have not made much of a dent on HP, IBM, and Dell either. A solid partnership with EMC has delayed further forays into the enterprise storage market, and Cisco seems puzzlingly interested in low-margin access businesses (Linksys, Flip) and their Cius tablet.

There are other players in the enterprise space as well. EMC has diversified under CEO Joe Tucci, taking a dominant position in server virtualization (VMware) and making a strong enterprise security acquisition (RSA). But the many faces of enterprise storage remains EMC’s strength, and they seem content to partner with Cisco for a stack sale. Hitachi, NEC, and Fujitsu also offer varying enterprise hardware and software stacks, but their comparatively small sales presence in the US market limits their ability to execute. In the final analysis, only IBM, HP, Dell, and perhaps Oracle can claim to be enterprise IT superpowers at this point.

Image credit: Steam Engine by Stuck in Customs

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Meet the Enterprise IT Superpowers
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I’ve long wished for a new enterprise storage superpower. Competition is good for everyone, and the enterprise storage space has always been highly competitive. Traditional SAN storage powers (EMC, HDS, HP, and IBM) have been under continual attack from tech-heavy upstarts like EqualLogic, LeftHand, Compellent, Xiotech, and 3Par. The smaller (revenue-wise) NAS market has been more serial, with NetApp knocking off Auspex, then challenged by EMC. Yet innovators have been thick there as well, from Exanet to Ibrix, Isilon to Onstor.

Through it all, one thing has been clear: The major companies, though perhaps lagging in technology, were usually able to withstand the attack of the upstarts through sheer strength of salesforce. Storage is a strategic investment, and selection of a storage platform is much more far-reaching than many IT product decisions. The inertia of an installed storage environment makes it a real challenge to switch vendors, giving the established players massive leverage.

It became clear to me and many others that the best way for upstart companies (and, by extension, technologies) was to be part of an established vendor’s sales process. OEM relationships were a big part of this (witness the success of BlueArc and even NetApp and HDS) but acquisition was a much stronger proposition. If customers were warmer to OEM products than independent sales, they are much hotter when it comes to acquired technology. HP, Dell, IBM, and EMC have all demonstrated the power that comes when an established company buys a startup and puts the power of their sales force behind these new products.

This explains Dell’s fantastic success with EqualLogic. They took a product that was emerging as dominant in its niche (midrange iSCSI SAN) and blasted it into the market, while at the same time optimizing manufacturing and deployment. EMC did the same with Clariion and DataDomain, and HP is showing strong signs of health with LeftHand and Ibrix. Then there is IBM, who took XIV out of Israel and made it a source of irritation to the rest of the industry.

Many industry watchers have long wondered what would happen if the smaller guys got together, forming a new superpower of their own. Would 3Par, BlueArc, and Sepaton be a real challenger? What about Xiotech or Compellent and Isilon or FalconStor? Is mixing and matching some smaller companies a recipe for success? The answer was often a counter-question: What if someone like Dell, who knows how to manufacture and sell, picked them up instead? This seemed much more like a sure-thing, since the established management and financials stave off potential integration issues.

It appears that this is the future. Established players will pick up smaller companies, fortifying their offerings and accelerating sales in a way the little guys weren’t capable of. Dell’s billion-dollar acquisition of 3Par reportedly headed off a similar offer from HP, and will likely spark another acquisition. I imagine the management teams at Compellent and Xiotech just got a lot busier…

Clearly, Dell and HP are playing this game. IBM and EMC are in it, too. But what about Cisco and Oracle? Could they be planning storage acquisitions of their own, to the detriment of partners like EMC and Hitachi? What about the strong contingent from Japan, NEC and Hitachi? And who gets picked up next? We shall see!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Dell + EqualLogic, Exanet, Ocarina, 3Par = What?
This post was categorized as Computer History, Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Enterprise Storage

• HDS’ HAM-Fisted Announcement did not impress, with many wondering (So Long And Thanks For All The Fish) what exactly the company had announced. But by week’s end (USP-V and Hitachi High Availability Manager) we had figured out How It Works and even had some time for jokes.
• What should be in the cloud? How about some profiling questions nobody’s asking re: cloud applications
• Want your own iSCSI array? StarWind has responded to your comments, 2TB at no cost!
• What Is a Backup? Some great discussion (Backup to the Future) and a solution: When Is A Copy A Backup?
• Howard Marks is also doing a great job with backup and archiving: See Of Backups and Archives, Storing Archival Data – Part Deux, and All Archive Data is Not Alike
• I joined Chris Evans, Marc Farley, and Greg Knierieman for Infosmack Episode #6 – HDS USP-V announcement, EMC non-competes and changes at VMworld
• ‘Nuff said: And people wonder why I hate HP, Google Could Learn a Thing or Two from EMC, Virtualization: Why Microsoft cares

Misc

• Tim O’Reilly took a look at a technology I just didn’t get, but changed my mind with Google Wave: What Might Email Look Like If It Were Invented Today?
• Want a crazy car to get your mind off of GM and Chrysler? How about the 500-hp Ariel Atom 500 V8
• A wonderfully long and detailed article: Lessons from the Vaccine–Autism Wars
• Finally, take a look at this amazing video: Louis CK “Everything’s amazing, nobody’s happy”

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Back From The Pile: May 30, 2009
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This got me thinking: The real distinction is between buyers and sellers of products and services. There is a spectrum on the sell side between being an independent and a company man, but we are all vendors. Our credibility comes from who we are, not where we work.

The Business of Consulting

Consulting is a simple and perilous business, and it is governed by a simple formula: Billing rate times billability is gross revenue, everything else is overhead. You keep what’s left over.

Only three elements matter to a consulting business, and all are variable

There are really only two reasons a customer hires a consultant:

1. They need specific skills or knowledge
2. They need focus or manpower

The consulting business rests on these simple elements, and we all try to make the best of them. But at the end of the day, a company that has to make its money selling services has to decide what it wants to be:

• Strategic and professional services (PS) organizations focus on maximizing rate by specializing in a special area of skill and take shorter-term engagements.
• Body shops or outsourcers focus on maximizing billability by providing low-rate manpower in long-term engagements.
• Subcontractors focus on reducing overhead by running lean and outsourcing services to non-employees (typically self-employed “1099″ consultants).

I’m using common jargon here, and most consulting businesses wouldn’t want to be classified into one of these buckets, but the business speaks for itself. For any given consulting shop, ask yourself one question: Which of the three factors in the equation is being maximized?

This simple math deeply affects the life of the consultant. Some decide to go it alone, happily risking a steady income for a higher rate as a 1099 subcontractor. Others choose to stick with the steady life of the outsourcer, accepting a lower pay but knowing where they’ll work each day. Personally, I always gravitated towards the strategic and PS roles because the focus on skills made me feel special.

Who Are You?

Let’s step back to that continuum of supply-siders for a moment. In my field of specialization, enterprise data storage, one can immediately identify some positions along the spectrum:

• PR people relentlessly and (sometimes) creatively push their clients. They are among the most vendor-focused folks out there because that’s their job!
• Marketing communications folks have to be creative to create a product-centric point of view, so you can expect solid vendor-angled messaging from them.
• Product marketing people tend to be much more technical and tend to truly believe in the virtues of their product, since they helped shape it.
• Subject-matter experts in the vendor and reseller community know the plusses and minuses of the products inside and out and love to get into deep technical discussions.
• Analysts present their opinions in a balanced way, but the topics they focus on are driven by the vendors they work with.
• Consultants vary in focus depending on the aims of their company, but tend to be more utilitarian, asking “what does this do for a customer?”

I have great affection for folks in every category on this list. Take Sunshine Mugrabi, for example, a PR pro who has taken that job to a whole new level of relevance with her work for Ocarina Networks. Or Marc Farley, who works somewhere in 3PAR marketing but still manages to crank out hilarious videos and thoughtful commentary. Or Chad Sakac, whose knowledge of VMware and storage makes him credible as much more than “an EMC guy”. And what about storage analyst extraordinaire, Steve Duplessie? This list could go on and on, so please accept my apologies, but I could not possibly include everyone I respect.

What’s the common denominator that makes someone credible? Simply that they rise above their positions to protect and project their personal reputations. Yes, they all work for vendors and they all deliver sales, but their work benefits the community well beyond that.

Credibility

The net is simply that we each build up or tear down our own credibility in life. Consultants, analysts, and pundits do not corner the market, and simply being in one of these fields does not make one especially independent or trustworthy. What matters is what we do with whatever soapbox we have. Do you trust me?

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Consulting Is A Perilous Business But Credibility Is What Matters
This post was categorized as Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Does data encryption throw storage efficiency out the window? Not always!

One of the great ironies of storage technology is the inverse relationship between efficiency and security: Adding performance or reducing storage requirements almost always results in reducing the confidentiality, integrity, or availability of a system.

Many of the advances in capacity utilization put into production over the last few years rely on deduplication of data. This key technology has moved from basic compression tools to take on challenges in the fields of replication and archiving, and is even moving into primary storage. At the same time, interconnectedness and the digital revolution has made security a greater challenge, with focus and attention turning to encryption and authentication to prevent identity theft or worse crimes. The only problem is, most encryption schemes are incompatible with compression or deduplication of data!

Incompatibility of Encryption and Compression

Consider a basic lossless compression algorithm: We take an input file consisting of binary data and replace all repeating patterns with a unique code. If a file contained the sequence, “101110″ eight hundred times in a row, we could replace the whole 4800-bit sequence with a much smaller sequence that says “repeat this eight hundred times”. In fact, this is exactly what I did (using English) in the previous sentence! This basic concept, called run-length encoding, illustrates how most modern compression technology functions.

Replace the sequence of identical bits with a larger block of data or an entire file and you have deduplication and single-instance storage! In fact, as the compression technology gains access to the underlying data, it can become more and more efficient. The software from Ocarina, for example, actually decompresses jpg and pdf files before recompressing them, resulting in astonishing capacity gains!

Now let’s look at compression’s secretive cousin, encryption. It’s only a small intellectual leap to use similar ideas to hide the contents of a file, rather than just squashing it. But encryption algorithms are constantly under attack, so some very smart minds have come up with some incredibly clever methods to hide data. One of the most important advances was public-key cryptography, where two different keys are used: A public key used for writing, and a private key to read data. This same technique can be used to authenticate identity, since only the designated reader would (in theory) have the key required.

Cryptography has become exceedingly complicated lately in response to repeated attacks. Most compression and encryption algorithms are deterministic, meaning that identical input always yields the same output. This is unacceptable for strong encryption, since a known plaintext attack can be used with the public key to reveal the contents. Much work has focused on eliminating residues of the original data from the encrypted version, as illustrated brilliantly on Wikipedia with the classic Linux “tux” image. The goal is to make the encrypted data indistinguishable from random “noise”.

What happens when we mix these powerful technologies? Deduplication and encryption defeat each other! Deduplication must have access to repeating, deterministic data, and encryption must not allow this to happen. The most common solution (apart from skipping the encryption) is to place the deduplication technology first, allowing it access to the raw data before sending it on to be encrypted. But this leaves the data unprotected longer, and limits the possible locations where encryption technology can be applied. For example, an archive platform would have to encrypt data internally, since many now include deduplication as an integral component.

Why do we prefer compression to encryption? Simply because that’s where the money is! If we can cut down on storage space or WAN bandwidth, we see cost avoidance or even real cost savings! But if we “waste” space by encrypting data, we only save money in the case of a security breach.

A Glimmer of Hope

I had long thought this was an intractable problem, but a glimmer of hope recently presented itself. My hosting provider allows users to back up their files to a special repository using the rsync protocol. This is pretty handy, as you can imagine, but I was concerned about the security of this service. What happens if someone gains access to all of my data by hacking their servers?

At first, I only stored non-sensitive data on the backup site, but this limited its appeal. So I went looking for something that would allow me to encrypt my data before uploading it, and I discovered two interesting concepts: rsyncrypto and gzip-rsyncable.

rsync is a solid protocol, reducing network demands by only sending the changed blocks of a file. But, as noted, compression and encryption tools change the whole file even if only a tiny bit has been altered. A few years back, the folks behind rsync (who also happen to be the minds behind the Samba CIFS server) developed a patch for gzip which causes it to compress files in chunks rather than in their entirety. This patch, called gzip-rsyncable, hasn’t been added to the main source even after a dozen years, but yields amazing results in accelerating rsync performance.

The same technique was then applied to RSA and AES cryptography to create rsyncrypto. This open source encryption tool makes a simple tweak to the standard CBC encryption schema (reusing the initialization vector) to allow encrypted files to be sent more efficiently over rsync. In fact, it relies on gzip-rsyncable to work its magic. Of course, the resulting file is somewhat less secure, but it is probably more than enough to keep a casual snooper at bay.

Both of these tools are similar to modern deduplication techniques in that they chop files up into smaller, variable-sized blocks before working their magic. And the result is awesome: I modified a single word in a large word document that I had previously encrypted and stored at the backup site and was able to transfer just a single block of the new file in an instant rather than a few minutes. My only real issue is the lack of integration of all of these tools: I had to write a bash script to encrypt  my files to a temporary directory before rsyncing them. I wish they could be integrated with the main gzip and rsync sources!

If you are interested in trying out these tools for yourself, and if you use a Mac, you are in luck: Macports offers both tools as simple downloads! Just install macports, type “sudo port install gzip +rsyncable” to install gzip with the –rsyncable flag, then type “sudo port install rsyncrypto” and you’re done! I’ll post more details here if there is interest.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Compression, Encryption, Deduplication, and Replication: Strange Bedfellows
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Paul goes on to quote me, saying I “didn’t think primary data de-duplication technology was ready for prime time just yet.” I absolutely did say these words, but I am not sure if my point came across.

I’ve recently expounded about the benefits of deduplication technology, but have warned that it might not be all it’s cracked up to be in primary storage environments. By “primary” I mean those storage environments serving mission-critical applications. Although dedupe works great for backup and archiving, the random I/O, low latency, and high throughput of primary storage (and especially virtualized servers) might be too much for current systems. And as of now, only NetApp, Riverbed (soon), and startups greenBytes (see my story) and Ocarina (more on them another time) were willing to go on record with me as supporting deduplication of primary storage.

So what I meant was that deduplication is not yet ready for prime time in primary storage applications. No one should hesitate to use the technology for backup or archiving at this point, but make sure you do a thorough evaluation of the specific product you are selecting to make sure it delivers the performance you require!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Is Deduplication Ready for Prime Time?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.