It’s fun to bash Microsoft. It’s easy, too, with Apple solidly conquering the high end of the PC and mobile markets and Google’s command of the Internet. But how fair are these articles skewering Microsoft, such as “Microsoft’s chronic lack of innovation” published today at Techworld? I suggest that Microsoft innovates as well as, if not better than, any other massive company. But no one innovates like an outsider.

Note: I am a Microsoft MVP in the area of File System Storage and will be on the Redmond campus all week as part of their Global Summit for MVPs. I am not a Microsoft apologist or sycophant and have been both harshly critical when the company deserved it and full of praise at other times. Mostly I just focus on the value of enterprise information technology and try to give all companies and products equal skepticism.

Run Versus Change

Large businesses tend to group projects into two categories:

1. Run the Business projects focus on maintaining the status quo, keeping the money flowing in, and satisfying the demands of existing customers.
2. Change the Business projects are far trickier, attempting to innovate and add new products or services to keep up with the competition.

This methodology has come about through years of experience balancing efforts that either upset the apple cart or let the apples rot. Look at the history of business and you will find that most successful businesses strike a balance between run and change. Those businesses that have failed have done so because they did not strike this balance, either ignoring their current needs in an attempt at reinvention or stifling change in the name of risk management.

One of the criticisms leveled against the American automakers is that they focused too heavily on serving core markets and too little on innovating into new ones. Thus, the average age of Cadillac, Buick, and Lincoln drivers shot upward and no young person would be seen in one of their cars; Dodge, Ford, and Chevrolet derived nearly all of their profit from massive pickup trucks and stopped developing more economical cars; all of the Big Three relied too long on still-profitable older designs while their competitors developed newer, better-performing ones. In short, the American auto industry nearly collapsed because they put all their energies into running the business and very little into changing it.

High tech businesses often have the opposite focus. Companies like Yahoo and AOL allowed successful existing businesses to wither while they tried unsuccessfully to expand into new markets and take on new competitors. Yahoo Mail accounted for more than one third of that company’s traffic last year and remained the dominant webmail platform even after half a decade of challenge from Gmail and more from Microsoft Hotmail. But Yahoo spent that time thrusting this way and that into every conceivable business with little success. One can see the same pattern at eBay, Motorola, 3COM and many others.

Microsoft’s Balance

It would have been very easy for Microsoft to fall into this trap over the last decade, and indeed they have often made this sort of change-over-run move. Consider the massive money Microsoft invested or wasted (depending on your perspective) on unprofitable and perhaps even quixotic mobile, online, and gaming properties: If a constant stream of revenue from Windows, Server and Tools, and Office was not available to balance this, Microsoft would have had a shareholder revolt on their hands. As I mentioned last week, Google is in a similar position, deriving nearly all of their revenue from advertising even as they try and fail to innovate in other areas.

Silicon Alley Insider's Dan Frommer and Kamelia Angelova presented this chart of Microsoft's profits on Feb 10, 2010

One school of thought is that Microsoft has simply milked the Windows and Office cash cows rather than innovating, but others might criticize the company for trying to establish a presence in too many areas instead of focusing on their core products. The fact that these two opposite viewpoints are widespread indicates a third option: Microsoft is trying to balance innovation and consistency both inside and outside their core areas of competence.

Consider Office, Server, and Windows: The combined effect of Microsoft’s monopoly power and sheer inertia would not keep the company dominant in these two areas forever if they did not keep innovating. There have been many points over the last 20 years where Microsoft has been vulnerable on the desktop. The 32-bit transition from Windows 95 to Windows XP was long and painful, and Microsoft is repeating this with Windows 7. The company must innovate to the very core of the operating system to make these transitions, though most of these improvements go unnoticed. Monopoly or not, I cannot comprehend a successful Microsoft desktop strategy if “Windows 2010″ was a simple evolution of Windows 95.

But Windows Server’s storage components are the core of my expertise, and Microsoft deserves much credit for innovation here. The creativity and originality of Windows Server storage features is lost on most, but that doesn’t mean they are any less innovative. I’ve written about the iSCSI initiator, MPIO, and VSS before, but there is much more than that. The peer-to-peer system used in BranchCache is startlingly creative, for example, but the pundits probably never heard of it. Under-the-covers innovation to support and continue a company’s success deserves credit, too.

Even Microsoft’s “branching out” activities are starting to see some success. Bing is much more impressive than many (including me) would have guessed, and users are beginning to notice. And the Xbox gaming platform, which has finally begun turning a profit, is a dominant player in that market. Indeed, many young people know Microsoft more as a gaming company than anything else.

Applauding Innovation

This kind of success in a new area is not unprecedented, but many companies have failed the test. In 1960, General Motors introduced one of the most innovative automobile platforms in history to take the small-car market head on. Every one of the so-called “Y-body” cars featured innovative features and industry firsts, from the Corvair’s rear-mounted aluminum flat-6 engine to Oldsmobile’s first production use of a turbocharger. Amazingly, most of these innovations flopped, and GM’s replacements were much more conventional. Innovation does not always succeed, especially when applied to a new market, so one should especially applaud when it does.

So is Microsoft innovative? Although the company has a reputation for bullying, monopolizing, and destroying competitors with “me too” offerings, a deeper look reveals true creativity both in core and speculative products. A massive company like Microsoft must be of two minds: Focused on continuing their core business while branching out into new areas. Success in either of these endeavors demands innovation, and Microsoft has undoubtedly succeeded.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Microsoft’s Overlooked Innovation
This post was categorized as Computer history, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although “don’t be evil” isn’t Google’s official corporate motto, the company and its admirers have embraced the concept implicitly and explicitly. But pride goeth before a fall, and the buzz around Google isn’t just about their new social networking feature: Cynicism and disillusionment with Google is growing.

Why Do They Hate Me?

Last week I wondered out loud about this: When did everyone get so cynical and disillusioned with Google? It’s ironic that answers rolled in on Twitter and FriendFeed even as Google was stomping into their turf with Buzz. What response did I get?

1. Google’s corporate censorship moves, especially in China, look pretty evil to some
2. Google’s lack of innovation outside search, especially its repeated attempts to “take over” social media, soured many
3. The poor and informal customer support provided by Google is notorious
4. Many just don’t “get” Google’s mission anymore – are they a search engine, an advertising platform, a software or hardware vendor, or what?

Companies are going to make mistakes, especially massive and aggressive ones like Google. It is inevitable that their compromised position on freedom of speech in China (and India) would raise hackles, but the company apparently decided it was acceptable to gain entry there. But many of Google’s other moves are more troubling to some.

Paved With Good Intentions?

Almost every penny of Google’s prodigious revenue comes from personalized advertising. Google plainly states this in their annual reports: “Advertising revenues made up 99% of our revenues in 2006 and 2007 and 97% of our revenues in 2008.” The company spends about 13% of this revenue on research and development, 9% on sales and marketing, 7% on administration, and 5% on stock-based compensation. I imagine many of these numbers will come as a shock to average Internet users, many of whom probably assumed Google was less dependent on advertising revenue and spent much more money to employ and house so many great software developers.

Google’s stated mission may be “to organize the world’s information,” but that’s not what the company actually does. It runs a massive collection of Internet properties which serve to collect personal information and serve advertisements. Many of Google’s employees seem to be genuinely interested in making the world a better place, or at least organizing the world’s information, but good intentions don’t pay the bills. Shortly after launching its eponymous search service, Google began gobbling up the lucrative advertising market it now dominates.

Not everyone is bothered by this. Many, including myself, are happy users of Google’s excellent products, including search, Gmail, Reader, News, and Maps. I’ll knowingly put up with targeted ads to subsidize these services because I trust that Google really is anonymizing and protecting my information. I’m sure most users don’t really think much about privacy and freedom when searching the Internet or sending an email, but even those that do have been content with Google.

Yet even Google fans have to admit that not every product is excellent. Many, like Wave, seem half-baked while others, like Orkut, seem more like misfires. Google almost missed the boat on the social web and now seems desperate to catch up. Core technology like PubSubHubbub is heading in the right direction, but Google has been unable to stitch it all together. Perhaps Buzz will be able to ride Gmail’s coattails to success, but we have seen so many failures before.

What Is Google?

I think the core criticism of Google is more fundamental than concern about censorship, advertising, privacy, or failed products. Instead, alarms are ringing at Google’s repeated and well-funded attempts to be much more than an organizer of information.

In 2008, the company tried and failed to muscle in on wireless spectrum, a move many thought was a carefully executed trick to force open the United States mobile phone market. They then introduced their own mobile phone operating system, Android, putting themselves in direct competition with Symbian, Microsoft, and especially Apple. This intensified with the Google-branded Nexus One, the removal of CEO Eric Schmidt from Apple’s board, and the launch of multi-touch features. Google seems to be stepping up to directly challenge Apple for dominance of the new mobile computing world.

The company is also moving into fixed broadband, offering temporarily-free Wi-Fi at major airports and announcing a plan to give free gigabit fiber optic service to communities in the United States. They are becoming a telecom company with Voice, and some have suggested a buyout of T-Mobile or the launch of Google-branded phone service. Google is also a cloud computing company, a hosting provider, and even an electric power company.

Even if all of these moves are taken as support of the company’s core mission, they do lead one to question Google’s corporate mission. If their advertising business wasn’t the limitless cash cow it has historically been, I’m sure Google’s investors would be asking some hard questions. What business does an advertising company have supporting blimps, gene sequencing, and electric cars?

Even if you are not worried about the money or the wisdom of these investments, it begs the question, “what is Google?” I believe this is the source of Google’s buzz-kill. In dominating the Internet, Google has tapped into a vein of confusion, concern, mistrust, schadenfreude, and downright hostility. I suppose it comes with the territory.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Google’s Evil Buzz Is Building
This post was categorized as Apple, Computer history, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I’m an old-school UNIX guy. This is one reason I like Mac OS X so much: It’s a solid UNIX system under a pretty and functional GUI. Unlike (probably) most Mac users, I find myself using terminal sessions frequently, often running a few different windows with ssh sessions to various UNIX systems.

My affinity for the command line probably explains my fascination with clever fonts. If you’re like me, I think you’ll chuckle at the bizarre combination pictured below.

Two stupidly-cool fonts in action: My normal DEC VT220-like terminal and a super-large terminal with super-tiny Tom Thumb displaying Hamlet. All of Hamlet. No kidding.

VT220 Terminal

That green-on-black window is my standard terminal. It’s an awesome duplicate of the original DEC VT220 font that really brings back my days sitting at “the Wedge terminals” at WPI. The font is “Glass TTY VT220” by Viacheslav Slavinsky. I’m running it at 20 points to make it look perfect.

In the window is Carole Gunst’s blog post from HighTechHistory.com announcing publication of the autobiography of Digital co-founder Harlan E. Anderson, “Learn, Earn, and Return: My Life as a Computer Pioneer.” I thought that was apropos.

Tom Thumb Hamlet

The one in the back is just plain weird. It’s a terminal using a miniscule yet readable font, “Tom Thumb” by Robey Pointer (based on a font by Brian Swetland). I used FontForge to convert Robey’s BDF to an OS X-compatible dfont package and fired it up in Terminal at 6 points.

Since I was using the 27″ iMac with crazy-huge 2560 x 1440 resolution, I got curious about just how much information could be displayed in a Terminal window at once. Running Terminal at 633 x 197 characters yields 124,701 characters on screen, along with the scroll bars, header, and menu. That’s enough for every word of Hamlet on screen at once. Including footnotes. Wow!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Two Stupidly Cool Terminal Fonts
This post was categorized as Apple, Computer history, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Using system memory for storage is something of a lost art these days. Although system RAM capacity has become plentiful, cheap and quick disk storage is just as common. But many of today’s I/O intensive tasks can still benefit from the untouchable quickness provided by a ramdisk. Happily, most operating systems are still capable of creating and using ramdisks. This article discusses the creation, use, and performance of ramdisks in Mac OS X.

On The Extinction Of Ramdisks

System RAM is not typically used for storage anymore

Ramdisks were a cheat to make slow system I/O tolerable, but it was never an easy decision. These days, computers are fast enough that ramdisks are rare. Every part of the computer has gotten quicker and cheaper over the last two decades, including hard disk drives, RAM, and CPUs. A typical laptop like my MacBook Pro might have 4 GB of system RAM, 320 GB of hard disk capacity, and dual 2 GHz CPUs.Those specs would have seemed inconceivable just a few years ago: I still have a Dell laptop with a 20 MHz CPU and two expensive upgrades: 1 MB of RAM and a 20 MB hard drive!

Far more impressive, though, are the numerous performance optimizations that have appeared. I/O channels are quicker than ever, and caches have appeared at most performance bottlenecks. It’s hard to believe, but computers used to rely much more on raw storage. Many hobbyists recall waiting for data to load from cassette tape drives, or that one key performance advance in Intel’s 80486 chip was its on-chip cache. Today’s CPUs have three levels of cache, and operating systems have gotten much better at caching data internally as well.

Hard disk drives have sprouted their own caches, too, making I/O seem quicker than it is. My 1997 Toshiba laptop used an un-cached 4200 rpm hard disk, but even laptop drives today sport 16 MB or more and spin at 5400 rpm or faster. Although RAM was scarce back in the day, the performance gained from a ramdisk was worth the effort. Faster I/O and strategic caching has largely eliminated this need. Increasing operating system intelligence also plays a major part – witness the failure of hybrid hard drives.

The Modern Use Case

A long-standing argument in the field of computer system performance revolves around whether to manually reserve resources and place data or to let the system dynamically manage resources on its own. Computers do a decent job if allowed to, often adapting quicker and more efficiently to changing demands. But some jobs are harder to automate, and buffers and caches don’t always catch the right data.

Purely temporary data can be written to a ramdisk as a high performance scratch space. There is no need to store this on disk at all, and merely placing it there is likely to “pollute” the buffers, pushing out real useful data.

A ramdisk can also increase security somewhat by never allowing certain data to be written to disk. Ramdisks are sure to be flushed on the next reboot, but data on disk can linger and be discovered later.

One could even argue that today’s high-performance solid state disk is simply an evolution of the old ramdisk concept. NAND flash and RAM both offer the high random performance of the ramdisks of yore, and many analysts expect them to displace high-performance hard disk drives in the coming years. So perhaps ramdisks are not extinct after all!

Ultra-performance storage systems like this Violin array use RAM for storage

Using Ramdisks In Mac OS X

Mac OS X has the ability natively to create and use ramdisks. Here’s the simple procedure:

1. Create the ramdisk with hdiutil – This terminal command will create a 1 GB ram disk and report the name used. The number following the “ram://” statement is a number of 512 KB disk blocks – multiply the desired ram size in MB by 2048 (that’s 1024 times 2) to derive other sizes.
   hdiutil attach -nomount ram://2097152
2. Format the ramdisk with diskutil – The next command will format the newly-created ramdisk using the diskutil tool. We will use /dev/disk3, but you should specify whatever disk device hdiutil reported from step 1. You can also specify a different filesystem or disk name.
   diskutil eraseVolume "HFS+" "ramdisk" /dev/disk3
3. Your ramdisk will now appear – OS X will now mount the newly-created ramdisk and it will appear on your desktop. Note that the contents will be destroyed if it is unmounted or if the operating system is rebooted!
4. Delete your ramdisk – It’s worth saying once again: The ramdisk will disappear and all data will be deleted when you eject it or power off your computer. Be sure to back up any important data from it, then eject it with the following command:
   hdiutil eject /Volumes/ramdisk

Script It!

My backup script called for the creation of a tar file, compression, encryption, and rsync copying. I decided that I could perform the first three functions on the ramdisk to increase performance and reduce the chance that an unencrypted version would remain on the disk if the script failed.

A simple command line combination of the above creates the necessary ramdisk:

diskutil eraseVolume "HFS+" "ramdisk" `hdiutil attach -nomount ram://2097152`

This failed once I put it in a shell script, however. The issue was the finicky nature of diskutil – hdiutil’s output included a number of whitespace characters, and diskutil failed in the script whenever it encountered this. So I used the cut command to grab just the first field of the output, like so:

diskutil eraseVolume "HFS+" "ramdisk" `hdiutil attach -nomount ram://2097152 | cut -d " " -f 1`

This solved my problem, and I was able to tar and compress to /Volumes/ramdisk in my script. I added the hdiutil eject command to the script right after the encryption step to wipe out the unencrypted data. Although this isn’t foolproof security by any means, it’s better than nothing.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Ramdisks: Back From the Brink of Extinction
This post was categorized as Apple, Computer history, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Making a list? Who's naughty and who's nice?

Who’s naughty and who’s nice? The average computer geek of the last decade would have placed Microsoft atop the naughty list. The average corporate IT manager’s nice list probably wouldn’t have included EMC and Oracle. Yet Google, Apple, Sun, HP and even IBM don’t have this frequent negativity directed towards them. What’s the difference between naughty and nice when it comes to IT companies? I’ve heard complaints of the greed and arrogance of these companies, though their boosters would point out that it’s easy to envy the success of others.

But things are changing. Microsoft has a bona fide hit on their hands, with Windows 7, Xbox, and Bing re-introducing the company to new customers that don’t harbor old grudges. Inside corporate IT, the halo cast by VMware seems to highlight the re-energized EMC in much the same way. With their competition taking dents in the ongoing battles, Microsoft and EMC just don’t look so bad anymore.

Microsoft: Hearts and Minds

The blooms in many Microsoft competitors’ rose gardens seem to be fading. With “do no evil” Google only finding lucre in the filthy advertising business and the naughtiness of “evil as we wanna be” Apple peaking, Microsoft’s Internet and consumer efforts are starting to seem downright approachable. That’s one way to change your image: Wait for your competitors to catch up and your customers will catch on. The geek parade still loves Google and Apple, but their ambitious drive and massive revenue are distasteful to many.

Every time I write about Apple products, at least one credible geek has to call me out for being a fanboy. The core of their arguments seem to combine scorn for friendly interfaces and pretty hardware, a distaste for Apple’s huge profit margins, and a belief that the faithful wear Apple-tinted glasses when looking at alternatives. I guess Apple users look like a bunch of sissies to the more manly geeks in the audience.

In fact, it’s become something of a badge of pride to stick by Microsoft, even as the UNIX weenies and Apple-heads wander off. They ask “who’s got the most market share in desktops and servers?” Windows Vista’s appetite for hardware and unstable nature might have challenged them, but the quick, slick, solid Windows 7 has reaffirmed their faith. And they know that those who throw stones at Windows Server are living in the past: Ridiculous naming aside, Windows Server 2008 R2 is every bit as great in the data center as Windows 7 is on the desktop.

But there’s more to Microsoft than Windows. Even ardent Microsoft haters have to admit Bing is solid, functional, and even clever. Indeed, Microsoft has taken the search battle right to Google and is working hard to innovate past their rival. Xbox has a solid beachhead in the gaming world, challenging successful and innovative products from Nintendo and Sony. Azure puts a developer-friendly face on the nascent cloud computing market and is anything but a “me-too” to Amazon EC2 and VMware. Barring any major product or PR disasters, Microsoft is well on the way to renovating their sagging corporate image.

EMC: Keeping It Real

EMC is leaving the little storage market behind and doesn't look as big and scary in the larger IT world

What Microsoft is to average computer users, EMC is to enterprise data storage folks. No one denies that they make great products, and have dominated the market for two decades. Although they don’t have the massive share Microsoft has in the desktop OS market, no one comes close to EMC in enterprise storage. They spent the last decade steadily growing to control 25% of the market leaving a wealth of competitors fighting it out far below.

Through all this growth, however, EMC has never been loved by their customers. I’ve known literally dozens of IT shops who refused to buy from EMC, even though the sleazy sales tactics that turned them off (and indeed the sales reps themselves) are reportedly long gone from the company. Like Microsoft, EMC hasn’t softened its approach as much as their competitors have hardened theirs. With the market getting tougher, the tough guy doesn’t look so bad anymore.

I hear that things have improved inside the company, too. All giant corporations have their share of intrigue, politics, and dead weight, and EMC is certainly no exception. But the reports I hear from insiders are positive, and improving all the time. EMC is making some smart moves, giving acquisitions the independence to thrive and building revenue outside their enterprise storage base. Hiring great folks like Scott Lowe, Christopher Kusek, and Ed Saipetch doesn’t hurt, either.

Customers seem to be sensing a change, too. It’s hard to hate VMware, RSA, Legato, and the rest of EMC all at once, though some have grudges against two or three. EMC is successfully diversifying into other areas of information technology. Like Microsoft, EMC’s new customers never learned the old stereotypes. Now that they’re swimming in a much larger pond, EMC looks neither as big or as bad as it once did.

You Will Decide

Are EMC and Microsoft really turning the corner? We will all know in a few years. If the geeks of tomorrow no longer resent their success and hold past mistakes against them, both companies could enter a renaissance not only of credibility but of business success.

Santa Claus image: Public domain from Project Gutenberg

Gorilla image: public domain from Pearson Scott Foresman

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Are Microsoft and EMC beginning a renaissance of geek respect?
This post was categorized as Apple, Computer history, Enterprise storage, Gestalt IT, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It’s that time again, when everyone who thinks they’re a pundit (that would be everyone with a blog or Twitter account) has to make predictions for the coming year. But predictions are perilous: Get it right and you look like a mere trend-watcher; get it wrong and you look like a fool. It’s such a hassle! So I’m doing something different this year: I’m going to make predictions for 2009 now that it’s over, and reflect on just how smart I am (not) to have made them. Or something.

What I Would Have Gotten Right

I definitely could have predicted a lot of what happened in 2009. I mean, these were slam dunks!

1. Twitter rocks the world – I wasn’t early to Twitter, but I spent the early part of 2009 evangelizing its benefits to companies and co-workers alike. Considering how common Twitter is today, it’s hard to believe how roundly criticized and misunderstood it was this time last year. Yet here we are, on the verge of 2010, and Twitter has seeped onto our business cards, presentation templates, and web sites. I might not have predicted how stable (!) Twitter got by the end of the year, though.
2. Apple’s Macs and iPhones rule – I switched to the iPhone and the Mac in 2007 and 2008, respectively, but it looks like I wasn’t much of an iconoclast after all: By November, half of the Tech Field Day delegates were using MacBooks, and the Windows and Blackberry holdouts have started vocally defending their operating system choice. Pretty much like Mac folks used to do way back in 2008.
3. The recession is a serious pain – Companies put the brakes on spending and hiring, many even shifting both into reverse in 2009. This came as no surprise to humans capable of thought. The impact on enterprise IT companies was similarly predictable: Although most were able to survive, the impact of 2009 will continue to be felt for years. I might have predicted it would be worse, though I’m glad to say I would have been wrong.
4. EMC, NetApp, HDS, HP, and IBM continue to quibble – Surprise: Big company bloggers spend way too much time criticizing the products and actions of each other and way to little time talking about the true value of their own products.

Non-IT slam-dunk predictions: Obama was reviled by the right; the war in Afghanistan continues; people do stupid stuff in the name of reality shows.

What I Probably Could Have Predicted

Although some details would likely have been missed, I think I would have seen these coming.

1. Cloud compute and storage hits the enterprise – I was a believer in the cloud this time last year, and I bet my future on it by taking a position at enterprise cloud storage provider, Nirvanix, in March. I would have predicted that enterprise buyers would be putting serious thought to buying cloud products, but the scope has surprised me. We’re talking enough petabytes that the non-cloud players felt compelled to strike back with the private cloud pitch. Awesome!
2. Sun and Data Domain were acquired – My money would have been on Dell, IBM, or HP as buyers for this pair, but EMC wouldn’t have been outside my guesses. Still, Oracle buying Sun and vocally committing to keep it going, SPARC and all, would never have come to mind. But I wouldn’t have guessed against it either, so I’ll give myself a point here!
3. Cisco and EMC buddy up – I’ve long thought an outright merger of these two was in the cards, but even the recession couldn’t make the financials work. A partnership would have been on the list, and Acadia came as no surprise to anyone.
4. Cloud outages and data loss – I definitely could have predicted that high-profile cloud services would fall over throughout the year, and that some would lose data. Not all are enterprise-grade, after all. But the outages at Google, Rackspace, and Amazon, and Microsoft’s Danger data loss, surprised me. Don’t those guys have their acts together?
5. IT conferences falter – I spoke at Interop in 2009, but it lacked the 20,000-strong crowd it once had. Storage Decisions and Storage Networking World managed to fill their halls, but the old-school IT conference has lost its luster. Although VMworld remains strong, attendance was definitely off.
6. FCoE and SSD are still starting – I’ve been lukewarm on Fibre Channel over Ethernet and Solid State Drives, but I’m a bit surprised that storage vendors didn’t push them harder in 2009. I might have guessed there would have been more customer uptake to match the buzz.
7. SMB storage is hot – There’s a hole in the storage market between $1,000 and $20,000, and companies like Drobo and Iomega are rushing in to fill it. Now that ESX has solid iSCSI support, I expect a world of innovation here. (Oops, that sounds kind of like a 2010 prediction!)

Also in the predictable category: Goldman Sachs and Bank of America thrived while others fell; Ford is the strongest of the remaining US automakers; Boeing finally got the 787 off the ground.

What I Never Would Have Guessed

I’m not perfect, even in retrospect. Some of the Tech news from 2009 was just completely off the wall.

1. Microsoft Bing: This time for sure! – Seriously, Microsoft should stick to in-house thinking instead of trying to copy its rivals. Yet somehow, miraculously, Bing appeared and did not suck. In fact, I’m hearing regular (non-techie) folks around town talking about using the search engine. I’ve even used it! Could they actually have a winner?
2. Windows 7 rocks – Really? Seriously? Could Microsoft have come up with a solid replacement for Windows XP?
3. Ship it! – It’s not even 2010, and enterprise storage buyers can go out and purchase NetApp’s OnTap 8, EMC’s FAST, EMC Atmos Compute, and unicorn tears. Well, maybe not unicorn tears.
4. Still no GDrive – Seemingly every company has a cloud storage platform, from Amazon to Rackspace, Nirvanix to EMC, so why not Google? Could GDrive join Duke Nukem Forever as the most famous vaporware of the decade?
5. The executive shuffle – Dave Donatelli was supposed to lead EMC, not HP. Alan Atkinson was supposed to launch another startup, not take over Xiotech. At least NetApp was gentle.
6. Mac OS X (still) lacks iSCSI and ZFS – Come on, Cupertino, what’s wrong with you guys? I’ve been hyping ZFS for years, and iSCSI is commonplace. Yet Snow Leopard is stingy with both. Makes me want to hiss like one of those blue folks in Avatar.
7. Gestalt IT is a success – On a personal note, Gestalt IT didn’t even exist this time last year, and now we have a successful IT infrastructure blog and social media event. Amazing!

Other total shockers: Everyone loves Michael Jackson again; digital Beatles tunes are available everywhere but iTunes; Obama’s Nobel Peace Prize arrives 10 years early.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | My 2009 IT Industry Predictions
This post was categorized as Apple, Computer history, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Western Digital is first to bring "Advanced Format" 4k-sector drives to market, but buyers should beware of installing them in old systems!

Personal computer hard disk drive access methods have repeatedly been forced to adapt to ever-expanding capacity. Physical (CHS) addressing and 28-bit LBA have been abandoned, but the disk drives themselves retain the tiny 512-byte sectors they have had since IBM introduced fixed-block addressing in the 1970’s. But time marches on, and Western Digital is leading the change to larger 4 kilobyte hard disk blocks. Although this new “Advanced Format” includes mechanisms for backwards compatibility, buyers should be wary of these new drives for the time being.

A Little History

A long, long time ago, before there were personal computers, IBM set the standard for data storage with their CKD (“count key device”) format. Variable-sized records were stored on a raw disk with gaps between them. Each included a sequence number (“count”), an optional key, and the data itself.

This flexible format proved impractical as multi-process systems evolved in the 1970’s, so IBM introduced a new format: Fixed-block architecture (FBA). FBA disk drives were divided up into equal-sized blocks and addressed by their physical location on a specific cylinder accessed by a specific head. Each cylinder/head pairing had its own series of sectors, starting at the outer edge of the physical disk platter and moving inward.

This system of addressing hard disk capacity with a triple number identifying the cylinder, head, and sector (“CHS”) continued in the PC space in the 1980’s. Since early hard disk drives were tiny by modern standards, the PC industry chose the smallest sector size specified by IBM, 512 bytes. The earliest MS-DOS/BIOS PCs were limited to just 504 MB, thanks to incompatible defaults for the CHS address space. This forced a transition, initially raising the addressable capacity to 7.8 GB.

But the writing was on the wall: CHS just didn’t make sense. Storage protocols, including ATA, were being pressed into service for devices that didn’t have cylinders or heads, and the 128 GB limit of ATA-1 wouldn’t cut it forever. So the industry transitioned again, abandoning physical references in favor of a simple logical block addressing (LBA) scheme. By the time ATA-6 was introduced in 2002, all systems used LBA, and the new 48-bit address system allowed a maximum of 128 PB of capacity per device.

Though hard disk capacity had ballooned from the megabyte to the terabyte range (a million-times increase in 30 years), the 512-byte disk sector remained, each with its own error-correcting code (ECC) and gap.

Up To Date

Processors and file systems have long since moved on from 512-byte blocks of data, with 4 kilobytes being the most common size used by the x86 CPU and NTFS, ext3/4, and HFS+ filesystems in Windows, Linux, and Mac OS X. With just about every storage access triggering the reading or writing of eight disk sectors, an increase in hard disk sector size to 4 KB seems obvious. Stepping up to 4K enables better ECC checksums, making disks more reliable (in theory) and freeing up capacity.

Modern operating systems, including Windows Vista, Windows 2008, Windows 7, Mac OS X 10.4-10.6, VMware ESX 3.x-4.x, and most recent versions of Linux, will have no problem with 4 KB sectors in most cases. Installing a 4K drive, like Western Digital’s new “R” line, and partitioning and formatting it fresh should present no issues for most users. In fact, these operating systems will not even “know” they are talking to any special kind of drive, since Western Digital presents its new 4 KB sectors as plain old 512 byte sectors for compatibility reasons.

What’s Your Alignment?

Other systems present a challenge, however. MS-DOS traditionally started filesystems at sector number 63, and most other PC operating systems (including old versions of Linux and VMware) followed this convention. This presented no issue at all as long as disks used 512-byte sectors: Requesting 8 sectors starting at number 63 was the same as requesting eight starting at sector 64. But things get mucked up when the disk drive uses 4 KB sectors.

See the misalignment here? A starting offset of 63 moves the filesystem cluster across the 4K sector boundary. This misalignment means that every filesystem cluster access results in two disk sector accesses. This is a Very Bad Thing, especially for to write performance. Layers upon layers of convention, standards, and compatibility have left us with a problem.

Western Digital solves this problem in one of two ways:

1. Jumper pins 7 and 8 on the drive and it will internally offset all of its logical 512-byte sectors by one
2. Correctly align all partitions written to these new drives

The jumper method, though simple, is not a great idea. If the drive was ever moved or reformatted to a system that aligns its volumes differently (as do all modern operating systems), it would be puzzlingly slow. Instead, it’s best to simply realign any filesystems written to the drives, and Western Digital is providing a utility do just that.

USB, Drobo, and TiVo, Oh My!

Most external drive enclosures will simply pass read and write requests to these new drives as they get them. This means that an external USB enclosure with an Advanced Format drive will have exactly the same issues as an internal drive if it is partitioned under Windows XP or another old operating system. Not only that, but portable drives often get moved from system to system, so an XP-formatted USB disk will have this misalignment issue even when attached to a Windows 7 machine! As these 4K drives begin hitting the market, users should be careful to note if they are correctly aligned.

But what about other disk-using devices like Drobos and TiVos? The jury is still out here. TiVo appears to use misaligned partitions, so I don’t recommend installing 4K drives in a TiVo at this point. I contacted a number of other manufacturers, most of whom are “looking into” the matter.

Data Robotics did respond, and even posted a tech note about 4K drives on their support site. Their short answer is no, that 4K drives should not be used in Drobo storage devices. However, it is unclear if they are affected or just erring on the side of caution until they can test these new hard disks. They also promise an update allowing the use of 4K drives in the future.

I would be cautious about using 4K drives in anything other than a newish PC or Mac at this point. I will continue following the situation and will post updates on my blog as news comes out.

More Info

The following articles also offer great insight into these new 4K drives:

• Western Digital: Advanced Format Technology White Paper
• StorageMojo: Why we need 4k drives
• AnandTech: Western Digital’s Advanced Format: The 4K Sector Transition Begins

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Drobo, XP Users: Beware of 4K “Advanced Format” Drives!
This post was categorized as Apple, Computer history, Enterprise storage, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Single-parity RAID is under attack. Caching is the hottest trend in storage. The end of the high-performance disk drive is imminent. What happened? Increasing areal bit density has caused disk capacity to grow much faster than disk performance. A presentation at Storage Networking World by Ronald Bianchini of Avere exposed the mathematics of this phenomenon. Of course, hard disk platters are not getting larger – quite the opposite. But the bits are getting smaller, so the effect is the same:

• Capacity increases exponentially based on the formula for the area of a disc: π times radius squared
• Sequential performance increases algebraically based on the formula for the circumference of a disc: π times diameter

Therefore, sequential performance grows smoothly with disk density, but capacity increases much faster. Double the density of disk media and you can read twice as many bits in the same amount of time, but the disk now contains four times as much data. Iterate this a dozen times, a miracle performed regularly by hard disk drive manufacturers, and you have a serious bottleneck to both performance and reliability.

Disk capacity has outpaced performance over the last decade

Back in 2004, I gave this metric a name: Flush time. It is a simple calculation to answer the question, how long would it take to read the entire content of a hard disk drive? Let’s look at some real-world examples:

• In 2000, a 45 GB Western Digital 450AA disk could stream data at 25.4 MB/s, requiring 30 minutes to flush every byte out its UDMA/66 interface. This was a massive and slow drive at the time – enterprise disks were much faster. A 2000 Quantum Atlas 10K II SCSI drive (36 GB and 31 MB/s) could flush in 19 minutes!
• A 2004-era Seagate Barracuda 7200.7 boasted 160 GB ad averaged 44.5 MB/s, requiring about an hour for a full flush.
• By 2007, high-performance drives like the Hitachi 15K450 had hit 450 GB and about 100 MB/s in sustained throughput, but flush times were well over an hour.
• Today’s enterprise drives can push 200 MB/s and average 160 MB/s across the entire 600 GB of capacity. But this is still about an hour for a flush. But large-capacity SATA drives are much more popular for bulk storage. The Samsung Spinpoint F2 EcoGreen drive I use in my Drobo only delivers about 110 MB/s, requiring almost four hours to flush at 1.5 TB of capacity! Think this is unusual? Check out Hitachi’s popular E7K1000, which needs 2.5 hours at 1 TB and 118 MB/s.

What will happen to flush time over the next half decade if density continues to increase? How about 16 TB drives, 400 MB/s, and RAID rebuilds that last more than half a day?

Since (traditional) RAID rebuilds are directly impacted by flush time, today’s massive disk drives are killing RAID. And flush time is only the minimum required time – most RAID rebuilds take much longer! Then there is the issue of media reliability!

Note: Yes, I know there are alternative RAID schemes that get around this problem. Far from ignoring that point, I’ll be promoting these in future posts! Stay tuned for more on these topics…

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Flush Time
This post was categorized as Computer history, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I am spending a few weeks examining the truths and fictions that bind our industry together. Let’s start with one of my favorite old canards: That enterprise storage must be overpriced because bare disk drives are so cheap.

I have seen this straw man argument set up by so many throughout my career that it has become laughably predictable. Every time a new high or low point is set for enterprise storage cost, someone is there pointing out that a bunch of disk drives is vastly cheaper. Let’s call this the Dumb Disk Fallacy: Only a fool would claim that dumb disks are comparable to enterprise storage.

Are we supposed to be surprised that raw materials make up such a small percent of the cost of an integrated system? One can prepare a delicious meal at home using supermarket-bought ingredients for a quarter the cost of a restaurant outing. A single glass of wine at a bar often costs as much as a whole bottle at the store, yet the restaurant industry is holding on even through a recession.

Lack of expertise only one factor: Even one who can not whip up a soufflé can certainly boil some spaghetti, warm up some sauce, and pour a glass of Chianti! Convenience is another driver, since many lack the time required to shop and cook. But there are other benefits as well: Dining out is a social activity and restaurant meals allow us to sample unfamiliar cuisine. Clearly, there are many reasons for one to pay far more for a finished product than for the ingredients it contains.

The same is true of enterprise storage. Although they make up a significant proportion of the bulk of a given storage array, hard disk drives are often a small element in the overall cost. One does not price a salad by its lettuce or a lasagne by its pasta, after all. Modern enterprise storage systems are defined their capabilities, ranging from performance to reliability to flexibility. Indeed, core components like disk drives and processors have long since been commoditized, and many vendors are leveraging much larger-scale commodity subsystems these days. The day will soon come when major vendors will differentiate entire product lines based solely on the software they run.

Now, it is possible to build very cheap storage systems, and the price of these can even approach the raw disk cost. But finished storage systems will never be as cheap as just a bunch of disks (JBOD, for newbies), because of the following:

1. Disk arrays include lots of hardware components beyond disk drives – chassis, power supplies, controllers (complete with CPUs, RAM, etc), and cabling and connectors
2. Data must be protected using extra disk capacity for parity, mirrors, snapshots, and spares
3. Arrays include software to orchestrate the whole operation; even free software requires development, testing, and integration
4. Support and maintenance contracts are required for any production system
5. Supporting software is often a requirement, too, for configuration, operation, management, and integration with servers and applications
6. The companies selling the array deserve a bit of profit or they’ll go out of business and your support contract won’t be worth much
7. Don’t forget utilization: A half-empty disk costs twice as much as a full one, and it is awfully hard to make use of 100% of available capacity without a storage network of some kind

So how close to “dumb disk” can a production storage system get? Google’s is probably the closest: They use custom server boards, custom chassis, custom power supplies, custom racks, and low-end disk drives to keep hardware costs at a minimum; the entire thing is composed of commodity components, too; the array software is all written and supported in-house for very low overhead; the capacity is very highly utilized due to the inherent flexibility of the applications they support. So Google has attacked every one of these areas in an effort to drive out costs. But would it really be practical for folks other than Google to invent, construct, and run their own storage system?

Instead of dreaming about building our own storage solution, let’s look at the spectrum of storage available to the modern enterprise IT shop:

1. Bare disk drives are widely available with a handful of companies producing just a few different models. The idiosyncracies of these are interesting to some, but a hard disk drive is useless alone
2. Direct-attached storage offerings range from single-drive USB-connected external disks to multi-drive racks, yet these too are useless without a server to drive them. These normally lack any sort of intelligence or advanced features and quite a bit of effort is required to ensure acceptable levels of performance or reliability. But so-called JBOD or DAS storage can be incredibly cheap to purchase.
3. True storage arrays range from the most basic home NAS boxes to the most advanced enterprise systems. All are purchased primarily for their features, with price tags merely differentiating between competitive offerings. For this reason, storage developers focus the vast majority of their engineering, sales, and marketing efforts on advanced capabilities rather than per-GB cost.
4. Storage as a service is a different realm entirely. Although advances in manageability and utilization and commoditization of hardware allow today’s cloud storage offerings to be offered at very attractive per-GB price points, low cost is merely a welcome side benefit. Engineering and support resources focus almost entirely on enhancing user experience, while hardware is small component of the cost of delivering enterprise-class storage as a managed service.

I’m not an apologist for overpriced enterprise storage, but I recognize that arrays are more than just a collection of disks. I look forward to the day when we finally dispense with the dumb disk fallacy and focus instead on the real value added by enterprise storage innovation. But the realist in me knows that this straw man will continue raising his head for some time to come. I am sure that developments like EMC’s unified CLARiiON and Symmetrix hardware platform, the spread of software appliances on commodity server hardware, and excellent free storage software like Nexenta and FreeNAS will only add fuel to the fire. Therefore I entreat you, dear reader: Do not succumb and pitch disk that is dumb!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | The Dumb Disk Fallacy
This post was categorized as Computer history, Enterprise storage, Gestalt IT, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I usually welcome discussion (and even argument) about the things I know best: There is always more to learn, and the best insights come through engaging those who disagree with us. But some ideas have been argued so well for so long that they deserve enshrinement. For example, although non-scientists like to argue about evolution and climate change, the scientific community no longer feels that their theories in these areas require much discussion. Like gravity and relativity, they have been accepted as a foundation upon which to build more interesting hypotheses.

My field of enterprise storage has its share of generally-accepted theories:

• Availability, backup, and archive form a Data Protection Trinity: They are unique requirements calling for focused solutions.
• The Rule of RAID: Combining multiple disk drives in creative ways allows us to change the inherent reliability and performance of the system.
• When it comes to storage management, Homogeneity is Paramount: A single storage administrator can manage thousands of identical systems but would be hard-pressed to support a half-dozen unique ones.
• The entire history of computing demonstrates that Connectivity Trumps Capacity when sizing systems: Performance bottlenecks always limit the scalability of storage systems.

Each of these theories underpins the our industy’s daily routine of storing and retrieving the data that drives modern society. These storage theories are also targets for innovation, with the best minds constantly trying to bend or break them.

This album of storage theories also has a B-side, however. These are the no-longer-true theories that have been transcended, as well as the dubious beliefs that were never really true.

• Commutability of Management and Cost is highly suspect: Unless one is considering only identical and homogenous systems, the total cost of ownership (TCO) or number of administrators associated with a given system (TB/admin) cannot be compared between environments.
• The Price of Parity: The impact of parity calculations and multi-disk commits used to kill write performance, giving RAID-5 a bad name. But write-back caches and array intelligence have all but eliminated this “write penalty” for modern enterprise systems.
• Whenever the high cost of enterprise storage is to be refuted, someone is bound to trot out The Dumb Disk Fallacy, claiming that per-GB array costs ought to be comparable to the price of a bare disk drive. But the value of enterprise storage has always been greater than the sum of its parts.

Over the next few weeks, I will be sharing focused articles about these “holy cows” of the enterprise storage world. I encourage everyone in the industry to join me in taking a step back and shining some light on these and other truisms. Which do you agree with or dispute? Are there other theories that I have overlooked?

© sfoskett for Stephen Foskett, Pack Rat, 2009. | We Hold These (Storage) Truths…
This post was categorized as Computer history, Enterprise storage, Everything, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.