• My writing
  • 9 Ideas To Make Your Blog More Reader-Friendly
  • Creating Web-Friendly Collateral For Product Launches
  • The Best Black Friday Deals For Nerds Like Me (2010 Edition)
  • Do Not Buy Weird, Cheap, Off-Brand Android Tablets!
  • Overcoming The Limits Of Thin Provisioning With Automatic Provisioning!
  • What To Look For In A Laptop (Holiday 2010 Edition)
  • Are Hybrid Hard Drives A Good Alternative To An SSD?
  • Buy The Speedy Seagate Momentus XT 500 GB Hybrid Drive For Under $120!
• Enterprise tech
  • NetApp asks rival to cease and desist FUD, while EMC and IBM fight
  • Storage Caching vs Tiering Part 2
  • Saving our data from digital decay
  • All the Space You Need: The Advantage of Thin Provisioning With Cloud Storage
  • Competing With Cisco
  • Overland Offers 2-for-1 Storage Capacity for SnapServers
  • Storage biggies lining up to buy Seagate?
  • Novell Rides Off Into The Sunset
  • Cisco IOS Hints and Tricks: FCoE, QCN and Frame Relay analogies
• Cool geeky stuff
  • Binary marble adding machine
  • Worst gadget ever? Ars reviews a $99 Android tablet
  • The Muppets – Popcorn
  • How to Bypass Carriers Apple-Style
  • Review: Aerobie AeroPress coffee maker

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Back From the Pile: Interesting Links, November 26, 2010
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Four Horsemen of Storage System Performance: These four ugly gentlemen stand between you and your data.

Why do some data storage solutions perform better than others? What tradeoffs are made for economy and how do they affect the system as a whole? These questions can be puzzling, but there are core truths that are difficult to avoid. Mechanical disk drives can only move a certain amount of data. RAM caching can improve performance, but only until it runs out. I/O channels can be overwhelmed with data. And above all, a system must be smart to maximize the potential of these components. These are the four horsemen of storage system performance, and they cannot be denied.

The Chain of Command

It is tempting to think of storage as a game of hard disk drives, and consider only The Rule of Spindles. But RAM cache can compensate for the mechanical limitations of hard disk drives, and Moore’s Law continues to allow for ever-greater RAM-based storage, including cache, DRAM, and flash. But storage does not exist in a vacuum. All that data must go somewhere, and this is the job of the I/O channel.

To be useful, storage capacity must connect to some sort of endpoint. This could be the CPU in a personal computer or an embedded processor in an industrial device. Indeed, there are endpoints and I/O channels throughout modern systems, with potential bottlenecks, caches, and smarts at each point. “Storage people” like me tend to think too small – imagining that the I/O channel ends at the disk drive, the “front end” of the array, or the storage network. But data must travel further, all the way to its final useful point in the core of the CPU.

Once we consider I/O as a long chain of interconnected endpoints, we begin to see the fact that I/O constraints at any point can strangle overall system performance. This is not merely an academic exercise: Optimizing the I/O channel is a consuming passion for most practitioners of enterprise IT, including architects, engineers, and system developers. And, like a good game of Whack-a-Mole, increasing the speed of one link causes another chokepoint to rear its head.

Parallel and Serial I/O

Imagine you had a warehouse full of boxes to move across the country as fast as possible. There are a few options available to you:

1. A fast truck can zip back and forth with just a few boxes
2. A train is slower, but its many cars can haul a huge quantity

But there are realistic limits to both capacity and speed: The train has to fit on the tracks, and the truck can’t move at the speed of light. Plus, one must consider the time taken to load and unload the chosen vehicle.

We continually shift between parallel and serial I/O paradigms

The same trade-offs are true of computer busses: Serial channels can be optimized to zip individual bits back and forth, or parallel busses can be designed to carry whole bytes (or more) at a time. The simplicity of serial communications is tempting, but designers continue to resort to parallelization for added throughput.

Note: Most serial protocols actually feature two links, making them “full duplex”: One for transmit and another for receive.

Serial storage interconnects are dominant currently, with fraternal twins SAS and SATA coming to dominate the disk interface landscape. SAS and SATA share the same 1.5, 3, and now 6 gigabit per second serial physical interconnect, offering more than enough throughput for conventional hard disk drives and edging out older serial (Fibre Channel, SSA) and parallel (ATA and SCSI) alternatives.

Networks (Ethernet, Fibre Channel, and InfiniBand) are predominately serial as well, as are lower-end interconnects like USB and FireWire. Serial communication also dominates in the system bus world, with serial PCI Express toppling parallel PCI.

But parallel variants are often offered for increased throughput: Multi-lane PCI Express and bonded multi-link InfiniBand make up a fair portion of the installed base, while load balancing MPIO drivers are common in Fibre Channel storage. And let’s not forget that the “X4″ variants of Ethernet use multiple bonded links as well.

The Definition of Bottle Neck

Most English speakers have encountered the French term, “cul de sac”, meaning “bottom of the bag” or dead end. But hard disk drives have plenty of “bottom end”, or storage capacity. When it comes to disks, the issue is usually at the neck of the bag: Data just can’t be pulled out of a hard disk drive fast enough.

Emptying a barrel of wine through a spigot takes hours, but pry the end off and the floor is covered in a moment!

The density of modern hard disk drives (the capacity of our barrel) has been growing much more rapidly than the I/O channels serving them (the spigot). Where once a hard disk drive could be filled or emptied in an hour or two, modern drives take days or weeks!

I once called this “flush time“, but I think the wine metaphor is much more appetizing!

This “bottle neck” has serious implications beyond basic storage performance. Data protection is impacted, since ever-larger storage systems can no longer be backed up by dumping their content; system reliability is reduced, since week-long RAID rebuilds increase the risk of multiple drive failures; and cost containment efforts are also impacted, since adding spindles drives up prices.

Nowhere is this bottleneck more evident than in portable devices. Modern drives (like the 1 TB Seagate USB drive I recently reviewed) have massive capacity and pathetic performance. The USB 2.0 interface just can’t keep up, and this creates a limit to the expansion of capacity. It would take half a day to fill that drive under perfect conditions at 25 MB/s, reducing its value as a massive data movement peripheral. The emerging USB 3.0 standard promises to alleviate this performance issue for now, as illustrated with Iomega’s new external SSD.

Cache and solid state storage can help, but they have their own bottlenecks. Storage arrays typically use Fibre Channel or SAS SSDs, and their front-end interface remains the same. The best-performing SSDs use the PCI Express bus directly rather than emulating hard disk drives over SCSI interfaces. And even PCI Express might not be enough to handle the massive I/O of NAND flash or DRAM. In each case, the bottleneck moves down the chain.

A Chain of Bottlenecks

Let’s follow a typical I/O operation from the disk to the CPU core and count the I/O channels:

1. A read head senses the state of a bit of magnetic material on the surface of a disk
2. The head transmits this signal to a buffer on the disk controller board
3. The data is picked up by the disk controller CPU and transmitted over a SATA or SAS connection
4. The storage array or RAID controller receives the data and moves it over an internal bus to another buffer or cache
5. The data is picked up by another CPU in the array controller and sent out another interface using Fibre Channel or Ethernet
6. The data is buffered and retransmitted by one or more switches in the storage network
7. The host bus adapter (HBA) on the server side receives the data and buffers it again before sending it over a local PCI Express bus to system memory
8. The server memory controller pulls the data out of system memory and sends it via a local bus to the CPU core

There are actually many more steps than this, but the picture should be clear by now. There are many, many I/O channels to consider when it comes to storage, and the drive interface is just one potential bottleneck.

We constantly move bottlenecks around - as one link is improved, another choke-point appears

Optimizing Storage I/O

Tactical steps to improve storage performance typically focus at one link in the chain: Drive vendors move from 1.5 Gb to 3 Gb SATA, or SAN buyers upgrade from 4 Gb to 8 Gb Fibre Channel. But the basic architecture of enterprise storage has remained constant for over a decade, and the reliance on block SCSI commands endures. This is all about to change.

One critical bit of I/O optimization exists at the point of connection between the various chipsets inside the server. AMD pulled the memory controller off of the “northbridge” with their Athlon line. Intel did the same with their Nehalem and is eliminating the northbridge entirely with the Lynnfield/Jasper Forest CPU lines. This gives serious bandwidth to the crucial PCI Express-to-CPU-core link, moving the bottleneck downstream.

We are in the midst of a massive upgrade of the storage network as well. Between 8 Gb Fibre Channel and iSCSI and Fibre Channel over 10 Gb Ethernet, not to mention persistent interest in InfiniBand, storage network throughput is rapidly expanding. As with the internal PC connections, the expansion of network bandwidth has pushed the bottleneck to the storage array interface for the time being.

Microsoft and Intel recently pushed over a gigabyte per second over 10 GbE using iSCSI, but they needed multiple storage targets to feed that connection. It isn’t that modern storage systems couldn’t push that kind of I/O (indeed, arrays are tens to hundreds of times faster internally thanks to their spindles and cache), but that the conventional storage protocols are tightly linked to a single “front-end” interface. The current state of the art for storage array design is moving to distributed models, exemplified by pNFS and scale-out NAS concepts like MaxiScale (now acquired by Overland).

Once the array interfaces can pump out massive I/O, attention will turn once again to the disk interfaces themselves. Although 6 Gb/s SAS and SATA is now a reality, this interface is inappropriate for future high-performance SSDs. Arrays designed around flash or DRAM are likely to switch to PCI Express as their internal connection of choice for performance and to optimize data placement on these new devices. Companies like Nimbus and NetApp are already moving in this direction.

Time To Get Smart

Hard disk drive spindles make up the bulk of storage capacity, but small amounts of cache make them far more effective. But both of these horsemen must operate within the constraints of the I/O channels they pass through. This brings us to the final horseman of performance: Smarts. Clever designers have created clever controlling mechanisms to overcome the limits of spindles, cache, and I/O channels.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | The Four Horsemen of Storage System Performance: I/O As a Chain of Bottlenecks
This post was categorized as Computer History, Enterprise storage, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although it's fun to speculate about blockbuster purchases, smaller acquisitions make much more sense

The industry is buzzing with rumors that Isilon will be the next big storage acquisition now that 3PAR is safely in HP’s hands. The alleged buyer, EMC, is itself surrounded by rumors of becoming a shared feast for Oracle and Cisco. But there are much more accessible and tasty items on the enterprise IT acquisition menu, and the lower cost of entry makes them much more likely to come about!

Buying Talent, Technology, and Market Share

There are three good reasons companies look to buy others:

1. They lack the talent to play in a certain market. Although engineers are often touted as the prize in tech acquisitions, other kinds of talent (sales, marketing, and leadership) often make more of the difference. But buying talent is perilous because (especially in California), talent can walk out the door.
2. The next gap to be filled is technology itself. Startups are often more innovative and creative, filling new product niches and taking risks on new concepts. This manifests itself as product, of course, but IP and roadmap are equally important.
3. There is also market share to consider. Companies love to crow about their sales leadership, and very little matches the messaging that comes from overtaking a competitor in sales. Buying up a smaller, but not insignificant, competitor is a great way to achieve this!

There are a few more reasons companies get bought, of course. Companies sometimes build up vast cash reserves, making them ripe for a buyout. Others bring together assets that are more valuable split apart and sold off than together. Still other acquisitions are made to keep a competitor from having a key component of the market.

There are many great companies in enterprise IT, often packed to the gills with talent, technology, and market share. NetApp, EMC, Juniper, Symantec, and so many others are loaded with value, yet not all are about to be acquired.

What’s Stopping You?

It’s not easy for a company to buy another. The most-obvious roadblock is financial: As I discussed regarding Oracle, HP, and NetApp, companies can use cash, debt, or stock to make the purchase. Cash is clean and tidy, but each has its own trade-offs. At the end of the day, the purchase price can be too big a hurdle to overcome.

NetApp and Juniper have market caps around $17 billion, making them a very expensive purchase. Symantec is a little smaller at $12 billion, but it’s still large enough to be out of play for many companies. And EMC, at $42 billion, would be a historically-large acquisition, probably requiring more than one company at the table.

Although far smaller, Isilon begins to look expensive once one looks at the prospects for return on investment. Currently priced just under $2 billion, Isilon has solid clustered NAS technology and is a major player in high-performance computing. But they are not even ranked in the top external storage vendors according to IDC, and EMC and NetApp rule the mainstream NAS market. Their annual revenue grew respectably from $61 million in 2006 to $120 million in 2009, but that’s small-potatoes in enterprise IT terms. And with just $79 million in cash and short-term investments, they won’t be bought for plunder either.

Ripe Fruit

Isilon is just too small to justify a multi-billion dollar acquisition, but there are other tastier possibilities in play. Three in particular come to mind:

1. FalconStor has excellent technology, proven over a decade and used by thousands of customers. They have excellent OEM connections, too. And their recent CEO ousting makes them a killer bargain at under $150 million. Seriously – they could make that much revenue next year given a solid sales re-launch. at the very least, their stock (NASDAQ:FALC) is a solid buy!
2. Overland (NASDAQ:OVRL) is in the midst of a re-birth, but their purchases of Snap and MaxiScale make them an acquisition target in my mind. Their astonishingly-small market cap of just under $17 million is less than their quarterly revenue. Yes, they’re losing money, but this would change quickly in the hands of a strong benefactor.
3. Finally there’s big-data protection specialist, Sepaton. It’s hard to say exactly what a purchase would entail, since Sepaton isn’t a public company, but it’s probably safe to assume that the price would be small and the acquired debt somewhat larger. But their technology is just waiting to be blasted prime-time by some big-data gorilla.

See the common theme here? Great technology, huge potential, and a miniscule price. Although none of these would be much of a market share win initially, all could be serious threats to the established players with some sales effort. Any of the big equipment companies (EMC, Dell, IBM, Oracle, or Cisco) would make hay with Overland, and any big software player (Oracle, CA, IBM, HP, or Symantec) would find much to love in FalconStor or Sepaton.

Stephen’s Stance

Although multi-billion dollar blockbuster deals are more fun to speculate about, smaller technology purchases like these can have much greater leverage. I expect at least one of these companies to be snapped up in the next year, and wouldn’t be surprised to see all three gone before Isilon, EMC, or Symantec. Indeed, I just don’t see the business case for an Isilon acquisition today.

Image credit: Cash Register 70 by zizzybaloobah

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Why FalconStor, Overland, and Sepaton Ought To Be Acquired Before Isilon
This post was categorized as Enterprise storage, Gestalt IT, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Apparently, some were annoyed, offended, or hurt by this post. For that, I sincerely apologize. My intent was to reach out to my friends at the company in this difficult time, but what I meant doesn’t help those who are directly affected.

I have only one question: What is the right thing to do when bad things happen? Should one offer help or remain silent? This is not a rhetorical question – I’d like some advice.

Although nothing official has yet been announced, enterprise storage giant EMC is apparently conducting a massive workforce reduction today. Update: EMC reported in January that they would take a restructuring charge in Q4 2008 and would reduce overall headcount by 2400. The job losses reported here were part of this restructuring. I hear that there is a voluntary early retirement program, too.  Multiple sources have confirmed to me that perhaps as much at 10% of the folks from Hopkinton will be let go. Update 2: The Boston Globe reports EMC’s claims that only “a small number of workers” has been let go. The Register reports that the numbers are “small and modest.” I personally know of a handful who were affected, and they claimed widely divergent losses in their departments, so I have no idea what the number was.

Update: I’m hearing conflicting reports from inside the company, with some absolutely confirming the RIF and others scratching their heads. Might be divisional rather than company-wide, but a good number of people both in Hopkinton and elsewhere are apparently losing their jobs. A number of others are confirmed to have been let go over the past three weeks.

This follows last week’s news that Quantum would also let 180 employees go, and Pillar and Overland also reduced their respective workforces this quarter.

There are also rumors that one of my favorite storage resources (which will remain nameless for now) will not last into 2009.

I’m really speechless about all of this. There are many, many fine people involved at these companies, and I wish all of them the best.

If you are affected, please drop me a line. I can’t offer you a job, but I promise to do everything I can to help you land on your feet.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | EMC Cuts Staff as Recession Continues
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.