• My State of the Art Thin Provisioning series
  • Storage is Not Getting Cheaper
  • Thin Provisioning: Attacking Storage Utilization
  • De-Allocating is the Core Issue for Thin Provisioning
  • Thin Provisioning: Playing the Telephone Game
  • Monitoring Filesystem Metadata For Thin Provisioning
  • Zero Page Reclaim: Savior of Thin Provisioning?
  • What is WRITE_SAME? Green Eggs and Ham!
  • The Bridge: Veritas Thin (Provisioning) API
  • Is TRIM Useful For Thin Provisioning?
• The Raising the Floor podcast series
  • Raising the Floor 1: Cloud Storage Gateways
  • Talking Cloud Storage Gateways With Nasuni and Cirtas
  • Raising the Floor: How the Cloud Impacts Enterprise IT
• Tech Field Day links
  • Data Robotics Is First Three-Time Tech Field Day Presenter
  • The Nerd’s Going to Have a Field Day! | The Networking Nerd
• And more links!
  • I wrote this piece for SearchStorage: Is PCIe SSD right for you? Deploying PCI Express SSD devices
  • I was interviewed for another thin story: Thin provisioning driven by technology advances, best practices
  • My friend Erin Collopy from HP is starting a blog: A new year, a fresh slate « Social Swimmer
  • Bob Plankers always impresses with his technical writing: How To Create and Measure NTFS & ext3 Disk Fragmentation
  • I loved this idea from Freedom to Tinker: Monitoring all the electrical and hydraulic appliances in your house
  • Chris Mellor spills the beans on EMC’s big announcement next week: Storage Virtualization, Unified and the VNX
  • Chris also posted a great piece about InPhase: How I watched a holographic storage company implode
  • Then there’s this Winextra bit on Microsoft’s Biggest Mistake of 2010
  • A new EMCer blog, Storage Savvy, has a nice pNFS piece: Why pNFS can be a big deal even if NFS4.1 isn’t…
  • Martin Glassborow is always worth reading: Reading the Ruins
  • Floyd Strimling makes a go at prognostication: Fun Alert: Insane 2011 Predictions That May Come True
  • Another great Bill Hill post on cloud: Cloud Crossroads
  • And Ian Foster isn’t happy with Amazon: Amazon – Masters of marketing or perfecting the art of customer dissatisfaction?

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, January 7, 2011
This post was categorized as Computer History, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• Enterprise technology
  • VMware Info: rant : Why do I have to log into your Knowledgebase?
  • Customers embracing Vblocks, but VCE sales harmony a work in progress
  • An Introduction to the Force10 S25N
  • Predictions For Storage Networking In 2011
  • Inter-Fabric Traffic in UCS
  • Thoughts on pNFS
  • Has HP Got the ‘X Factor’?
  • Microsoft Looking Like An End-Stage Company
  • Internet – Not to Scale
• Consumer technology
  • Protip: Use Safari Reader to Send Multi-Page Articles to Instapaper
  • The 4G forgery
  • Apple’s Next Macintosh OS
• My posts
  • Microsoft Office 2011 for Mac Is (Finally) Here!
  • How To Create a Data DVD Using Only the Mac OS X Finder
  • Commodity Hardware Always Wins
  • 4G: Is It Really A Standard If No One Cares?
  • The Tablet Wasteland and the iPad Oasis
  • LaCie Releases USB 3.0 Driver For Mac OS X (But It Only Works With LaCie Drives)

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Back From the Pile: Interesting Links, November 5, 2010
This post was categorized as Apple, Enterprise storage, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Four Horsemen of Storage System Performance: These four ugly gentlemen stand between you and your data.

Why do some data storage solutions perform better than others? What tradeoffs are made for economy and how do they affect the system as a whole? These questions can be puzzling, but there are core truths that are difficult to avoid. Mechanical disk drives can only move a certain amount of data. RAM caching can improve performance, but only until it runs out. I/O channels can be overwhelmed with data. And above all, a system must be smart to maximize the potential of these components. These are the four horsemen of storage system performance, and they cannot be denied.

The Chain of Command

It is tempting to think of storage as a game of hard disk drives, and consider only The Rule of Spindles. But RAM cache can compensate for the mechanical limitations of hard disk drives, and Moore’s Law continues to allow for ever-greater RAM-based storage, including cache, DRAM, and flash. But storage does not exist in a vacuum. All that data must go somewhere, and this is the job of the I/O channel.

To be useful, storage capacity must connect to some sort of endpoint. This could be the CPU in a personal computer or an embedded processor in an industrial device. Indeed, there are endpoints and I/O channels throughout modern systems, with potential bottlenecks, caches, and smarts at each point. “Storage people” like me tend to think too small – imagining that the I/O channel ends at the disk drive, the “front end” of the array, or the storage network. But data must travel further, all the way to its final useful point in the core of the CPU.

Once we consider I/O as a long chain of interconnected endpoints, we begin to see the fact that I/O constraints at any point can strangle overall system performance. This is not merely an academic exercise: Optimizing the I/O channel is a consuming passion for most practitioners of enterprise IT, including architects, engineers, and system developers. And, like a good game of Whack-a-Mole, increasing the speed of one link causes another chokepoint to rear its head.

Parallel and Serial I/O

Imagine you had a warehouse full of boxes to move across the country as fast as possible. There are a few options available to you:

1. A fast truck can zip back and forth with just a few boxes
2. A train is slower, but its many cars can haul a huge quantity

But there are realistic limits to both capacity and speed: The train has to fit on the tracks, and the truck can’t move at the speed of light. Plus, one must consider the time taken to load and unload the chosen vehicle.

We continually shift between parallel and serial I/O paradigms

The same trade-offs are true of computer busses: Serial channels can be optimized to zip individual bits back and forth, or parallel busses can be designed to carry whole bytes (or more) at a time. The simplicity of serial communications is tempting, but designers continue to resort to parallelization for added throughput.

Note: Most serial protocols actually feature two links, making them “full duplex”: One for transmit and another for receive.

Serial storage interconnects are dominant currently, with fraternal twins SAS and SATA coming to dominate the disk interface landscape. SAS and SATA share the same 1.5, 3, and now 6 gigabit per second serial physical interconnect, offering more than enough throughput for conventional hard disk drives and edging out older serial (Fibre Channel, SSA) and parallel (ATA and SCSI) alternatives.

Networks (Ethernet, Fibre Channel, and InfiniBand) are predominately serial as well, as are lower-end interconnects like USB and FireWire. Serial communication also dominates in the system bus world, with serial PCI Express toppling parallel PCI.

But parallel variants are often offered for increased throughput: Multi-lane PCI Express and bonded multi-link InfiniBand make up a fair portion of the installed base, while load balancing MPIO drivers are common in Fibre Channel storage. And let’s not forget that the “X4″ variants of Ethernet use multiple bonded links as well.

The Definition of Bottle Neck

Most English speakers have encountered the French term, “cul de sac”, meaning “bottom of the bag” or dead end. But hard disk drives have plenty of “bottom end”, or storage capacity. When it comes to disks, the issue is usually at the neck of the bag: Data just can’t be pulled out of a hard disk drive fast enough.

Emptying a barrel of wine through a spigot takes hours, but pry the end off and the floor is covered in a moment!

The density of modern hard disk drives (the capacity of our barrel) has been growing much more rapidly than the I/O channels serving them (the spigot). Where once a hard disk drive could be filled or emptied in an hour or two, modern drives take days or weeks!

I once called this “flush time“, but I think the wine metaphor is much more appetizing!

This “bottle neck” has serious implications beyond basic storage performance. Data protection is impacted, since ever-larger storage systems can no longer be backed up by dumping their content; system reliability is reduced, since week-long RAID rebuilds increase the risk of multiple drive failures; and cost containment efforts are also impacted, since adding spindles drives up prices.

Nowhere is this bottleneck more evident than in portable devices. Modern drives (like the 1 TB Seagate USB drive I recently reviewed) have massive capacity and pathetic performance. The USB 2.0 interface just can’t keep up, and this creates a limit to the expansion of capacity. It would take half a day to fill that drive under perfect conditions at 25 MB/s, reducing its value as a massive data movement peripheral. The emerging USB 3.0 standard promises to alleviate this performance issue for now, as illustrated with Iomega’s new external SSD.

Cache and solid state storage can help, but they have their own bottlenecks. Storage arrays typically use Fibre Channel or SAS SSDs, and their front-end interface remains the same. The best-performing SSDs use the PCI Express bus directly rather than emulating hard disk drives over SCSI interfaces. And even PCI Express might not be enough to handle the massive I/O of NAND flash or DRAM. In each case, the bottleneck moves down the chain.

A Chain of Bottlenecks

Let’s follow a typical I/O operation from the disk to the CPU core and count the I/O channels:

1. A read head senses the state of a bit of magnetic material on the surface of a disk
2. The head transmits this signal to a buffer on the disk controller board
3. The data is picked up by the disk controller CPU and transmitted over a SATA or SAS connection
4. The storage array or RAID controller receives the data and moves it over an internal bus to another buffer or cache
5. The data is picked up by another CPU in the array controller and sent out another interface using Fibre Channel or Ethernet
6. The data is buffered and retransmitted by one or more switches in the storage network
7. The host bus adapter (HBA) on the server side receives the data and buffers it again before sending it over a local PCI Express bus to system memory
8. The server memory controller pulls the data out of system memory and sends it via a local bus to the CPU core

There are actually many more steps than this, but the picture should be clear by now. There are many, many I/O channels to consider when it comes to storage, and the drive interface is just one potential bottleneck.

We constantly move bottlenecks around - as one link is improved, another choke-point appears

Optimizing Storage I/O

Tactical steps to improve storage performance typically focus at one link in the chain: Drive vendors move from 1.5 Gb to 3 Gb SATA, or SAN buyers upgrade from 4 Gb to 8 Gb Fibre Channel. But the basic architecture of enterprise storage has remained constant for over a decade, and the reliance on block SCSI commands endures. This is all about to change.

One critical bit of I/O optimization exists at the point of connection between the various chipsets inside the server. AMD pulled the memory controller off of the “northbridge” with their Athlon line. Intel did the same with their Nehalem and is eliminating the northbridge entirely with the Lynnfield/Jasper Forest CPU lines. This gives serious bandwidth to the crucial PCI Express-to-CPU-core link, moving the bottleneck downstream.

We are in the midst of a massive upgrade of the storage network as well. Between 8 Gb Fibre Channel and iSCSI and Fibre Channel over 10 Gb Ethernet, not to mention persistent interest in InfiniBand, storage network throughput is rapidly expanding. As with the internal PC connections, the expansion of network bandwidth has pushed the bottleneck to the storage array interface for the time being.

Microsoft and Intel recently pushed over a gigabyte per second over 10 GbE using iSCSI, but they needed multiple storage targets to feed that connection. It isn’t that modern storage systems couldn’t push that kind of I/O (indeed, arrays are tens to hundreds of times faster internally thanks to their spindles and cache), but that the conventional storage protocols are tightly linked to a single “front-end” interface. The current state of the art for storage array design is moving to distributed models, exemplified by pNFS and scale-out NAS concepts like MaxiScale (now acquired by Overland).

Once the array interfaces can pump out massive I/O, attention will turn once again to the disk interfaces themselves. Although 6 Gb/s SAS and SATA is now a reality, this interface is inappropriate for future high-performance SSDs. Arrays designed around flash or DRAM are likely to switch to PCI Express as their internal connection of choice for performance and to optimize data placement on these new devices. Companies like Nimbus and NetApp are already moving in this direction.

Time To Get Smart

Hard disk drive spindles make up the bulk of storage capacity, but small amounts of cache make them far more effective. But both of these horsemen must operate within the constraints of the I/O channels they pass through. This brings us to the final horseman of performance: Smarts. Clever designers have created clever controlling mechanisms to overcome the limits of spindles, cache, and I/O channels.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | The Four Horsemen of Storage System Performance: I/O As a Chain of Bottlenecks
This post was categorized as Computer History, Enterprise storage, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This week’s highlights included lots about NFSv4 and IBM’s new Storwize V7000. Some great networking content, too!

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

• NFSv4: Part 1 COMPOUND RPCs
• A Comparison of Current Spanning-Tree Elimination Strategies
• Fail Factors–Why Startups Die: Negative Inertia and the Anti-”Why?” Culture
• Bad Analogies and Avoiding Vendor Lock-in
• IPv4 Exhaustion: What About Class E Addresses?
• Wither XIV?
• Why are flash prices dropping?
• Traveling East-West Might Get A Little Easier: Highlights from the TRILL RFC5556
• VAAI – Not Just for Virtualisation?
• Manage Data Not Storage
• POLL: Is A Transcontinental OSS Wifi Network Even Possible?
• Death Defying….
• 2010-42-3 Its the little touches – Storwize V7000
• IBM Storwize Product Name Decoder Ring
• Storagebod’s Blog: Wize Up
• Followup to EMC AX4 SAN Trouble

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Back From the Pile: Interesting Links, October 17, 2010
This post was categorized as Enterprise storage, Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Overland's SnapServer will soon scale out to hundreds of nodes, thanks to MaxiScale's technology

Overland Storage is showing intriguing signs of life. Once relegated to OEM tape library duty, Overland received an injection of cash and (more importantly) talent this year. Now the company is stepping up the technology behind their SnapServer NAS array by acquiring scale-out file storage company, MaxiScale. They intend to bring the scalable capacity and performance normally associated with enterprise and high-performance computing systems to the mass market.

Overland In A Snap

Snap Appliance and Overland Storage are familiar names in the enterprise storage space, but the two companies have had a convoluted history. Snap was born in 1998 as a product of Meridian Data and acquired by backup giant Quantum one year later for about $85 million. Although successful, the Snap NAS products were not at home among Quantum’s tape-heavy product lines. The division was re-purchased in 2002 as Snap Appliance by private investors, led by current CEO Eric Kelly, for just $11 million. Two years later Snap was again absorbed by a familiar storage name: This time $100 million brought it into Adaptec.

Quantum competitor Overland Storage re-purchased Snap from Adaptec in 2008 for just $3.6 million and brought Kelly back as CEO in 2009. Kelly brought back former Snap (and Data Robotics) marketing guru Jillian Mansolf this year, and they were soon joined by Dr. Geoff Barrall, founder of Data Robotics and BlueArc. Although a public company, Overland (NASDAQ:OVRL) also raised an additional $12 million in private equity financing, an unknown amount of which went to purchase MaxiScale.

You may also want to read The Rebirth of Overland Storage

This brings us to today, with Overland Storage on the rise, with strong OEM and channel sales in tape and disk storage devices. Overland’s product lines include the NEO tape libraries, REO virtual tape devices, SnapSAN block arrays and various SnapServer unified and NAS systems.

The highlight of Overland’s storage offerings is the SnapServer N2000, a unified iSCSI and NAS array solidly placed in the entry- to mid-market. Like many competitors, Overland relies on a single- or dual-controller storage “head” based on industry-standard CPU technology and scales with additional disk “shelves”. Unusually, Overland’s N2000 line currently relies on a single SAS connection daisy-chained to its E2000 expansion units. One hopes dual connections and faster connectivity are coming soon.

MaxiScale: Scale-Out NAS

Venture-funded startup MaxiScale developed a scale-out NAS capability, allowing multiple NAS servers to be consolidated into a unified namespace without the usual bottlenecks of clustered solutions. The company headed for the cloud storage market, boasting massive scalability and management benefits.

MaxiScale’s technology differentiator was in their rejection of conventional clustering and scale-out approaches. Many scale-out solutions rely on shared-nothing coordinated nodes, but it is often difficult to scale and manage these solutions. This has caused much effort to build cluster coordination through metadata, exemplified by SAN filesystems (including EMC’s MPFS and Quantum’s StorNext) as well as clustered NAS solutions like pNFS.

This MaxiScale illustration shows their distribution of metadata and content as well as their unified namespace

MaxiScale’s “FLEX” technology takes an entirely different approach. It distributes “peer sets” of data across multiple nodes like pNFS but also distributes metadata rather than relying on a single central node. Combining this with a unified namespace, MaxiScale claims to scale to thousands of nodes with linear performance. The company had targeted cloud service providers interested in massive scale using commodity hardware.

Overland + MaxiScale = SnapScale?

Overland appears to be bringing MaxiScale back down to earth, leveraging their technology to combine a set of in-datacenter SnapServers into a unified, high-performance cluster. Although a number of other companies claim scale-out capabilities (Isilon and HP X9000 spring to mind), this is a unique capability in the SMB and midrange NAS market, and Overland will surely make hay as a low-priced alternative.

The company could quickly implement MaxiScale’s software on their existing industry-standard hardware and push a line of “SnapScale” appliances. But things will get more interesting once they begin integrating the technology into their GuardianOS software. Imagine a scale-out solution that enables a company to grow from a single sub-$10,000 NAS to a vast cluster of thousands of nodes with linear performance gains. Even being able to add a half-dozen NAS heads would be a welcome alternative to Dell, HP, and NetApp offerings. In the longer term, Overland could use this technology to break from the head-and-expansion architecture they currently employ, shifting to a cloud-like distributed architecture.

Stephen’s Stance

Overland is making all the right moves. Eric Kelly should be commended for taking bold action to re-energize the company with an infusion of money, talent, and technology. The MaxiScale technology is impressive, and adding it to Snap for a mid-market offering is potentially game-changing.

There are lots of SMB and midrange NAS systems, and most are fairly similar in terms of technology. Overland fell in that group as well until now, but the addition of MaxiScale is a real differentiator for the company. Mansolf and Barrall have proven that they can take out-of-the-ordinary technology and market it to the masses, and this acquisition gives Overland another chance to execute. I will be watching their progress with great interest!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Overland Snaps Up MaxiScale to Scale Up Snap
This post was categorized as Computer History, Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Do we really need parallel NFS?

Did some pNFS proponent slip a love potion into the coffee at EMC? Suddenly it’s pNFS time at the company known for its reluctance to embrace file sharing and filesystems in general. The purple prose is flying, with Chad Sakac declaring himself “a big fan of the application of NFS” and Chuck Hollis extolling the “inherent simplicity and ease-of-management of NFS.” The NetApp guys must be amused by the bear hug from Hopkinton, but many are seeing deja-vu all over again.

Chad’s Icky Bits

(Apologies for that heading, but those are Chad’s words, not mine)

Chad Sakac’s red rose for pNFS included a few thorns aimed at good old NFSv3. He calls these the “icky bits” and spills some ink over them:

1. “NFS Server failure behavior,” says Chad, leads to issues as serious as “a guest OS crash” and administrators “resorting to unnatural acts” to compensate. He talks about EMC’s DART OS being optimized to fail over in under a minute to avoid application issues and the difficulty in actually accomplishing this feat.
2. Chad also points out that “NFS client limitations” can lead to “unexpected bottlenecks.” Load balancing large workloads across multiple gigabit Ethernet NICs means hand-tuning, since NFS pins traffic to a single MAC address.

Certainly these limitations were known to many in the storage industry, but haven’t they also been addressed repeatedly? NetApp, EMC, and BlueArc do indeed suggest adjusting NFS heartbeat values to allow time for the cluster to recover, but this seems more a limitation of their clustered server architecture than of NFS itself. Scale-out NFS servers from Isilon and HP don’t seem to require these “unnatural acts.”

As for client limitations, manually balancing client loads is a reality in many large storage architectures, not just NFS. Perhaps the fact that NFS can handle so many more I/O requests in a given timeslice makes this more of an issue, but it tends to be transient.

Chad has repeatedly expressed his love for NFS, especially as a datastore for VMware. Clearly, he intended to point out these “icky bits” to highlight the possibilities for pNFS. But the method used (calling them “icky” for one) resembles mud slinging.

Chuck Wants pNFS

(Chuck’s titles also lend themselves to mis-reading)

Chuck Hollis is more careful in his wording, extolling the virtues of pNFS without calling anything “icky”. Indeed, there’s just one NetApp dig: He says their “emulated containers of LUNs” are “hardly optimized”, which is a welcome change of tone from previous debates.

But the underlying message is the same: pNFS is new and wonderful, encouraging proliferation of hand-holding, flower distribution, and rainbows. Again I ask, is this really true? Is pNFS ready for this kind of adulation when, as Chuck points out, “it’s going to take a while before the rest of the portfolio, industry and ecosystem catches up.  Maybe a year or so.”

Seriously? A year until pNFS is ready for mass enterprise adoption? Admittedly, EMC has been working on pNFS (as MPFS) for a long time, but predictions of “just another year” for a major protocol transition set off warning bells. This is doubly true when most clients (including VMware) don’t yet offer even basic support.

Stephen’s Stance

One wonders if airing this dirty laundry is an attempt to highlight EMC’s pNFS work or to discredit plain old NFS as a datacenter protocol. As I wrote about in Our New Thing Is Awesome (‘Cause Our Old Thing Sucked), the “parade of progress” sometimes degenerates into “out with the old,” and this is perilous for purveyors of durable goods like storage systems.

I am also very concerned with the proliferation of “layout types” within pNFS. It seems that every vendor has a hand in the protocol, and each is adding their own technology to the mix. We started with files and now have both objects and blocks. Will these be widely supported? Do we really need them? Or will pNFS start looking like Bluetooth: Bloated, incompletely-implemented, and ignored except for special use cases.

But my motivation behind this post is simpler than that. I would like to pose a question: Is NFS (v3) really that “icky”? Do we really need pNFS? Or have these problems been solved previously?

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Is NFS v3 Really That Bad?
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

He talks about how parallel NFS (pNFS) is set to trickle down to the enterprise from the high-performance compute labs. It’s always amusing to me to think of things trickling down to the enterprise storage market, but in this case he’s right – massive clusters (and Panasas for that matter) have yet to make much of a mark on the enterprise computing world.

He goes on to talk about how escalating disk capacity has lead to unacceptable rebuild times in RAID sets. You tell ‘em, Garth! Apparently, he’s been beating the multiple-parity drum since 1989 (!?!) – I hadn’t realized that RAID 6 was that old, since it’s not in the paper, but he claims they invented it, too, way back when. Who am I to contradict the Bob Metcalfe of storage?

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Garth Gibson: Still Relevant After All These Years
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.