This is how we used to avoid hotspots in 1998: Carefully planning every detail of the storage layout.

Each disk drive consists of a spindle of spinning platters with read/write heads move back and forth. Each time you access a piece of data that’s not in cache, the drive moves its arm over the platter to access the correct piece of data. Since each drive can only access one piece of data at once, and since caches can only hold so much data, tuning a system to minimize the number of requests per drive is essential.

Manual storage array layout was an art, but we never fooled ourselves into thinking our designs were optimal. There were just too many intractable problems, so we had to compromise at every turn:

• We usually had no performance data to base our layout decisions on, so we had to rely on guesses and rules of thumb
• Workloads tend to change over time and manual layouts are painful to modify
• The smallest unit of allocation was an entire LUN or drive, so even the best disk layout mixed hot and rarely-accessed data everywhere
• Much of the allocated space was unused, so we used expensive disks to store nothing

One might think that, 10 years later, advances in technology would have solved these basic issues. But for many people using many of the so-called modern mainstream enterprise storage systems, these problems remain.

Like all good systems administrators, I’m a natural control freak. I am uncomfortable letting the system manage itself, having been burned too many times by computers (well, software really) making stupid decisions. It’s analogous to the backlash against anti-lock brakes, traction control, and automated transmissions among racing enthusiasts.

Do we allow technology to help us get better performance, or do we try to micro-manage everything? Photo by ClearInnerVision

But the time has come to let go. We don’t have to micro-manage storage anymore, and we have much to gain by letting the array do the work:

• Just as traction control can manage each wheel independently, something a driver could never do, modern virtualized storage systems can allocate small “chunks” to the optimal drive type, creating a better layout than anyone could manage with LUNs
• Dynamic optimization technology can move these chunks around, adapting as loads change
• Thin provisioning can go a step further, not wasting drive capacity for unused space
• Wide striping and post-RAID storage systems have a higher threshold before performance suffers due to spindle hotspots
• Widespread availability of tiered storage, including advanced caches, solid state drives, high-performance SAS and FC, and cheap bulk disks, gives us many more options

As I mentioned, not all systems have these capabilities, and not all implementations are created equal. I’m concerned about misuse of thin provisioning, for example, but it’s hard to argue with its effectiveness in many circumstances. Find out how granular your system’s allocation is – some remain LUN-only, while others are much more effective, using tiny chunks.

These new storage automation technologies really become essential once high-dollar flash storage is added to the mix. If you’re paying 30 times more for a flash drive, you want to make sure you’re making the best use of it that you can! Look at IBM’s recently-announced SAN Volume Controller (SVC) and solid state drive (SSD) combination, for example: It will almost certainly have fine-grained thin provisioning of SSDs, and should be able to dynamically move data between flash and disk storage and even between different storage arrays, but I still have questions on how granular this capability will be. HDS can do similar things with their USP-V. NetApp’s V-Series NAS systems will do dynamic allocation, thin provisioning, and data deduplication to enable a better return on the flash drive investment. I’d love to see 3PAR, Compellent, Dell/EqualLogic, and HP/LeftHand apply their solid dynamic allocation tech to solid state storage as well!

Then there’s the 800 lb gorilla: EMC. More enterprise SSD has probably been shipped out of Hopkinton than every other vendor combined, and both the CX and DMX support (optional/expensive) “virtual provisioning” (aka, thin provisioning) of flash storage. But EMC’s Optimizer is not widely used, and only migrates entire LUNs based on user input – hardly the kind of dynamic and granular technology needed to optimally use all of that flash storage. I’m sure the company is working on addressing this issue, though. Perhaps it will appear in the DMX-5 announcement we are all expecting this year?

This article can also be found on Gestalt IT: How I Learned to Stop Worrying and Love Storage Automation

© sfoskett for Stephen Foskett, Pack Rat, 2009. | How I Learned to Stop Worrying and Love Storage Automation
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Frankenweenie saves young Victor in Tim Burton's macabre short film

When is a solution integrated and when is it a Frankenstein-like mashup of tangled tech? Apparently, that line is crossed when it’s your competitor’s offering…

In my time in the storage industry, I’ve seen enough franken-storage come and go to make me skeptical whenever a new “integrated” solution is announced. But a lot of this stuff works just fine, so I also know that integrated solutions aren’t always bad!

The latest industry blog flame war centers around NetApp’s recently-announced solid state storage solution, which pairs a V-Series NAS head and a Texas Memory Systems RamSan-500 flash storage system. Perhaps NetApp’s Val Bercovici did get a bit over-excited in his post on the topic, but he wasn’t just talking about the RamSan: He was laying out how NetApp’s WAFL technology can work in an SSD world, and using some recent performance test numbers on that solution as well as their PAM cache cards as an illustration of this.

The next thing you know, we have EMC’s Storagezilla and IBM’s Barry Whyte calling the company out for what they (and others. like Storagebod) see as an underwhelming product offering. That’s all well and good, and I’ll let the reader decide if NetApp’s moves warranted a press release, but now things have gotten uglier…

EMC’s Chuck Hollis called the whole RamSan idea to account, saying it was “Frankenstorage“, causing NetApp’s Alex MacDonald to engage in a little “I know you are but what am I” in reference to EMC’s CLARiiON/Celerra “unified storage” solutions.

It’s time to bring some sanity to this whole integrated solution concept. Every product in the storage world is an amalgamation of OEM parts to one extent or another, and there are always integration issues. Certainly many of EMC’s offerings could be the subject of name-calling: They use STEC SSD drives in the DMX, they use Quantum deduplication engines in their CDLs, and their Celerra NS platform does include a complete Fibre Channel SAN behind the curtain. But they’re not alone, and not even wrong in doing this: Every vendor relies on OEMs, and as a wise man said, “working with an OEM gives you the flexibility to pick best of breed technologies” and that’s exactly what customers want. Any objective person would welcome qualification and integration of TMS’ RamSan with a solid platform like the NetApp V-Series – it’s a certifiable win for the customer. Just like they would be happy to see EMC leveraging great technology from Quantum and STEC.

Chuck goes on to point out some downsides to these OEM combinations, and they’re certainly fair criticisms:

• When you’re buying this from this guy and that from that guy, it’s bound to cost more because everyone needs their cut.
• Since all attempts at unified heterogeneous device management have failed, a combo is certainly harder to manage than a single device.
• With multiple vendors in the mix, fingerpointing is common once support is needed.

But these criticisms can be mitigated by the vendors themselves. They can give up some margin in order to gain market share. They can create unified management interfaces for the combinations they sell and support. And they can really support what they sell, refusing to give in to the temptation to say “not my problem” when the going gets rough. And companies deal with these problems all the time! Frankenstorage doesn’t have to be so scary…

This post can also be found on Gestalt IT: The Difference Between “Integration” and “Frankenstein”

© sfoskett for Stephen Foskett, Pack Rat, 2009. | The Difference Between “Integration” and “Frankenstein”
This post was categorized as Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The tech industry has been buzzing about solid state drives (SSDs) again lately, but many questions remain. Even after many major vendors (Apple, EMC, and Dell to name a few) have introduced NAND flash-based disk into their core products, it is unclear whether non-disk storage will fly or flop. I’m betting it will find a nice niche, but that traditional spinning disks are here for a good long time.

Apple’s Flashing Success

When Apple switched from hard disks to flash in their mainstream product line, the world was abuzz with the novelty: Would flash displace hard drives? Sure, the company still offered disk-based storage for those needing vast capacity, but most people found that 8 GB or so of storage was plenty for daily use. Of course, instead of the MacBook Air, I’m talking about the iPod family, which contains just a single disk-based model.

Like the Air, the iPod demonstrates that what matters in the “take it with you” market is portability in the form of low weight, perceived durability, and compact dimensions. And NAND flash excels when it comes to packaging. The flash-based iPod is an excellent semaphore for this market segment in other ways, too. Audio files are fairly small, so music users don’t need all that much storage, relatively speaking. They will gladly ignore the cost per GB, too, at such small capacity points: iPod Nano buyers pay ten times more per GB than iPod Classic buyers.

In the case of the iPod, the compact size and joggable durability afforded by the flash iPods is worth the money to most buyers, not that flash player has sufficient capacity to meet their needs. The MacBook Air teaches a slightly different lesson: Although reviewers are quick to point out that the speed and battery life difference between the hard disk and NAND flash versions of the mini notebook are negligible, early buyers were happy to pay $1000 extra to skip the disk. In this case, they paid for quick access time, light weight, and durability that exist as much in their perception as in real-world benchmarks.

EMC’s Heavyweight Champion

In the exact opposite corner of the data storage world lurks EMC’s top-line Symmetrix DMX storage array. When the company announced the availability of NAND flash drives as their top-tier choice for storage, it turned the heads of the whole enterprise storage industry. Although the technology implementation is substantially different from Apple’s iPod, EMC’s move suggests that another group of customers exists who are similarly unimpressed by a low cost per GB: Enterprise application managers.

Many have suggested that enterprise flash is not yet competitive in terms of price, capacity, reliability, or even performance. And they have publicly disagreed with EMC CEO, Joe Tucci, who claimed effective parity after 2010 at last year’s EMC World event. After all, today’s enterprise flash drives are far more than ten times more expensive than their spinning brothers, and disk capacity continues to march higher by the month.

But the comparison is not about the cost of apples or oranges. In the enterprise storage space, flash drives sot at the top of the pyramid, with just a few units added into the traditional tiered storage mix as a “tier zero” of maximum performance. It is not as simple as pulling out a set of 146 GB FC drives and replacing them with a similar number of flash units. Instead, a few key applications or data sets are migrated up to the pinnacle, with the rest of the stack remaining the same.

There is huge promise when this tiered model is combined with storage virtualization, especially the automated variety. If the tiny percentage of storage that truly needs top-tier performance could be moved to a few solid state disks, the whole stack will benefit from reduced device contention. If automation could make the decision on a block-by-block basis, the effectiveness would be much greater.

I Still Remember

There is another kind of solid state disk in play, too. For over two decades, company after company has pushed the idea of packaging high-performance DRAM as a disk substitute for enterprise storage, just as EMC has now done with NAND. These RAM-based disks offer even higher performance and prices than their flash-based cousins, and none has taken the industry by storm.

Way back when a tiny EMC was one purveyor of solid state storage, I recall the philosophical conundrum posed by the devices: Is it better to package DRAM as storage and use it in a conventional manner or to use that same memory as a cache for actual disks? The market voted for the latter, with EMC and others introducing in-array cache to accelerate RAID to great effect. System memory expanded in parallel, with modern servers optimally caching data in three or more levels internally as well.

Where Does the Flash Go?

For most uses, this is precisely the correct configuration. The priciest and quickest “storage” is placed close to the CPU, with performance and cost dropping and capacity increasing as one moves outward.

Where does flash belong, then? Apple teaches us that NAND flash delivers the goods when it comes to the portable market, and it is likely that the use of this technology in this area will only continue to grow. And EMC shows that there is a need for higher performance in the enterprise storage world as well, though perhaps not enough for pure DRAM devices.

The message is clear: As long as the cost of disk continues to lead, NAND flash will remain a niche product. There are certainly markets for NAND-based devices, from portable computing to the enterprise, but disk just works too well to be displaced. While one can never see too far into the future of storage, it seems clear that conventional hard disks will remain the dominant media for a few more generations of technology at least.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Flash Forward or Flash Back?
This post was categorized as Apple, Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Register did a nice job of summing up the (late 2008) flash positioning of the various storage companies, and I recently posted a strategic look at this core issue. Note that some, like HP and Sun (and probably IBM), seem to have an end-to-end strategy, while others are firmly in one camp or the other. In the “not yet” column, apparently, are 3PAR, BlueArc (though they offer TMS RAM), Dell/EqualLogic, HP/LeftHand.

Update: 3PAR has joined the “disk” camp.

It’s a Cache

Fusion-IO has rocketed to the forefront of the cache side with their PCI Express flash boards for servers. Joining them in this position are the following companies:

• HP (blades and servers)
• NetApp (PAM read cache card)
• Pillar (Slammer cache)
• Sun (read and write cache)

It’s a Disk

STEC is the darling of the flash-as-a-disk world, though Intel, Marvell, and Samsung are also playing here. Joining them in the corner are the following:

• EMC (STEC flash drives shipping the DMX now and CLARiiON in the future)
• Compellent (flash drives)
• HDS (flash drives in the USP-V)
• HP (flash drives in the MSA, perhaps, and maybe that Oracle thing)
• IBM (Fusion-IO storage behind SVC and standard flash drives in the DS5000)
• LSI (flash drives in the 7900)
• NetApp (ssd drives)
• Pillar (flash drives)
• Sun (flash drives in Thumper and JBOD)
• Xiotech (flash drives in the next Emprise canisters)
• 3PAR (flash drives in InServe)

So there you have it. It’s both a disk and a cache, depending on whether you sell servers or arrays apparently. If you sell both, it’s both. Simple!

If anyone wants to correct any of this, drop me a line or comment below!

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Is Flash A Disk Or A Cache?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Fram approaching in front of iceberg upernavik, CC-by-SA copyright Kim Hansen

Solid state (NAND flash) storage is all the rage right now, but there are many lingering questions regarding its true performance, reliability, and cost. But no question is more important in determining its ultimate usefulness than that of location: Where should flash storage be placed to maximize return on investment?

Storage companies have argued that flash disks can be used most effectively in external storage devices, arguing that it’s simpler to just leverage existing storage technologies. Server companies have tended to prefer to place it inside the server, asking why, if flash disks are capable of massive random I/O performance and extremely low latency, one would put them at the other end of a Fibre Channel or iSCSI connection, which introduces latency and tends to combine I/O operations?

The Case For Servers

The argument for placing flash in (or very close to) servers boils down to two key contentions:

1. Distance = latency, so moving quick flash devices away from I/O-hungry CPUs erodes their effectiveness
2. Granularity (or lack thereof) is the core problem facing storage management, so moving flash (and other types of storage) closer to the omniscient application is likely to bring greater effectiveness

The server folks are relying on a technical argument – that placing high-speed cache where it could theoretically do the most good is the right decision. And they are right, in a perfect world: A flash-aware application talking to a low-latency flash device over PCI ought to really fly!

There is some disagreement in the server-side argument as well: Is flash a “fake disk” or a new level of caching between RAM and storage? It seems that the pitch leans toward the latter, even when the SSD appears as a disk drive. This is what Fusion-IO is pitching: They skip old-school disk connections like SATA and SAS altogether, placing their storage on PCI Express and asking hardware and software vendors to integrate it as best they can. Consider Sun’s flash integration for ZFS, for example. Note, by the way, that Intel’s Turbo Memory products also offer PCIe flash, despite what you might be hearing.

Of course, we can just use a flash drive in place of an internal hard drive. Just about everyone makes something like this now, and they work pretty well in some cases. Then there are hybrid drives, which have gone nowhere so far.

But will this work? We need operating systems and applications that can make use of this local flash, and that has been a problem. Intel’s flash-on-the-motherboard idea never caught on, even as Vista included ReadyBoost, because the truth is that operating systems, file systems, or applications must be re-engineered to really make use of flash in a server. That’s happening, but slowly.

The Case for Arrays

Then we turn to the other end of the storage pipe. EMC put flash in the DMX in January, and Compellent is doing it as we speak. IBM went wild with a bunch of Fusion-IO drives and an SVC over the summer, too. All this proves that flash works in storage arrays!

Why? Simply because modern storage arrays are already engineered to make good use of disk drive capabilities. This is a “what works” strategy – even though it doesn’t sound as nice in theory, the array doesn’t need lots of re-engineering to see some benefit from flash. And post-RAID virtualized systems like that Compellent can really make hay with a few super-speed flash drives, since they can move hot blocks to flash dynamically.

Sure, there is latency between the CPU and the flash drive, but storage arrays are really computers in their own right. So they can derive the same benefit from flash that a server could, and they can share that benefit to connected servers rather than leaving it locked up.

Why Not Everywhere?

How about we end the debate. Flash works great in the server, and it works great in the array. Why not just put it anywhere it makes sense in your particular environment? Have an operating system, application, or file system that can make use of server-side flash? Go buy a Fusion-IO card! Have a virtualized enterprise storage array? Get some SSD there, too. And remember that it’s not all about NAND flash – RAM-based solid state storage from companies like Texas Memory Systems, Gear6, and Violin are even faster!

But remember one thing: This stuff is still very very expensive, so you have to really need the performance to make a case for flash.

Image by Kim Hansen, GFDL or CC-BY-SA (source)

© sfoskett for Stephen Foskett, Pack Rat, 2008. | SSD: So Close and Yet So Far
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Let’s take a second to review: EMC became the first modern storage vendor to include NAND flash-based solid state drives as a standard tier of storage in an enterprise storage product when they announced flash in the Symmetrix DMX in January. Although every other vendor has made “me too” comments since then, enterprise flash remains pretty rare. Could Compellent really be the second major vendor to actually do something, coming along 10 months later?

More than a year ago, I rhetorically asked where the enterprise solid state drives were. In that post, and others that followed, I suggested that SSD wouldn’t really “work” as a mainstream tier unless a storage array was smart enough to dynamically allocate content to this “tier-0″ in a granular fashion. In other words, adding a big lump of flash to a static storage array and trying to manually allocate it on a LUN-by-LUN basis to hot applications is not likely to meet anyone’s cost/benefit sniff test!

But if a post-RAID storage system was smart, it could really make use of the technology, and that’s what makes Compellent’s announcement so interesting. They dynamically move blocks (rather than the much-bigger LUNs) around, and could thus make a smaller amount of flash go a lot further. Add a few flash drives and let the system tune itself! This is a big differentiator, folks!

Of course, this is not just Compellent’s advantage. Any fully-virtualized system could do the same, and we’ve heard such talk from folks as diverse as HP (I’d love to see it in both EVA and LeftHand), IBM (for real in SVC, not the science experiment), Sun (combined with ZFS), Dell/EqualLogic, and I’d love to hear it from 3PAR. Bring it on, folks! Listen to Greg!  Let’s get this technology integrated, tested, released, and in the field!

Update: Compellent probably won’t ship their SSDs in volume ’till Q1. But Chris Evans seems to agree with me 100%, and Dell is talking SSD (but no promises yet).

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Compellent Does Enterprise SSD Right
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

EMC SSD – Old is New Again

Check out The Storage Anarchist’s definitive (for now) take on the flash announcement!

A long time ago in Hopkinton, EMC sold a line of solid state storage systems for high-performance applications. This was way back, before blogs, the world wide web, WiFi, Network Appliance, you get the picture. The Allegro/Orion/Atom was actually introduced even before NAND flash itself, in 1988! That product evolved into the Symmetrix platform we now know, and as of today history has wrapped back on itself with EMC’s introduction of solid state storage for the Symmetrix.

I’ve long hollered that typical consumer flash drives weren’t suitable for the enterprise storage market, and EMC has now confirmed that I was right. Rather than slap a commercial flash drive into a storage array and call it a day, EMC and their supplier , STEC, reinvented the flash drive altogether. Please re-read that, and drill it into your head – the new EMC/STEC SSD is an altogether different animal than other flash-based SSDs!

So what’s wrong with consumer-level flash and what’s right about EMC’s SSD? Typical multi-level cell NAND flash (as used in your thumb drive or iPod) used two tricks to boost sequential performance and density at the expense of longevity and write performance:

1. They write large blocks of data at once, which increases sequential performance to an acceptable level but requires large amounts of data to be rewritten when a small amount changes, hurting random write performance and longevity
2. They store multiple values in a single cell which boosts density but exacerbates the problems pointed out above

As Tom’s Hardware noted when they tested commercial NAND-based SSD drives, random write performance was downright terrible because these drives just weren’t built for this I/O pattern.

EMC’s new SSD drives are completely different from the drives tested by Tom’s. They use single-level NAND cells, eliminating problem number one and greatly reducing wear. They also include generous on-”disk” RAM caches and optimized array firmware leveraging the array’s own cache to reduce the random write performance hit noted in problem number one. Finally, the SSD drives include extra capacity set aside for when highly-used cells wear out. With these three modifications in place, I bet EMC really will be able to wring out amazing performance and acceptable longevity.

So what’s the catch? Price. Although I don’t yet have any pricing information, I’d be shocked if these new SSD drives aren’t 10 times more expensive than their spinning metallic brethren. Consumer-level flash drives are already expensive on a per-GB basis, with a 64 GB SATA unit going for $1,500 or more, and EMC’s exotic drives will be much more expensive than this. That single-level NAND, for one, will drive cost through the roof, since these chips aren’t mass produced at anywhere near the same volume as regular multi-level NAND. Add in the fact that these are custom drives built just for EMC, and you’re looking at some serious dough.

Since the company is positioning these drives at the highest of the high end, that price probably won’t matter as much. Any application that really needs this kind of storage will (grudgingly) bear whatever the cost. But don’t expect massive volume shipments in the foreseeable future.

One More Thing…

In the best Steve Jobs tradition, EMC also released another major new upgrade for the DMX, “virtual provisioning.” Although it’s likely to be overlooked by those overcome with flash fever, this has much broader appeal for the DMX customer base. Essentially, it’s an EMC-think version of thin provisioning, which allocates actual disk capacity as it is used by applications rather than as it is provisioned by storage administrators.

What sets EMC’s virtual provisioning apart from everyone else’s thin provisioning? Not a lot, actually – except that it’s universally supported across DMX configurations. So you can do whatever you want with it, even replicate thin-to-thin, which is nice. But calling this by a new name just because it (allegedly) works is a little misleading in my book…

You can read more about these announcements at Chuck Hollis’ blog, Storagezilla, and probably everywhere else in short order. I can’t wait to hear what Hu, Toigo, Tony, and the rest have to say! Who’d have thought that the day before Macworld would be this interesting?!

Update:

• TechTarget reports that the flash drives will be 30 times more expensive than conventional disks!
• STEC’s press release for the Zeus-IOPS drives has some more performance numbers. I’m just diving into their product information for the drive.
• Chris Evans, The Storage Architect, noted the addition of 1 TB drives as well.
• Robin Harris suggests that flash disks are more like 75x the performance of a disk drive, rather than the 30x EMC is consistently claiming, and also that Texas Memory Systems’ Ram San offers nearly twice the performance!
• Industry veteran, Fred Moore, chimed in with some history of SSD storage!
• Gear6′s response will be interesting – they’ve got some very cool technology for accelerating storage!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Flash! EMC’s DMX is the New New Thing Again
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.