Although I consider it the main stumbling block for thin provisioning, communication (or lack thereof) is being addressed with metadata monitoring, WRITE_SAME, the Veritas Thin API, and other ideas. But communication isn’t the only issue.

Let’s talk about page sizes. You’ll often see vendors tossing this “softball” objection at their competitors, claiming that their (smaller) page size makes for more-effective thin provisioning. And that’s true, to a some extent, but perhaps not the end of the story.

Look at the top block in this stack. The light background box is the page, and the colored boxes represent data. If your storage is written in “pages” of this size, you can’t thin it.

What if we used a smaller page? What if my page is a quarter of that size, as in the second row? I still can’t thin it out, because my data is spread all over the place.

Remember worrying about fragmentation back in the days of DOS and Windows and FAT filesystems? It’s kind of like this.

Because we’re using zero page reclaim, the whole page has to be zero to be reclaimed. If your data is all over the place, if there’s even one bit that’s not zero on a page, we’re not going to reclaim that whole page.

Now let’s return to our illustration. If we use a little bit smaller page, as in the bottom two rows, we can reclaim some space. If we use a really tiny page, we can reclaim half the space even.

We’re still not reclaiming all the space, though. At the beginning of this series, I showed the “simplified perfect-world” thin provisioning illustration. In that picture, the half-empty barrel was perfectly reclaimed thanks to this technology. We will never get there unless we are using really minuscule pages. But we can get somewhat close. Maybe we can thin out three-quarters of the empty space.

But some vendors use really big pages. Some folks made fun of Hitachi for using 42 megabyte pages, since, if there’s one bit in 42 megabytes of potential ones or zeros, the Hitachi will not thin that. It also won’t migrate it for automated storage tiering. But others use even-bigger pages; up to a gigabyte in size. And 42 MB isn’t that bad in practice.

I know of a company that’s doing four-kilobyte pages. And EMC actually allocates one-gigabyte slices of storage for writing on the CLARiiON, even though their thin size is 8 KB. So is the CLARiiON page size 8 KB or 1 GB? It’s very confusing to me (and probably the customer too)…

The trouble with 4 K or 8 K pages is it makes an awful lot of pages to keep track of. Consider the analogy of hard disk drive sector sizes. An ATA disk could only get to 2.1 terabytes until recently, because they still used 512-byte sectors. And 512 bytes times the biggest 32-bit number is 2048 GB. So 512 bytes makes for greater efficiency in theory, but hurts scalability in practice. So, the disk drive industry is moving to 4 K sectors.

It’s exactly the same thing as with thin provisioning. So, you’ve got to keep track of all these gazillions and gazillions of pages. So, from a vendor perspective, you can save a lot of horsepower and make it a lot easier to implement if you have bigger pages. It also means you’re not moving stuff around as much when using these big pages for automated tiering.

I’m not going to throw rocks at HDS or anyone else over page sizes. I actually don’t think 42 MB is that bad, because the biggest problem with underutilization is not inside a file system. In my experience, the big problem is storage that’s not used at all.

When I used to do storage assessments, it was very common to find LUNs that were allocated ant not used at all; not even touched. Your page size doesn’t matter if a LUN is not even touched: It’s going to be thinned out no matter what. So, regardless of the page size, thin provisioning will probably save more space outside a filesystem than within one, especially if your systems administrators are doing a reasonably good job of storage management. And even if they’re not doing a good job, there’s probably 42 megs of zeros that can be thinned out anyway.

So, I’m not as worried about the size of the pages. Granularity is an architectural decision, and larger pages are not the end of the world. Ask your vendor if they support thin provisioning and what the granularity or page size is, and think about how that’s going to affect you. At the end of the day, it’s probably going to yield about the same result no matter what the page size is.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Granularity of Thin Provisioning Approaches
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

EMC mentioned this schedule on their public earnings call, repeated it in discussions with Enterprise Storage Forum, and corporate PR confirmed it to me this morning. FAST is, if you will excuse the pun, fast-approaching!

There’s actually a very-significant difference between FAST versions 1 and 2:

• FAST version 1 (coming in December 2009) can automatically migrate an entire LUN between tiers of storage on the Symmetrix V-Max based on access patterns. This is a nice-to-have feature, but does not yet live up to the promise of automated tiered storage, as pioneered by Compellent and offered in various forms by many other storage companies. I am told that FAST version 1 also does not support virtually-provisioned (thin) volumes. Although FAST will make it much easier to take advantage of solid state flash drives (EFDs to EMC), it will do nothing to improve utilization.
• FAST version 2 (coming in May mid-2010?) will automatically move smaller sub-LUN pieces of storage between tiers of storage, and will reportedly support virtual provisioning. It is likely that FAST would use the same extent size supported by virtual provisioning, which is 12 tracks or 768 KB. The combination of virtual provisioning and fully-automated storage tiering will finally EMC the ammunition they need to defend their turf against smaller challengers when it comes to these sticky technical features.

So there you have it. EMC V-Max gets FAST in December 2009 and May mid-2010! Now how about a schedule for geo-distributed clustering?

Update: Fixed two typos above. I wrote “May 2009″ instead of “May 2010″. You should see my checkbook!

Note: Although Lou Przystas told Enterprise Storage Forum that FAST V2 would come in May, EMC has apparently not set a hard date and are saying “mid-2010″ at this point. Even this is an aggressive ship date, but I won’t say EMC missed it unless we haven’t got it by the end of summer!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | EMC V-Max FAST: Coming in December … And 2010!
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This is how we used to avoid hotspots in 1998: Carefully planning every detail of the storage layout.

Each disk drive consists of a spindle of spinning platters with read/write heads move back and forth. Each time you access a piece of data that’s not in cache, the drive moves its arm over the platter to access the correct piece of data. Since each drive can only access one piece of data at once, and since caches can only hold so much data, tuning a system to minimize the number of requests per drive is essential.

Manual storage array layout was an art, but we never fooled ourselves into thinking our designs were optimal. There were just too many intractable problems, so we had to compromise at every turn:

• We usually had no performance data to base our layout decisions on, so we had to rely on guesses and rules of thumb
• Workloads tend to change over time and manual layouts are painful to modify
• The smallest unit of allocation was an entire LUN or drive, so even the best disk layout mixed hot and rarely-accessed data everywhere
• Much of the allocated space was unused, so we used expensive disks to store nothing

One might think that, 10 years later, advances in technology would have solved these basic issues. But for many people using many of the so-called modern mainstream enterprise storage systems, these problems remain.

Like all good systems administrators, I’m a natural control freak. I am uncomfortable letting the system manage itself, having been burned too many times by computers (well, software really) making stupid decisions. It’s analogous to the backlash against anti-lock brakes, traction control, and automated transmissions among racing enthusiasts.

Do we allow technology to help us get better performance, or do we try to micro-manage everything? Photo by ClearInnerVision

But the time has come to let go. We don’t have to micro-manage storage anymore, and we have much to gain by letting the array do the work:

• Just as traction control can manage each wheel independently, something a driver could never do, modern virtualized storage systems can allocate small “chunks” to the optimal drive type, creating a better layout than anyone could manage with LUNs
• Dynamic optimization technology can move these chunks around, adapting as loads change
• Thin provisioning can go a step further, not wasting drive capacity for unused space
• Wide striping and post-RAID storage systems have a higher threshold before performance suffers due to spindle hotspots
• Widespread availability of tiered storage, including advanced caches, solid state drives, high-performance SAS and FC, and cheap bulk disks, gives us many more options

As I mentioned, not all systems have these capabilities, and not all implementations are created equal. I’m concerned about misuse of thin provisioning, for example, but it’s hard to argue with its effectiveness in many circumstances. Find out how granular your system’s allocation is – some remain LUN-only, while others are much more effective, using tiny chunks.

These new storage automation technologies really become essential once high-dollar flash storage is added to the mix. If you’re paying 30 times more for a flash drive, you want to make sure you’re making the best use of it that you can! Look at IBM’s recently-announced SAN Volume Controller (SVC) and solid state drive (SSD) combination, for example: It will almost certainly have fine-grained thin provisioning of SSDs, and should be able to dynamically move data between flash and disk storage and even between different storage arrays, but I still have questions on how granular this capability will be. HDS can do similar things with their USP-V. NetApp’s V-Series NAS systems will do dynamic allocation, thin provisioning, and data deduplication to enable a better return on the flash drive investment. I’d love to see 3PAR, Compellent, Dell/EqualLogic, and HP/LeftHand apply their solid dynamic allocation tech to solid state storage as well!

Then there’s the 800 lb gorilla: EMC. More enterprise SSD has probably been shipped out of Hopkinton than every other vendor combined, and both the CX and DMX support (optional/expensive) “virtual provisioning” (aka, thin provisioning) of flash storage. But EMC’s Optimizer is not widely used, and only migrates entire LUNs based on user input – hardly the kind of dynamic and granular technology needed to optimally use all of that flash storage. I’m sure the company is working on addressing this issue, though. Perhaps it will appear in the DMX-5 announcement we are all expecting this year?

This article can also be found on Gestalt IT: How I Learned to Stop Worrying and Love Storage Automation

© sfoskett for Stephen Foskett, Pack Rat, 2009. | How I Learned to Stop Worrying and Love Storage Automation
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What’s Wrong With Thin Provisioning?

As I have explained in my storage seminars, thin provisioning is the opposite of what storage management professionals should be doing: Instead of managing usage, we just throw up our hands and say “you want 500 GB? Fine, you’ve got it!” while all the while only provisioning a fraction of that space. It’s a lie, and is thus bound to catch up with us sooner or later, and probably at just the wrong time.

People use disk space like money – their needs tend to expand to use up all they can get. Tell the users that you just added another 8 TB to the file server and watch their usage spike. Tell a database manager that they need to buy 20 TB and watch as their tablespaces magically start using 19. It’s human nature, and fighting this impulse to consume is precisely what management is all about. Traditional thin provisioning (or “virtual provisioning” in EMC-speak) does exactly this – it “tells” the downstream users of a storage resource that they have more capacity than is actually assigned to them and then grows capacity as it is used. To say that it is controversial is an understatement.

In certain instances, including Drobo and VMware growable disks, this can be beneficial since it’s a pain for most end-user OS configurations to resize a volume after the fact. In these special cases, I concede that thin provisioning is the right way to go. The same could be said for deduplicating or compressed storage – these simply have to be thin provisioned, since the actual allocation is completely abstracted by the compression algorithm. Thin provisioning can also help (slightly) for the OS volumes of virtual servers. But mainstream enterprise users have storage, server, and application managers, so they shouldn’t resort to “tricks” like thin provisioning – instead, they should manage their storage!

But the worst thing about thin provisioning is that it can’t un-provision storage. Let’s say a user uses your thin-provisioned file server as a temporary landing zone while switching to a new laptop. Or your database folks load their LUNs up with SQL dumps after an outage. Or your application folks fill up their test servers prior to going into production. Predictably, that thin-provisioned storage will expand, using up real disk capacity, to take the load (presuming enough capacity is available). The problem arises when they delete this temporary data – the storage array has no way of knowing that those blocks are no longer in use, so it cannot un-provision them. Suddenly your 500 GB thin-provisioned LUN is really taking up 400 GB even though it only has 20 GB of actual data on it, and you feel like a chump. Time to go manage your storage…

Zero-detection helps a little

Now back to 3PAR. Their new T-class InServ storage array has a special ASIC designed to attack a small chunk of the un-provisioning problem. It scans allocated storage, looking for blocks filled with zeros, and de-provisions them. This is nice – it’s a great tool to convert traditional storage to thin-provisioned storage. It’s also the first practical un-provisioning approach I’ve heard about, and might yield some capacity improvements for already-provisioned LUNs in certain special cases, though I’m not sure 3PAR is aiming for this market.

See, it’ll only work for zeroed-out storage, which is sadly extremely rare in the world of storage. It will detect capacity that has never been used, but most filesystems simply change their pointers when a file is deleted – leaving the data just where it was. 3PAR’s effort won’t work in this case. Even decommissioned servers often leave their LUNs full of old data, a security risk to be sure, and not a case that 3PAR could deal with, either.

The only way to make this work for already-used storage would be to add another step to the decommissioning process – zero out LUNs that are no longer in use as a way to send a signal to the storage array that it can un-provision that storage. But of course, we could also just send an email to the storage administrator to de-allocate the LUNs, leaving us in a much better position since we no longer have unused LUNs sitting on the storage array. Maybe we could modify the filesystem to zero out unused storage. Anyone have the source code for NTFS?

Seriously, though, this is a practical step in the right direction. We need better communication between applications, operating systems, and storage in order to enable lots of beneficial features. 3PAR is trying to enable some communication, and I applaud them for that. Just don’t expect too much.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | 3PAR’s Thin Un-Provisioning is Slightly Less Bad
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.