3PAR just introduced their third-generation storage hardware, bringing a novel feature to the world of thin provisioning: Hardware-assisted “zero-detection” to convert standard storage to thin provisioning. Although only certain special-case users will benefit from this technology, it’s nice to see someone working on one of the pitfalls of the technology – that it’s really hard to convert from “fat” to thin, let alone to un-provision storage.
What’s Wrong With Thin Provisioning?
As I have explained in my storage seminars, thin provisioning is the opposite of what storage management professionals should be doing: Instead of managing usage, we just throw up our hands and say “you want 500 GB? Fine, you’ve got it!” while all the while only provisioning a fraction of that space. It’s a lie, and is thus bound to catch up with us sooner or later, and probably at just the wrong time.
People use disk space like money – their needs tend to expand to use up all they can get. Tell the users that you just added another 8 TB to the file server and watch their usage spike. Tell a database manager that they need to buy 20 TB and watch as their tablespaces magically start using 19. It’s human nature, and fighting this impulse to consume is precisely what management is all about. Traditional thin provisioning (or “virtual provisioning” in EMC-speak) does exactly this – it “tells” the downstream users of a storage resource that they have more capacity than is actually assigned to them and then grows capacity as it is used. To say that it is controversial is an understatement.
In certain instances, including Drobo and VMware growable disks, this can be beneficial since it’s a pain for most end-user OS configurations to resize a volume after the fact. In these special cases, I concede that thin provisioning is the right way to go. The same could be said for deduplicating or compressed storage – these simply have to be thin provisioned, since the actual allocation is completely abstracted by the compression algorithm. Thin provisioning can also help (slightly) for the OS volumes of virtual servers. But mainstream enterprise users have storage, server, and application managers, so they shouldn’t resort to “tricks” like thin provisioning – instead, they should manage their storage!
But the worst thing about thin provisioning is that it can’t un-provision storage. Let’s say a user uses your thin-provisioned file server as a temporary landing zone while switching to a new laptop. Or your database folks load their LUNs up with SQL dumps after an outage. Or your application folks fill up their test servers prior to going into production. Predictably, that thin-provisioned storage will expand, using up real disk capacity, to take the load (presuming enough capacity is available). The problem arises when they delete this temporary data – the storage array has no way of knowing that those blocks are no longer in use, so it cannot un-provision them. Suddenly your 500 GB thin-provisioned LUN is really taking up 400 GB even though it only has 20 GB of actual data on it, and you feel like a chump. Time to go manage your storage…
Zero-detection helps a little
Now back to 3PAR. Their new T-class InServ storage array has a special ASIC designed to attack a small chunk of the un-provisioning problem. It scans allocated storage, looking for blocks filled with zeros, and de-provisions them. This is nice – it’s a great tool to convert traditional storage to thin-provisioned storage. It’s also the first practical un-provisioning approach I’ve heard about, and might yield some capacity improvements for already-provisioned LUNs in certain special cases, though I’m not sure 3PAR is aiming for this market.
See, it’ll only work for zeroed-out storage, which is sadly extremely rare in the world of storage. It will detect capacity that has never been used, but most filesystems simply change their pointers when a file is deleted – leaving the data just where it was. 3PAR’s effort won’t work in this case. Even decommissioned servers often leave their LUNs full of old data, a security risk to be sure, and not a case that 3PAR could deal with, either.
The only way to make this work for already-used storage would be to add another step to the decommissioning process – zero out LUNs that are no longer in use as a way to send a signal to the storage array that it can un-provision that storage. But of course, we could also just send an email to the storage administrator to de-allocate the LUNs, leaving us in a much better position since we no longer have unused LUNs sitting on the storage array. Maybe we could modify the filesystem to zero out unused storage. Anyone have the source code for NTFS?
Seriously, though, this is a practical step in the right direction. We need better communication between applications, operating systems, and storage in order to enable lots of beneficial features. 3PAR is trying to enable some communication, and I applaud them for that. Just don’t expect too much.
Hi Stephen, Marc Farley from 3PAR here. Thanks for the love and yes we are making steps in the right direction. I agree that thin provisioning is still in the up and coming phase of it’s technology life cycle.
As for the interacting with the other parts of the storage ecosystem, I think you’ll see progress made along those lines in the not too distant future. Otherwise, Thin Provisioning is a terrific purpose-based technology. Its not the optimal solution for user-exposed storage as you wrote, but it is extremely useful for line of business applications and back end application processes where “wild end user behavior” is minimal. These are environments where data creation and growth are more systematic. I don’t know why people wouldn’t want to use it for those scenarios.
Hi Stephen, Marc Farley from 3PAR here. Thanks for the love and yes we are making steps in the right direction. I agree that thin provisioning is still in the up and coming phase of it’s technology life cycle.
As for the interacting with the other parts of the storage ecosystem, I think you’ll see progress made along those lines in the not too distant future. Otherwise, Thin Provisioning is a terrific purpose-based technology. Its not the optimal solution for user-exposed storage as you wrote, but it is extremely useful for line of business applications and back end application processes where “wild end user behavior” is minimal. These are environments where data creation and growth are more systematic. I don’t know why people wouldn’t want to use it for those scenarios.
Hello Stephen,
You mention two glaring issues with Thin Provisioning 1) the inability to make “thin” existing “fat” volumes and 2) runaway volumes consuming space that cannot be recovered. This is a limitation that exists for all vendors that I am aware of, said one – Compellent. Their implementation of Thin Provisioning (they call it Dynamic Capacity) does address both of these issues head on.
When you implement a Compellent Storage Center and bring volumes to the SAN, their implementation allows you to do a “thin import” which will take the existing volume from say an EMC SAN that is allocated at 500GB but only has 44GB of real data, and import that so that only 44GB of space is utilized. This has been operational with Compellent for quite some time.
The second issue is addressable in Windows applications on Compellent today. When space is deleted on a windows volume, you run a simple “clean up” application called “Free Space Recovery” and that deleted space is now completely deleted and reusable. They are working on the other O/S’s, but it has been working this way since early this year, and honestly, it simply works.
When I look at the market for Thin Provisioning, there are only two companies who get Thin Provisioning right – 3PAR and Compellent.
Thanks
Paul Clifford
Davenport Group
http://www.davenportgroup.com
Paul,
Thanks for the comment! You are correct – Compellent’s thin import is impressive (as is their overall architecture). And their free space recovery app is key to my second concern about recovering once-used space – when it’s run. In short, the app communicates to the array that space can be recovered (thinned?) in the only way possible currently – on command.
It’s such a tough issue to crack – how to convince thin-unaware applications like filesystems to communicate with thin-capable storage systems. The only real solution is revolution – a new tightly-coupled file system, volume manager, and storage service. Until we get there, though, I do hope Compellent and 3PAR and others keep trying to crack this uncrackable nut!
Stephen
Hello Stephen,
You mention two glaring issues with Thin Provisioning 1) the inability to make “thin” existing “fat” volumes and 2) runaway volumes consuming space that cannot be recovered. This is a limitation that exists for all vendors that I am aware of, said one – Compellent. Their implementation of Thin Provisioning (they call it Dynamic Capacity) does address both of these issues head on.
When you implement a Compellent Storage Center and bring volumes to the SAN, their implementation allows you to do a “thin import” which will take the existing volume from say an EMC SAN that is allocated at 500GB but only has 44GB of real data, and import that so that only 44GB of space is utilized. This has been operational with Compellent for quite some time.
The second issue is addressable in Windows applications on Compellent today. When space is deleted on a windows volume, you run a simple “clean up” application called “Free Space Recovery” and that deleted space is now completely deleted and reusable. They are working on the other O/S’s, but it has been working this way since early this year, and honestly, it simply works.
When I look at the market for Thin Provisioning, there are only two companies who get Thin Provisioning right – 3PAR and Compellent.
Thanks
Paul Clifford
Davenport Group
http://www.davenportgroup.com
Paul,
Thanks for the comment! You are correct – Compellent’s thin import is impressive (as is their overall architecture). And their free space recovery app is key to my second concern about recovering once-used space – when it’s run. In short, the app communicates to the array that space can be recovered (thinned?) in the only way possible currently – on command.
It’s such a tough issue to crack – how to convince thin-unaware applications like filesystems to communicate with thin-capable storage systems. The only real solution is revolution – a new tightly-coupled file system, volume manager, and storage service. Until we get there, though, I do hope Compellent and 3PAR and others keep trying to crack this uncrackable nut!
Stephen
Similarly to Compellent’s approach we’ve, NetApp, addressed space recovery via a Space Reclaimer process on Windows on May 2007 with the SnapDrive for Windows v5.0.
Similarly to Compellent’s approach we’ve, NetApp, addressed space recovery via a Space Reclaimer process on Windows on May 2007 with the SnapDrive for Windows v5.0.
One comment I have on this is that Compellent’s ‘Free Space Recovery’ agent only work on RDMs. I have a large deployment of microsoft IIS servers that run VMFS on a Compellent SAN. I created them with thin provisioning in mind, but as time goes by, and data is written/deleted to my various volumes…I lose that provisioning, and am currently unable to reclaim that space.
Now, I have read on a Datacore forum thread that when ESX writes to a VMFS volume, it does so plainly without altering data. I haven’t verified this yet, but if it is indeed true, this could mean that zero detection could be of great benefit across various operating systems.
So does anyone know if using tools like ‘sdelete’ to write zeros to free space on a VMFS disk would show up as zeros in back-end storage?
One comment I have on this is that Compellent's 'Free Space Recovery' agent only work on RDMs. I have a large deployment of microsoft IIS servers that run VMFS on a Compellent SAN. I created them with thin provisioning in mind, but as time goes by, and data is written/deleted to my various volumes…I lose that provisioning, and am currently unable to reclaim that space.
Now, I have read on a Datacore forum thread that when ESX writes to a VMFS volume, it does so plainly without altering data. I haven't verified this yet, but if it is indeed true, this could mean that zero detection could be of great benefit across various operating systems.
So does anyone know if using tools like 'sdelete' to write zeros to free space on a VMFS disk would show up as zeros in back-end storage?
Awesome article, I am regular visitor of this website, keep up the excellent work, and I will be a regular visitor for a very long time..