Although the core issues with thin provisioning revolve around communication, it presents unique challenges to the storage array as well. We talked about granularity of pages, and the comments for that piece were extremely enlightening. Now let’s consider another key factor: Scheduling.

Note that the “provisioning” part is relatively easy to do on the fly: An array just has to allocate additional capacity as writes come in, which is something it does anyway. It’s the thin reclamation that poses a challenge, since this involves zero detection across a whole page of data in many cases.

Just like de-duplication, thin provisioning challenges the resources of the storage array to do background number crunching. And just like dedupe, the array engineers have a choice of when to do the reclamation processing: Well after writing or “in-line”. The extreme ends of this spectrum fall into two equally disappointing categories: Wholly ineffective or ridiculously intensive.

Let’s start with the “intensive” side: You could have the controller do thin provisioning automatically; that’s kind of what IBM does with SVC, for example, and 3PAR claims to do this too. The trouble is that the controller has to literally watch everything, and it’s got to reassemble whole pages, perhaps 42 MB or even one GB in cache. If it didn’t have all that data, it would have to go fetch it, put it into cache, look at it, make sure it was all zeros, then get rid of it. It’s really, really difficult to do automatic, in-line, thin provisioning. It’s a good thing to do, but it’s a hard thing to do.

So most vendors schedule thinning for later. In the “10 terabytes of zeros” example, they’re actually going to write 10 terabytes to disk, or at least through to cache. Then, at some point in the future, they’ll go back and reclaim that space. Some are pretty aggressive and reclaim capacity very frequently. Others are fairly lazy: The Drobo seems to reclaim only once or twice a day. A lot of people who have them are surprised when the thing springs to life and starts going, “Bada-bada-bada-bada-bada-bada.” Apparently it’s reclaiming storage at that time.

Some thin provisioning systems are even manually-initiated, and this is really pretty ineffective. The storage administrator has better things to do than reclaim storage all the time, so they are probably going to set a cron job to do it regularly at a specified time. If the system only does it on demand, that means that it doesn’t have the horsepower to do it automatically. Ergo, it’s sometimes going to conflict with “real work” and cause a problem.

I would look for a system that was fairly aggressive with thin reclamation. I was talking to the guys at Nimbus Data, for example, and they claim to do thin provisioning in-line all the time. I hope that we see more storage arrays that are doing that, and less that are doing it manually, on demand, because that’s just not as useful.

But considering that thin provisioning used to be almost useless, the fact that it’s now at least somewhat useful is gratifying.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Processing and Scheduling Thin Provisioning
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

If WRITE_SAME can be a semaphore for thin un-provisioning, what about TRIM? It sounds like a perfect fit, and has wider implementation to boot! Let’s take a deeper look.

First, I’d like to remind the reader that I’m not some specialist storage engineer. I am regular guy who wanted to know about this TRIM thing. I welcome corrections and feedback on this piece!

We should note that TRIM was invented for SSDs, not thin provisioning. TRIM lets an operating system tell a storage device that certain blocks of data are no longer used. Although this sounds like a thin provisioning tool, it’s not. TRIM was never intended to be used this way.

Even though Microsoft supports TRIM in the file system, and even though, hopefully, Apple will introduce TRIM support in OS X Lion, this doesn’t do what we need it to do for thin provisioning. In fact, TRIM is not anything for enterprise storage folks to be getting all that excited about.

One reason for is is that TRIM is an ATA command, and most enterprise systems are not connected to their storage using any kind of ATA-based protocol. Maybe there are SATA disks somewhere down the line, but, whether they use iSCSI, FC, SAS, or FCoE, the servers speak SCSI.

Even though there are SCSI analogs of of TRIM, these commands were really not designed for thin provisioning. They are designed for SSDs, which have a big performance problem: An SSD can delete data and write data just fine, but in order to update data, they actually have to read, delete, and write a big page of data. What happen is that, as an SSD gets full, it starts slowing down. And that’s a problem because people buy SSDs for speed. So TRIM is a way to tell the SSD, “Sometime in the future, you can go ahead and delete this if you feel like it,” and the SSD will take care of that later.

There’s great OS support for this, but unfortunately, this is not something that enterprise storage vendors are looking at. The only vendor (that I know of) that’s working on this, NetApp, did propose to T10 a mechanism to do thin provisioning based on TRIM. I don’t think it got anywhere. They say that it did, but I just don’t see it. HDS and EMC, seem to like UNMAP and SCSI, which are peripherally related, but I don’t know much about what they’re doing either. Perhaps they’ll let me know in comments on this post.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Is TRIM Useful For Thin Provisioning?
This post was categorized as Apple, Enterprise storage, Everything, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I began by introducing the core problem: Storage isn’t getting any cheaper due to storage utilization and provisioning problems. Thin provisioning isn’t all it’s cracked up to be, but it has a place. So what’s wrong with it?

The problem with thin provisioning starts with the telephone game. Did you ever play the telephone game as a kid? Maybe you whisper “I like potatoes” to the first person in a circle and when it comes back to you it’s changed to “Meet Mike on the patio” or something like that. It’s a totally different message.

What happens in the telephone game is that a little bit of information gets lost at each step along the path, and at the end of the chain you’ve basically lost all the information. And this happens all the time in computers, especially in data storage.

We storage guys are stuck at the bottom of a stack that includes many layers. Each of those layers loses something in translation, mostly because it’s pretending to be something that it’s not.

Think about storage today: We’ve got fake file systems pretending to live on fake discs that pretend to be directly connected to your computer. But they’re not.

Everything we do in enterprise storage is basically faking out something else so compatibility is maintained. And at each step (the file system, the database, the host, the network) you’re losing information. By the time you get to the storage system, there’s just no communication whatsoever.

This is the core problem with thin provisioning. The application knows what data is temporary, and that would be very useful for the storage system to act upon. But by the time the data gets there, the message is lost. Maybe the application tells a database. Maybe the database tells the file system. Maybe the file system tells the volume manager. But, that’s about as far is it’s going to go. So, this is really the issue. It’s the telephone game.

Let’s say we have a file system and some storage. We want to write some data. So, the file system says, “Hey, here’s my new block.” And the storage says, “Yeah, I got it.”

This is the classic way of doing thin provisioning. So the storage system is now only using the little blue box. The file system adds some new data, then some more data, and the storage just keeps growing. The rest of the space can be reallocated.

We’re good so far. This is so simple that most products in storage now have something like this. Of course, it took them 10 years to do it, but they finally have it.

So, we’re good. We can allocate storage. But, what about deallocate?

If I delete something, I have to tell the storage, and then it has to shrink the allocated capacity. But we’re not doing that. Most file systems don’t actually send that information on. When you delete a file, most file systems actually write more data instead of actually deleting anything.

Thin reclamation is the core technical challenge to thin provisioning, and the telephone game is the reason. Next we’ll present some solutions that are currently being worked out.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Thin Provisioning: Playing the Telephone Game
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I began by introducing the core problem: Storage isn’t getting any cheaper, even though raw capacity is cheap. Then I talked about storage utilization and provisioning problems. Now we’re going to talk about thin provisioning.

So, what is thin provisioning? I imagine everybody can just figure it out by the name. It means that instead of allocating storage that we’re not even going to use, we don’t allocate it, and give it to somebody else.

All right. We’re done. Anybody have any questions?

Ok, you caught me. This is actually not how it works. This is how you would like it to work, but this is not how it works. This is the “vendor illustrated” version of how it works. So let’s talk about why thin provisioning doesn’t work this way.

Thin provisioning is problematic. What happens is I allocate some space and suddenly I need a lot more. Perhaps I have temporary requirements and suddenly my application is spitting out log files like crazy, filling up all my storage, and locking everybody out. Oops, that’s a problem. But it’s probably exaggerated by thin provisioning opponents.

One of the biggest technical issues for thin provisioning is not the provisioning part: It’s fairly easy for a storage array to allocate on request:

1. Operating system: “I need a block; here’s some data I want you to write.”
2. Storage array: “I’ll just start allocating now.”
3. But, the operating system never goes back and says “I don’t need that block anymore.”

De-allocating (de-provisioning) is the challenge of thin provisioning. And that’s something that has to be tackled. And the reason that I say that thin provisioning isn’t quite as bad as it used to be is because that problem is being addressed.

Another problem we’re going to talk about is chunk sizes and formatting conflicts. We’re also going to talk about application conflicts.

There are a lot of problems with thin provisioning, but it’s getting better. I’ve been talking about this for years. I actually had a version of this slide that I used in 2003 or 2004, right when 3PAR started really pushing thin provisioning. And at that point I was saying “hold you horses everybody.” Now I’m getting a little more friendly to thin provisioning.

I realized, after years of griping, that my discomfort with so many technologies comes down to this: If people are trying to solve a business problem with a technical solution, they’re going to fail. It doesn’t matter if you’re talking about thin provisioning or anything else.

If your problem is that the organization won’t give you funds to actually buy storage, so you have to steal it from the next project that comes along and then reapply that storage to other projects just to keep your growth up, well, then thin provisioning is not going to solve your problem. If your problem is that you can only buy once a year, or you can only buy when the EMC salesman gives you a super discount and leaves something on the loading dock, well, thin provisioning is not your problem. Your problem is somewhere else.

But if your problem is that you have file systems that don’t like to grow and shrink, or if your problem is that you have a standard size allocation unit that everybody has to use and it always leaves some space empty, you are in luck. These are technical problems and those are things that you can solve with thin provisioning.

I wrote more about this topic in Use Process Solutions For Process Problems, Technical Solutions For Technical Ones

So, just think about it. Are you trying to solve a business problem or a technical problem? And again and again you’ll see that technical solutions should be applied to technical problems and business solutions should be applied to business problems. And there’s no solution to political problems, so what can you do?

© sfoskett for Stephen Foskett, Pack Rat, 2010. | De-Allocating is the Core Issue for Thin Provisioning
This post was categorized as Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

First, let’s clarify just which products and capabilities Symantec is offering here:

• Veritas Storage Foundation version 5.0MP3 for Unix/Linux includes SmartMove and the Thin Reclamation API
• Veritas Storage Foundation for Windows 5.0 only includes SmartMove at this point, but it will be updated to include Thin Reclamation at some point in the coming year

Although there is no real information on Symantec’s web site about all this yet, Symantec’s director of Storage Management and High Availability, Sean Derrington, assures me that their software is available now. Although no compatible arrays are in end-user hands, 3PAR will update their T-Class firmware to support the API shortly, and HDS and HP are on the way as well.

Thin Aware Software

Next, contrary to what I inferred from the announcement, there is no native thin provisioning capability in the file system or volume manager. So the first item in my list is right out. However, the volume manager is now “thin aware”, which means that it will communicate up to the file system and down to the array to coordinate more effective use of space.

When the volume manager is used with Veritas File System (VxFS) on UNIX or NTFS on Windows Server 2003 or 2008, it will automatically keep track of deleted files and will pass this information down the stack to the array. This is a major piece of functionality to add, especially to NTFS, “hole punching” (like NetApp) to maximize thin provisioning.

The Storage Foundation tools have also been updated to properly report on thin provisioned volumes. For example, the following screenshot shows three disk devices where encl1 supports thin reclamation and encl0 does not.

# vxdisk list
DEVICE        TYPE   DISK          GROUP         STATUS
encl0_0       auto   encl0_0       mydg online   thin
encl1_0       auto   encl1_0       mydg online   thinrclm
encl1_1       auto   ecnl1_1       mydg online   thinrclm

Thin Reclamation API

The Veritas Thin Reclamation API allows the Storage Foundation volume manager and file systems to communicate with thin-capable arrays when data is deleted on thin-ified LUNs, maintaining their thin-ness as you go. When a file is deleted, the file system will communicate to the volume manager that that space is no longer needed. When the server administrator runs the “vxdisk reclaim” or “fsadm –R” commands, the volume manager will communicate this information to the array (using SCSI commands) that any vacated disk blocks can now be reclaimed. Symantec expects folks to set up a cron job to reclaim space, or perhaps just run it when they see the need.

This is brilliant stuff, and ought to make thin provisioning shine in terms of array utilization. In an environment of thin-enabled Veritas volumes and supported storage arrays, the amount of space used on an array will be awfully close to the amount of space used in the file systems. This is a massive win - a capacity gain of on the order of 50%-70% in an average environment!

For more on this topic, see my recent post on storage utilization

If the storage array fully supports Symantec’s API, the tools will also report physically allocated storage behind thin and thin_reclaim devices.

# vxdisk –o thin list
DANAME        DISK SIZE(Mb)        PHYS_ALLOC(Mb)       DISK GROUP TYPE
encl0_0       2000                 50 mydg              thin
encl1_0       200                  50 mydg              thinrclm
encl1_1       500                  500 mydg             thinrclm

SmartMove

SmartMove is Symantec’s new capability for online migration from “thick” to thin LUNs. It is included in Storage Foundation for Unix/Linux and Windows and works with any thin storage array, not just those that support the API. This is basically a tweak to the old storage migration support we have all known and relied on in Veritas Storage Foundation for over a decade, except that it’s smart enough to not request blocks that it won’t use. One could theoretically “SmartMove” a volume regularly to reclaim space without using the API at all, but those commands are sure a lot simpler.

Note that SmartMove speeds up migration too, even for thick volumes! When you use a SmartMove-enabled version of Storage Foundation to move a volume, it will only send the blocks that have changed over the wire. This reminds me a little of VMware’s new I/O deduplication capability talked about at VMworld, but it’s focused only on migrations, not other I/O situations.

For more on this topic, see my recent post on VMware vStorage

The Plot Thickens

So I was wrong about one item, but the other two remain true. Is Symantec’s new capability a winner? I give it a silver medal – it’s good stuff, but some issues remain.

1. My primary concern remains – thin provisioning does nothing to address the lack of storage management that is so prevalent. It enables greater utilization of capacity, but does nothing to control how that capacity is used. This isn’t a beef with Symantec’s Veritas Storage Foundation or 3PAR or HDS or EMC or anyone in the thin industry, really. Instead, it is a wake-up call to all of the storage organizations out there who have filesystems full of uncontrolled junk!
2. My second concern is the lack of capacity management. Thin provisioning is a lie, promising more capacity than is available. This might be acceptable in certain controlled circumstances like operating system or application volumes, but telling end users that they have plenty of available space is a recipe for disaster. Storage use is like air – it expands to fill all available volume. Without capacity management, your thin volumes will be “overdrawn” and your storage “account” will be bankrupt.
3. Then there is the issue of proprietary APIs versus standards. Let me say right away that I always support standards over proprietary technology. But, at the same time, given the choice between nothing and something, I’ll take the proprietary API. Thin provisioning is a good idea with poor implementation. This API helps to make it useful in the real world, and having a market leader like Symantec behind it makes it all the more relevant. I certainly hope the entire storage industry will come up with a standard thin API, and when that happens I hope Symantec will support it. Until then, at least we have something.

I will be writing more about thin provisioning in the coming weeks. Until then, I continue to applaud Symantec, 3PAR, HDS, and HP for their work in making this technology somewhat more practical. Now how about VMware, Microsoft, Sun, and the Linux guys get some thin technology going, too?

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Symantec’s Thin API: The Plot Thickens
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.