Monitoring Filesystem Metadata For Thin Provisioning

I began by introducing the core problem: Storage isn’t getting any cheaper due to storage utilization and provisioning problems. Thin provisioning isn’t all it’s cracked up to be, since the telephone game makes de-allocation a challenge. So now let’s talk about how to make thin provisioning actually work.

There are 100 different ways of solving the de-allocation problem, some of which have gained some prominence. They all boil down to two options:

1. Make the server super-smart and have it communicate better
2. Make the storage super-smart and have it make educated guesses

There’s only a few ways that the server-side option can be implemented, and we’ll get to that. But first, let’s take a look at a sort of hybrid approach that relies on known server usage patterns: Metadata monitoring.

It’s really hard for the storage to really understand what the server is doing. The best example that I know of is the Drobo sitting under my desk.

I love this little black box. When I got it, I configured it as eight terabytes and I put a 160-gig disk in it. That’s thin provisioning. And over time, I’m swapping out the disks and I’m doing all my stuff, and it still looks like eight terabytes. Add data, delete it, swap disks, and it always just works.

Not a lot of people know how the Drobo works, though. One of the things that people have complained about is that it only supports certain file systems and partition schemes. The reason for this is a “magical” thing it’s doing that relates very, very closely to the topic of this discussion. The Drobo is the first thin provisioning box that I know of that directly monitors the file system.

What the Drobo does is this: It knows where the supported filesystems (HFS+, NTFS, EXT3, and FAT) keep the record of what’s been deleted. So the Drobo it watches that spot and when you delete something, it reclaims that space. No enterprise storage system can do this, and yet this little box under my desk does it all day long.

This is basically the super, ultimate smarts of storage. But, of course, it’s very limited. It faces a real challenge in an enterprise setting because there is much more variety. We have all these layers of virtualization and weird file systems and things like that to worry about. We just can’t expect a product like this to accommodate everybody, so we just can’t expect this kind of smarts to be put everywhere.

Instead, we have a variety of semaphores sent from the server to the storage array that attempt to solve the telephone game. That’s what we’re talking about next.