First, let’s clarify just which products and capabilities Symantec is offering here:

• Veritas Storage Foundation version 5.0MP3 for Unix/Linux includes SmartMove and the Thin Reclamation API
• Veritas Storage Foundation for Windows 5.0 only includes SmartMove at this point, but it will be updated to include Thin Reclamation at some point in the coming year

Although there is no real information on Symantec’s web site about all this yet, Symantec’s director of Storage Management and High Availability, Sean Derrington, assures me that their software is available now. Although no compatible arrays are in end-user hands, 3PAR will update their T-Class firmware to support the API shortly, and HDS and HP are on the way as well.

Thin Aware Software

Next, contrary to what I inferred from the announcement, there is no native thin provisioning capability in the file system or volume manager. So the first item in my list is right out. However, the volume manager is now “thin aware”, which means that it will communicate up to the file system and down to the array to coordinate more effective use of space.

When the volume manager is used with Veritas File System (VxFS) on UNIX or NTFS on Windows Server 2003 or 2008, it will automatically keep track of deleted files and will pass this information down the stack to the array. This is a major piece of functionality to add, especially to NTFS, “hole punching” (like NetApp) to maximize thin provisioning.

The Storage Foundation tools have also been updated to properly report on thin provisioned volumes. For example, the following screenshot shows three disk devices where encl1 supports thin reclamation and encl0 does not.

# vxdisk list
DEVICE        TYPE   DISK          GROUP         STATUS
encl0_0       auto   encl0_0       mydg online   thin
encl1_0       auto   encl1_0       mydg online   thinrclm
encl1_1       auto   ecnl1_1       mydg online   thinrclm

Thin Reclamation API

The Veritas Thin Reclamation API allows the Storage Foundation volume manager and file systems to communicate with thin-capable arrays when data is deleted on thin-ified LUNs, maintaining their thin-ness as you go. When a file is deleted, the file system will communicate to the volume manager that that space is no longer needed. When the server administrator runs the “vxdisk reclaim” or “fsadm –R” commands, the volume manager will communicate this information to the array (using SCSI commands) that any vacated disk blocks can now be reclaimed. Symantec expects folks to set up a cron job to reclaim space, or perhaps just run it when they see the need.

This is brilliant stuff, and ought to make thin provisioning shine in terms of array utilization. In an environment of thin-enabled Veritas volumes and supported storage arrays, the amount of space used on an array will be awfully close to the amount of space used in the file systems. This is a massive win - a capacity gain of on the order of 50%-70% in an average environment!

For more on this topic, see my recent post on storage utilization

If the storage array fully supports Symantec’s API, the tools will also report physically allocated storage behind thin and thin_reclaim devices.

# vxdisk –o thin list
DANAME        DISK SIZE(Mb)        PHYS_ALLOC(Mb)       DISK GROUP TYPE
encl0_0       2000                 50 mydg              thin
encl1_0       200                  50 mydg              thinrclm
encl1_1       500                  500 mydg             thinrclm

SmartMove

SmartMove is Symantec’s new capability for online migration from “thick” to thin LUNs. It is included in Storage Foundation for Unix/Linux and Windows and works with any thin storage array, not just those that support the API. This is basically a tweak to the old storage migration support we have all known and relied on in Veritas Storage Foundation for over a decade, except that it’s smart enough to not request blocks that it won’t use. One could theoretically “SmartMove” a volume regularly to reclaim space without using the API at all, but those commands are sure a lot simpler.

Note that SmartMove speeds up migration too, even for thick volumes! When you use a SmartMove-enabled version of Storage Foundation to move a volume, it will only send the blocks that have changed over the wire. This reminds me a little of VMware’s new I/O deduplication capability talked about at VMworld, but it’s focused only on migrations, not other I/O situations.

For more on this topic, see my recent post on VMware vStorage

The Plot Thickens

So I was wrong about one item, but the other two remain true. Is Symantec’s new capability a winner? I give it a silver medal – it’s good stuff, but some issues remain.

1. My primary concern remains – thin provisioning does nothing to address the lack of storage management that is so prevalent. It enables greater utilization of capacity, but does nothing to control how that capacity is used. This isn’t a beef with Symantec’s Veritas Storage Foundation or 3PAR or HDS or EMC or anyone in the thin industry, really. Instead, it is a wake-up call to all of the storage organizations out there who have filesystems full of uncontrolled junk!
2. My second concern is the lack of capacity management. Thin provisioning is a lie, promising more capacity than is available. This might be acceptable in certain controlled circumstances like operating system or application volumes, but telling end users that they have plenty of available space is a recipe for disaster. Storage use is like air – it expands to fill all available volume. Without capacity management, your thin volumes will be “overdrawn” and your storage “account” will be bankrupt.
3. Then there is the issue of proprietary APIs versus standards. Let me say right away that I always support standards over proprietary technology. But, at the same time, given the choice between nothing and something, I’ll take the proprietary API. Thin provisioning is a good idea with poor implementation. This API helps to make it useful in the real world, and having a market leader like Symantec behind it makes it all the more relevant. I certainly hope the entire storage industry will come up with a standard thin API, and when that happens I hope Symantec will support it. Until then, at least we have something.

I will be writing more about thin provisioning in the coming weeks. Until then, I continue to applaud Symantec, 3PAR, HDS, and HP for their work in making this technology somewhat more practical. Now how about VMware, Microsoft, Sun, and the Linux guys get some thin technology going, too?

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Symantec’s Thin API: The Plot Thickens
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I’ve been in the trenches on storage content (aka data) for a long time. I, too, have often reverted to the old “gigs of MP3s and porn” argument from time to time. But I’ve done enough filesystem assessments at real companies to realize that that’s not really the norm. In fact, I’ve rarely found much porn, music, video, or jokes on full-up corporate file servers. And I’ve analyzed enough storage environments to know that, while file servers are big, they’re not normally the majority user of storage in large data centers.

On the contrary, most enterprise storage is taken up by business applications, though not necessarily critical data. Email, backup, and certainly user file servers are big space users. But give me a few Oracle instances, source code repositories, or image processing servers, and watch those applications shrink in significance.

No matter what the application, though, the real issue with storage growth (and ILM) is the (in)ability of IT managers to do anything about it. Let’s say we had permission to delete really inappropriate data, which is not a sure thing. Would we IT folks even be able to recognize it? How would we locate it? Can we even view user files without violating user trust, company privacy policies, or even laws? Many countries (yes, not all data is in the USA), regulate access to data even inside a company.

Now let’s move into grayer areas of “unnecessary” corporate data. Many storage administrators can’t even name the applications that take up all that space, let alone understand the intricacies of the data under management.  To make a timely (and tired) Harry Potter analogy, IT are the house-elves of the business – powerful but subservient, with little input into what happens above and around them.  I’ve talked to business people who don’t want IT to have any input, relegating them to order takers and laborers.

This is a dangerous slide, however.  Lots of people have the capability to take IT orders and keep the lights on,  a realization that leads to outsourcing.  IT pros must prove their worth to the business in order to remain relevant and irreplaceable!

ILM is one way to do that.  To get back to Chuck’s post, we need to take the reins and try to understand data better.  We need to pick certain applications that lend themselves to automated data classification and tiered storage and try to get them under control.  Email is a great candidate, and that’s why email archiving applications have taken off recently.  File servers are coming along, too, especially with file virtualization in the ascendancy.

I’m particularly excited about what a smart IT manager I know called the “second wave” of SRM tools.  Rather than just collecting stock metadata (age, name, owner, etc), the latest filesystem scanning tools look inside a file, trying to better classify them.  Let’s say 1/4 of your file server is made up of Microsoft Word, Excel, and PowerPoint documents.  What can you do about that unless you can identify which are critical and which are not?  Each business will have its own criteria, and you need a flexible tool to scan them all and report back to you before you can “ILM” them.  That’s what lots of software vendors are currently working on, and though we’re at an early stage still, the results are promising.

Sadly, though, we in IT may soon find that we just can’t delete anything.  Even totally banned content like porn could be critical to a legal case against an employee,  and it won’t be long before we are expected to keep everything that shows up on our servers for a very long time.  Most companies have policies for hardcopy document retention, and many are currenyly diving into the world of data policy as well.  The default policy may be “keep until we decide what to do with it”, and this could cause the current trend of storage growth to accelerate!

If we can’t delete data, we will be forced to sail the Titanic rather than sink it.  Small companies can benefit most from the falling price of storage, since the entire storage footprint for a little shop is often under a terabyte.  But larger organizations will find that they need to start tiering their storage, and quickly in order to keep prices under control.

And then there’s green storage.   Again, Mr. Toigo makes the very valid point that the problem is in the business, not in the hardware we use.  But if we can’t do anything about data growth for the time being, we had better start tackling the technical challenges we face.  I’ve talked to many IT folks who are very worried about data center space, as well as the terrifying trio of heat, power, and cooling.  For them, green technologies are no laughing matter!  If you can’t get any more power, you have to lower your per-GB requirement and quickly.

It’s easy to say “understand your data and delete some”, but hard for IT pros to  actually do it.  Until we can tackle the strategic issue of data growth, we’ll have to continue fighting the tactical problems of storage.

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Sailing the Titanic (Why We Need ILM and Then Some!)
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.