Symantec's newly-announced "V-Ray" technology does ... something ... maybe

Data is getting bigger, virtualization is expanding, and data protection applications are ill-prepared to deal with it. This much we can all agree on. But Symantec’s introduction of “V-Ray,” which the company describes as “X-Ray vision into … virtual environments” has just left me puzzled. Is this “marketecture” or some sort of technology or product?

Symantec’s V-Ray Vision

As discussed at Symantec Vision 2011, there is a distinct need within virtualized environments to improve visibility and transparency. As additional layers are added, each is obscures those below it. Virtual machines simply lack the level of visibility seen in physical environments.

The solution, we are told, is “V-Ray.” It “provides transparency of backup images across physical and virtual environments,” which certainly sounds like a positive goal. And Symantec can leverage its “intellectual property around file systems, security, and storage.” But what exactly is Symantec doing here?

I had numerous discussions with Symantec folks at Vision 2011, and it turns out that V-Ray is, in fact, a term for a number of technological features common to many products. First delivered in NetBackup and Backup Exec, the technology known as V-Ray allows these products to identify files within virtual machine images, enabling file-level recovery. It will also be leveraged by Symantec’s Endpoint Protection security products, allowing quicker scanning when the product “knows” which files have already been scanned and which have changed.

Put together, these features will allow these products to interact more efficiently and completely, sharing configuration information and other metadata. The V-Ray concept will extend into management applications and across Symantec’s software portfolio.

Symantec also made this clever video. Too bad it doesn’t say what exactly V-Ray is!

Stephen’s Stance

Anything that enables better management of virtual machines is a win in my book, but I wish this “V-Ray” idea wasn’t so opaque. I’d love a single page specifying exactly which technologies fall under this umbrella and that this “deep technology” really does.

Symantec has great IP for managing storage and applications as well as protecting data, but they haven’t always been able to leverage and communicate this technology. V-Ray is a step in the right direction conceptually, merging their storage, backup, and security smarts and spreading the result far and wide. But right now it appears to be more “marketecture” than real substance. Here’s hoping it matures into some solid, useful offerings in time for Symantec Vision 2012!

Disclosure: Symantec paid my expenses to attend Symantec Vision 2011 and has repeatedly sponsored Tech Field Day and other activities I am involved with. I am under no obligation to the company to write about their products, positive or negative.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | What Exactly Is Symantec V-Ray?
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Thin provisioning needs communication to function, and zero page reclaim is only the array side of the story. WRITE_SAME helps reduce I/O load, but the server needs to use it. Wouldn’t it be nice if the operating system, file system, or volume manager would use these commands to help recover capacity?

We need a bridge. We need to cross the gulf between the storage array (that expects all these zeros or some other way to recover this space), and the host (who has this information)

That’s the thing that gets me riled up. The application and the file system know that this data is no longer used. If there was only some way, some bridge, that allowed them to talk to the storage array and say, “Hey, you can get rid of that now. I don’t need it anymore.” See? We need a bridge.

Well, guess what? We have some bridges. One of my favorite developments in storage of the last year is this Thin Reclamation API that Symantec has. I really like what they’re doing.

Symantec (formerly Veritas) have the file system and the volume manager. If you’re an enterprise guy like me (I was using HP-UX in 1996) you have used the Veritas File System and Veritas Volume Manager before.

Symantec calls all this Veritas Storage Foundation now. It has the information thin provisioning storage arrays need, and they’ve come up with a way to communicate: Being a software company, Symantec implemented just about every communication method, because not everybody uses this WRITE_SAME, and not everybody uses zero page reclaim. Some of them have their own API for recovery. So Symantec tried to implement as many thin methods as they possibly could right in the file system and volume manager.

I’ve talked to a lot of people who are using it, and they say that it works. They say that it’s brilliant. Because you turn on this feature and, suddenly, the file system starts telling the array “I don’t need this data anymore.” Suddenly one of my major complaints about thin provisioning goes away.

But I have a question: Why doesn’t everybody do this? Why isn’t this in the Linux file systems and volume managers? Why isn’t Microsoft working on this? Why doesn’t everybody have this capability? It’s not rocket science. It just takes a little bit of development effort to try to come up with some kind of way to have the file systems talk.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Bridge: Veritas Thin (Provisioning) API
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

In the previous post, I talked about how the Drobo uses metadata monitoring to solve the telephone game and make de-allocation possible. But that approach is challenging in complex enterprise environments. Instead, most enterprise arrays use a complex chain of semaphores to interpret signals from the connected hosts about the capacity that can be un-provisioned.

On the storage side, arrays can only use the information they have to de-allocate: The data that’s stored on them. They don’t know what application is using it, what file system it is. They don’t know anything at all.

But, somewhere along the line, someone had a big idea and said, “wait a second, what if we look for pages that are all zeros?” We’ll talk about pages a bit later, but for now, let’s talk about zeros. A zero is kind of a smoke signal coming up from over the hills that says, “there’s nothing valuable here.”

So the storage array watches for pages that are all zero and reclaims them. As protection against making a stupid mistake (what if you actually wanted to write all zeros?), anybody who asks for a page that has been reclaimed just gets all zeros back.

Most of the major vendors support this kind of zero page reclaim. This is good stuff. I don’t want to sound too critical of them because I appreciate them implementing at least this.

The problem is that there’s not a lot of ability to actually have those zeros be written. Almost no operating system writes zeros to deleted space. If they actually wrote pages of zeros, thin provisioning would work great.

So what do the storage vendors do? They come up with utilities that write zeros!

NetApp has SnapDrive, which zeros out empty space so that the Filer can go and recover that space. You run it whenever you want to run it. Eventually the storage array notices that you’ve zeroed out that space and it recovers it. Compellent and Symantec’s Veritas Storage Foundation have something like that, too. You can also force it using the SDelete command, and you can configure it using VMware ESX.

Zero page reclaim is pretty straightforward. It doesn’t take a lot of computing power – It’s not like you’re watching the file system for changes or anything. All you’re doing is occasionally going through and deleting pages full of zeros. So, you can post-process it, kind of like de-duplication.

There are quite a few issues with zero page reclaim, though:

• Things aren’t writing zeros
• Most of these implementations are page-based, which looks like a problem
• Theoretically, this drives more IO through the system, not less

This last is the biggest problem, really. In most cases IO performance is a bigger issue than capacity in enterprise storage. If I could give you all the capacity you could possibly want or all the performance you could possibly want, most people would pick performance. It used to be capacity, but now it’s all about performance. If infrastructure folks could get one for free and had to pay for the other, they would definitely pay for performance.

And zero page reclaim, the way that it’s implemented with SDelete or with eagerzeroedthick, is driving tons of IO. Basically, a delete is the same as a write because you have to write all these zeros over the bus. But there’s a way around that, too. And that’s the topic for the next piece in this series.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Zero Page Reclaim: Savior of Thin Provisioning?
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

First, let’s clarify just which products and capabilities Symantec is offering here:

• Veritas Storage Foundation version 5.0MP3 for Unix/Linux includes SmartMove and the Thin Reclamation API
• Veritas Storage Foundation for Windows 5.0 only includes SmartMove at this point, but it will be updated to include Thin Reclamation at some point in the coming year

Although there is no real information on Symantec’s web site about all this yet, Symantec’s director of Storage Management and High Availability, Sean Derrington, assures me that their software is available now. Although no compatible arrays are in end-user hands, 3PAR will update their T-Class firmware to support the API shortly, and HDS and HP are on the way as well.

Thin Aware Software

Next, contrary to what I inferred from the announcement, there is no native thin provisioning capability in the file system or volume manager. So the first item in my list is right out. However, the volume manager is now “thin aware”, which means that it will communicate up to the file system and down to the array to coordinate more effective use of space.

When the volume manager is used with Veritas File System (VxFS) on UNIX or NTFS on Windows Server 2003 or 2008, it will automatically keep track of deleted files and will pass this information down the stack to the array. This is a major piece of functionality to add, especially to NTFS, “hole punching” (like NetApp) to maximize thin provisioning.

The Storage Foundation tools have also been updated to properly report on thin provisioned volumes. For example, the following screenshot shows three disk devices where encl1 supports thin reclamation and encl0 does not.

# vxdisk list
DEVICE        TYPE   DISK          GROUP         STATUS
encl0_0       auto   encl0_0       mydg online   thin
encl1_0       auto   encl1_0       mydg online   thinrclm
encl1_1       auto   ecnl1_1       mydg online   thinrclm

Thin Reclamation API

The Veritas Thin Reclamation API allows the Storage Foundation volume manager and file systems to communicate with thin-capable arrays when data is deleted on thin-ified LUNs, maintaining their thin-ness as you go. When a file is deleted, the file system will communicate to the volume manager that that space is no longer needed. When the server administrator runs the “vxdisk reclaim” or “fsadm –R” commands, the volume manager will communicate this information to the array (using SCSI commands) that any vacated disk blocks can now be reclaimed. Symantec expects folks to set up a cron job to reclaim space, or perhaps just run it when they see the need.

This is brilliant stuff, and ought to make thin provisioning shine in terms of array utilization. In an environment of thin-enabled Veritas volumes and supported storage arrays, the amount of space used on an array will be awfully close to the amount of space used in the file systems. This is a massive win - a capacity gain of on the order of 50%-70% in an average environment!

For more on this topic, see my recent post on storage utilization

If the storage array fully supports Symantec’s API, the tools will also report physically allocated storage behind thin and thin_reclaim devices.

# vxdisk –o thin list
DANAME        DISK SIZE(Mb)        PHYS_ALLOC(Mb)       DISK GROUP TYPE
encl0_0       2000                 50 mydg              thin
encl1_0       200                  50 mydg              thinrclm
encl1_1       500                  500 mydg             thinrclm

SmartMove

SmartMove is Symantec’s new capability for online migration from “thick” to thin LUNs. It is included in Storage Foundation for Unix/Linux and Windows and works with any thin storage array, not just those that support the API. This is basically a tweak to the old storage migration support we have all known and relied on in Veritas Storage Foundation for over a decade, except that it’s smart enough to not request blocks that it won’t use. One could theoretically “SmartMove” a volume regularly to reclaim space without using the API at all, but those commands are sure a lot simpler.

Note that SmartMove speeds up migration too, even for thick volumes! When you use a SmartMove-enabled version of Storage Foundation to move a volume, it will only send the blocks that have changed over the wire. This reminds me a little of VMware’s new I/O deduplication capability talked about at VMworld, but it’s focused only on migrations, not other I/O situations.

For more on this topic, see my recent post on VMware vStorage

The Plot Thickens

So I was wrong about one item, but the other two remain true. Is Symantec’s new capability a winner? I give it a silver medal – it’s good stuff, but some issues remain.

1. My primary concern remains – thin provisioning does nothing to address the lack of storage management that is so prevalent. It enables greater utilization of capacity, but does nothing to control how that capacity is used. This isn’t a beef with Symantec’s Veritas Storage Foundation or 3PAR or HDS or EMC or anyone in the thin industry, really. Instead, it is a wake-up call to all of the storage organizations out there who have filesystems full of uncontrolled junk!
2. My second concern is the lack of capacity management. Thin provisioning is a lie, promising more capacity than is available. This might be acceptable in certain controlled circumstances like operating system or application volumes, but telling end users that they have plenty of available space is a recipe for disaster. Storage use is like air – it expands to fill all available volume. Without capacity management, your thin volumes will be “overdrawn” and your storage “account” will be bankrupt.
3. Then there is the issue of proprietary APIs versus standards. Let me say right away that I always support standards over proprietary technology. But, at the same time, given the choice between nothing and something, I’ll take the proprietary API. Thin provisioning is a good idea with poor implementation. This API helps to make it useful in the real world, and having a market leader like Symantec behind it makes it all the more relevant. I certainly hope the entire storage industry will come up with a standard thin API, and when that happens I hope Symantec will support it. Until then, at least we have something.

I will be writing more about thin provisioning in the coming weeks. Until then, I continue to applaud Symantec, 3PAR, HDS, and HP for their work in making this technology somewhat more practical. Now how about VMware, Microsoft, Sun, and the Linux guys get some thin technology going, too?

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Symantec’s Thin API: The Plot Thickens
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.