The good news is that this is no longer a technical problem: Modern virtualized and networked servers ought to have decent utilization of storage capacity, and technology is improving all the time. Consider the compounded impact of modern technology on storage capacity utilization:

• Shared storage (SAN and NAS) allows different servers to share a common pool of storage, reducing the likelihood that excess capacity will be stranded in isolated “puddles”. Pervasive use of NAS technology, and the rise of simple and inexpensive iSCSI SANs, means that every system in the modern data center can use shared storage.
• Organizational and architectural optimization allows storage to be provisioned from a common pool rather than building “stovepipe systems” with their own resources. Quicker provisioning also helps reduce over-provisioning.
• Network connectivity allows servers to share resources, including storage, on a peer-to-peer or client-server basis, ultimately resulting in things like cloud computing.
• Managed and utility services reduce the impact of low utilization, potentially focusing on efficiency or perhaps passing the buck to a service provider.
• Thin provisioning might help certain systems to keep less storage in reserve.

So why don’t things get better? It’s hard to be sure why people don’t use these pervasive tools to improve storage utilization, but I do have some ideas…

• Storage utilization might not be a priority. Utilization isn’t often in the critical path of performance or availability, so overtaxed IT departments aren’t going to focus on it.
• Incentives can be lacking. With the cost of storage constantly falling, the effort required to improve the efficiency of already-allocated storage can be just as easily spent migrating to a newer, cheaper storage platform.
• Virtualization has perversely harmed the efficiency of allocation. One might think that the ease and flexibility of virtual disks would improve things, but it hasn’t. Server and storage virtualization just adds another place to hide unused storage.
• Metrics remain a problem, since everyone gets all balled up trying even to talk about capacity utilization.

I think this last point is something we in the industry really ought to do something about. We say “utilization” but what do we mean? Chris Evans has proposed a set of metrics for the “storage waterfall“, and I mentioned back in October that this all boils down to three key metrics: Raw, usable, and used. The key question is where to apply them!

Way back before the 2001 bubble-burst, I managed professional services for a company called StorageNetworks. At that time, I was quite aggressive in pushing this same idea, even co-writing a whitepaper on the topic titled Measuring and Improving Storage Utilization. My co-author (Jonathan Lunt) and I recently reminisced about that paper, and we both agreed that everything in it still stands today, apart from the high dollar cost per gigabyte.

Each ratio in the storage waterfall can be diagnosed and improved

I suggest that the following key storage utilization ratios (taken directly from this paper) make just as much sense today as they did then:

• Array Overhead is the percentage of installed storage capacity that is not usable. Dividing Array Usable by Array Raw and subtracting that number from 100% yields the percent of overhead. Overhead here is usually due to the desired level of data protection (e.g. RAID, mirroring) rather than to poor management.
• Array Utilization is the percentage of usable array capacity that is allocated to hosts. It indicates the efficiency of storage deployment operations.
• Allocation Efficiency reflects the ratio of storage presented or allocated to hosts to the amount actually seen by them. In many mature environments this ratio is near 100% (i.e. all the storage allocated is being seen), but this ratio can be extremely difficult to determine. It relies on accurate measurements of both Array Used storage and Host Raw.
• Host Overhead reflects the amount of storage configured for use versus the amount the host can see. Since the Host Raw metric is a function of the storage administration team and the Host Usable a function of the systems administration team, this metric is a useful measurement of how well the two functions are cooperating. Data for this classification is collected from the host.
• File System Utilization is the amount of available file system space that actually contains data. File system utilization is familiar to most systems administrators. This metric is often shown in simple system commands like “df” on UNIX or “dir” on Windows. Data for this classification is collected from the host.
• Total Storage Utilization summarizes how well a company manages its storage assets across the entire business. This ratio is the default storage utilization metric used in publications and reflects the actual value an enterprise is deriving from its storage asset. Care is required in calculating this ratio to ensure that it accurately indicates utilization of the storage environment. Since the result of this ratio is often used in business cases and receives wide attention, it must be both logical and defendable.

To these, I would add another intermediate and optional set of virtualization metrics and ratios for environments with storage or server virtualization. One could also presumably add a higher-level set of application efficiency ratios as well.

In the paper, Jon and I also proposed three best practices to improve storage utilization:

1. Drive Array Utilization (Array Usable to Array Used) to greater than 90% (a storage administration responsibility)
2. Drive Allocation Efficiency: Bring Host Usable to be as close to Array Used as possible (a joint responsibility)
3. Drive Filesystem Utilization (“Host Usable to Host Used”) above 80% (a systems administration responsibility)

Go read the paper and let me know what you think. Are we still stuck in 2001?

This post can also be found on Gestalt IT: Storage Utilization Remains at 2001 Levels: Low!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Utilization Remains at 2001 Levels: Low!
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The gist is that email archiving can be an incredibly useful tool to manage legal risks and enable more effective discovery of message content. Electronic discovery is growing at an amazing clip, and most cases now include email messages – in fact, email is estimated to be 60% to 70% of all legal discovery in the United States today!

At the same time, companies are challenged with ineffective or un-enforced retention and litigation hold policies and rampant “underground archiving” of messages in offline PST files. And to make matters worse, the 2006 revisions to the Federal Rules of Civil Procedure drastically shortened the amount of time companies have to describe and produce electronic records! It’s simply impossible to begin recovering messages from backup tapes in response to legal requests – the only way forward is a proactive strategy based on email archiving.

If this piques your interest, grab a copy of that whitepaper, join me on Wednesday for the webinar, and check out some of my other email-related content. I’ll also be speaking on the topic at Storage Decisions in San Francisco, and would be happy to respond to your emailed questions any time!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Six Critical Steps For Managing Email E-Discovery
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I’ve been working on a new whitepaper for Hitachi focused on privacy laws in the United States and Europe, and what we mere storage folks can do to help comply with them. It’s pretty amazing stuff – the American system is totally different from the European, no surprise. Where they have sweeping top-down rights to privacy and consistent standards, we have bottom-up point laws, a patchwork of state laws, and self-regulation. I tell you, as someone who cares about privacy (and avoiding identity theft and credit fraud), it makes me want to move to Europe!

But seriously, without scooping the paper (which I’ll link to when it’s published), if you’re interested in learning more about privacy laws, I highly recommend the Privacy Journal‘s Compilation of State and Federal Privacy Laws, pictured at right. It’s $31, and loaded with data. Also, check out the Electronic Privacy Information Center web site, another excellent source of information!

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Complying with Data Privacy Laws
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I’m also putting together a list of all the articles and whitepapers I’ve written.  Kind of the ultimate toot toot list!

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Toot toot: Ten Tips For Smarter Email Archiving Whitepaper
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.