Consider Storage Utilization

The effectiveness of data storage allocation and utilization is pathetic. If they really investigate the matter, most enterprise IT organizations will discover that they are using less than one quarter of the expensive disk storage capacity they have purchased. Yet storage has the largest data center footprint in terms of floor space, power and cooling, and ranks near the top in terms of operational and capital expense. In short, businesses spend excessively on storage systems that they get very little use out of, and no improvement has been made in a decade.

A combination of technical and process roadblocks impede higher utilization of storage capacity. The technical issues are many: Inflexibility of storage allocation (it’s hard to “grow” a file system or RAID set while it is in use); the lack of communication between the various levels touched by storage (applications, operating systems, networks, systems, disks); static mapping between files and storage systems; and the perverse interaction between capacity and performance (spindles and heads, platter capacity, interface speed). These are compounded by business process issues that are probably more serious: Problematic interactions between business users of IT systems, IT application managers, and IT infrastructure teams; opaque or downright fraudulent growth projections leading to ineffective capacity planning; excessive lead time for storage provisioning up and down the stack; and bulk purchasing due to vendor and customer financial and budget calendars. I’m sure the reader can imagine a few more drivers besides, but all conspire to limit effective utilization of storage.

I have a long-standing love/hate relationship with thin provisioning, one of the many proposed technical solutions to the utilization problem. Thin provisioning eliminates many technical challenges: It simplifies adding capacity to the Drobo that serves as my home office storage center; the ability to automatically grow VMware images makes virtualization practical in the tight confines of a laptop; and it contribute to the usefulness of advanced solid-state storage systems like the new Nimbus S-Class. But I have serious reservations about using thin provisioning to over-subscribe enterprise storage systems due to failures of capacity planning and IT-to-business communication. Thin provisioning will only make process issues worse.

Fix the Process

Thin provisioning is fine if your problem is a technical one, but enterprise IT should focus instead on the process. Why can’t IT communicate with the business? Application owners have so little faith in the ability of IT to respond to simple requests (“I need more storage soon”) that they overestimate their needs by 100% and hope to never have to have that conversation. Thin provisioning blunts the cost of this excess capacity but does nothing to improve the underlying problem.

As far as storage utilization goes, fixing the process starts with storage as a service. Once storage is standardized, it can be provisioned much quicker and smoother, and this will reassure application owners. It will also reduce the impact of bulk purchases and enable easier forecasting and management. I have seen enterprises build and deploy uniform “cookie-cutter” storage solutions and it really does work.

“As a service” methodology is difficult to implement, however. It takes guts to make it stick, and open-mindedness to even give it a try. But it’s the only way to fix the process.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Use Process Solutions For Process Problems, Technical Solutions For Technical Ones
This post was categorized as Computer History, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I'm pleased to have been invited to write and speak on storage topics for a decade

I’m pleased to announce that I will be presenting at all four of TechTarget’s excellent end user-focused 2009 Storage Decisions conferences in North America! I’m also very excited to be developning two entirely new sessions for the show:

1. Tools and Tricks to Manage Capacity – Knowing how much disk capacity you have allocated, how much is actually being used, and what remains are basics for disk storage management. But it can be difficult to measure and describe storage capacity and make smart and effective decisions. In this session, we’ll cover the tools and best practices that are key to managing disk capacity.
2. Radical Tiered Storage to Reduce Cost and Expand Service – “Do more with less” is the slogan of the year, but storage pros have been doing exactly this for years. From HSM to tiered storage, data is moving away from simple disks and landing on flash memory, tape, and cloud services. Adopting an expansive tiered storage policy will not only reduce costs but can also improve performance and introduce new capabilities.

The 2009 show dates have been released, with more information at the official Storage Decisions site. Admission is free for qualified attendees, so register now for the session near you!

• Chicago – June 1-2, 2009
• Toronto – June 16, 2009
• New York – September 22-23, 2009
• San Francisco – November 17, 2009

Long-time readers of this blog know that both topics are top of mind for me, and I’ll be updating this blog and Gestalt IT with content on both capacity management and tiered storage over the coming year as I progress through the development and presentation of this content.

I spent the last two years presenting on the topics of storage virtualization and email archiving, and previously presented a two-hour “storage 101″ session. Although I loved that content, I’m never satisfied with my own work, so I’ll be happy to develop these two new presentations from scratch!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Two New Storage Decisions Sessions for 2009: Capacity Management and Radical Tiered Storage!
This post was categorized as Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Disk Inventory X is an amazing tool to zoom into your full disk and figure out what's taking up all the space!

I’m not an inherently tidy person. That’s one reason that I upgraded my MacBook Pro’s hard disk to 320 GB! But as a storage and data management consultant, I generally keep my computers much cleaner than my office.

So I was surprised to see a pop-up window on my Mac telling me that my hard drive was full! It was time to investigate data management applications for OS X. Luckily, I found some great tools, and these helped me isolate the space-wasting items quickly and easily.

This post is part of my series focused on Apple OS X tips and tricks.

• Access NTFS Volumes On Your Mac
• Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
• How To Move OS X Time Machine Backups To A New Disk
• Hallelujah! OS X Can Reduce PDF File Size!
• Custom Drive Icons in Mac OS X
• OS X Custom Drive Icons 2: Boot Camp and NTFS

Who’s To Blame?

When it comes to data storage, out of sight is usually out of mind. The primary culprit in the rampant over-saving of data is a lack of perspective and visualization: We tend to focus on the items we think are at fault rather than the true space wasters.

Consider your PC or Mac: Do you really know what is taking up all of your disk space? Unless you have a good tool, the answer is probably not. You might think that your “Documents” folder takes up most of the room, since you use it all the time and it has so many files in it. But even the bloated files produced by Microsoft Office pale in comparison to multimedia photo, music, and video files. And it is usually the folders that you don’t actively manage that are the worst space-wasters!

All operating systems have their secret corners that fill up with cruft, and many applications do a terrible job of cleaning up after themselves. In my PC days, I used to use Glary Utilities to clean out the corners of Windows and the awesome treemapping visualizations in WinDirStat to find where my storage was going. But since switching to my Mac I wasn’t aware of similar utilities.

Mapping the Tree

Let’s back up for a minute. Treemapping is an incredibly slick way to quickly zoom in on capacity problems. It divides up a rectangular region into color-coded blocks sized according to some factor for easy visualization. In this case, every file on a hard disk is represented according to its size and colorized by type. See the image at the top of the post for an example, using data from my laptop’s OS X drive.

I created this map with the awesome free Disk Inventory X application, which scans any drive connected to an OS X machine (even a NFS or CIFS share) and creates an interactive treemap. GrandPerspective is a similar app, but I like DI better after trying both.

One glance at the treemap and the eye is drawn to the large aqua file (my iPhoto library, all 25 GB of it) and the multitude of blue and red video files. File types are listed on another inspector, ordered in descending order of size, allowing me to see that I had more than 100 GB of TV shows and movies on my drive, and these made up the bulk of my “missing” space!

Click on a block, and DI displays the file as well as its location in the tree. We can then navigate up and down to locate other areas of wasted space, and DI highlights the current file or folder in the treemap to give a visual representation of what it contains.

Cruft

I found lots of junk (cruft) stored on my Mac:

• The Downloads folder seems like a junk magnet. I tend to open downloaded items from Safari’s Downloads window rather than using Finder, but when you select Remove From List there, it leaves the file in the Downloads folder. Even if you try to police your Safari download junk you can miss a few files. So this has to be the first place to look for cruft in OS X!
• I love how OS X installs apps - in many cases with a simple drag and drop to the Applications folder. But regardless of the install method, old dmg, iso, and pkg files seem to find their way onto my disk. Most of them are in Downloads, but I have found ‘em in lots of places.
• iTunes is particularly egregious, keeping every downloaded iPhone update package in ~/Library/iTunes/iPhone Software Updates. I had 1.3 GB of files sitting there! It’s a good idea to keep the current one around in case you need to restore your phone, but the old ones are just junk. This applies equally in Windows, by the way!
• When you double-click on a zip file in Finder, it expands the whole thing wherever you are, unlike Windows which lets you browse the zip without expanding it. I found quite a few of these weird folders all over my Documents folder, where I had opened a zip to have a look and left behind the entire contents.
• I keep my desktop clean, but I bet most people have at least a few gigs of junk hanging out there!
• VMware, Parallels, and VirtualBox create massive virtual disk files, and these can often auto-grow, becoming space vampires without your knowledge. Be careful if you’re using these apps!
• I’ve got two complete sets of my mail folders created by OS X Mail. I’m not sure why, but am shy to actually delete one.
• Don’t forget your external USB and FireWire drives, too! I deleted over 500 GB of redundant data from four external drives with a total capacity of just 1.3 TB!

Like any computer, Macs have to be periodically cleaned up. I’m keeping my eyes on the above areas using Disk Inventory X. What works for you?

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The good news is that this is no longer a technical problem: Modern virtualized and networked servers ought to have decent utilization of storage capacity, and technology is improving all the time. Consider the compounded impact of modern technology on storage capacity utilization:

• Shared storage (SAN and NAS) allows different servers to share a common pool of storage, reducing the likelihood that excess capacity will be stranded in isolated “puddles”. Pervasive use of NAS technology, and the rise of simple and inexpensive iSCSI SANs, means that every system in the modern data center can use shared storage.
• Organizational and architectural optimization allows storage to be provisioned from a common pool rather than building “stovepipe systems” with their own resources. Quicker provisioning also helps reduce over-provisioning.
• Network connectivity allows servers to share resources, including storage, on a peer-to-peer or client-server basis, ultimately resulting in things like cloud computing.
• Managed and utility services reduce the impact of low utilization, potentially focusing on efficiency or perhaps passing the buck to a service provider.
• Thin provisioning might help certain systems to keep less storage in reserve.

So why don’t things get better? It’s hard to be sure why people don’t use these pervasive tools to improve storage utilization, but I do have some ideas…

• Storage utilization might not be a priority. Utilization isn’t often in the critical path of performance or availability, so overtaxed IT departments aren’t going to focus on it.
• Incentives can be lacking. With the cost of storage constantly falling, the effort required to improve the efficiency of already-allocated storage can be just as easily spent migrating to a newer, cheaper storage platform.
• Virtualization has perversely harmed the efficiency of allocation. One might think that the ease and flexibility of virtual disks would improve things, but it hasn’t. Server and storage virtualization just adds another place to hide unused storage.
• Metrics remain a problem, since everyone gets all balled up trying even to talk about capacity utilization.

I think this last point is something we in the industry really ought to do something about. We say “utilization” but what do we mean? Chris Evans has proposed a set of metrics for the “storage waterfall“, and I mentioned back in October that this all boils down to three key metrics: Raw, usable, and used. The key question is where to apply them!

Way back before the 2001 bubble-burst, I managed professional services for a company called StorageNetworks. At that time, I was quite aggressive in pushing this same idea, even co-writing a whitepaper on the topic titled Measuring and Improving Storage Utilization. My co-author (Jonathan Lunt) and I recently reminisced about that paper, and we both agreed that everything in it still stands today, apart from the high dollar cost per gigabyte.

Each ratio in the storage waterfall can be diagnosed and improved

I suggest that the following key storage utilization ratios (taken directly from this paper) make just as much sense today as they did then:

• Array Overhead is the percentage of installed storage capacity that is not usable. Dividing Array Usable by Array Raw and subtracting that number from 100% yields the percent of overhead. Overhead here is usually due to the desired level of data protection (e.g. RAID, mirroring) rather than to poor management.
• Array Utilization is the percentage of usable array capacity that is allocated to hosts. It indicates the efficiency of storage deployment operations.
• Allocation Efficiency reflects the ratio of storage presented or allocated to hosts to the amount actually seen by them. In many mature environments this ratio is near 100% (i.e. all the storage allocated is being seen), but this ratio can be extremely difficult to determine. It relies on accurate measurements of both Array Used storage and Host Raw.
• Host Overhead reflects the amount of storage configured for use versus the amount the host can see. Since the Host Raw metric is a function of the storage administration team and the Host Usable a function of the systems administration team, this metric is a useful measurement of how well the two functions are cooperating. Data for this classification is collected from the host.
• File System Utilization is the amount of available file system space that actually contains data. File system utilization is familiar to most systems administrators. This metric is often shown in simple system commands like “df” on UNIX or “dir” on Windows. Data for this classification is collected from the host.
• Total Storage Utilization summarizes how well a company manages its storage assets across the entire business. This ratio is the default storage utilization metric used in publications and reflects the actual value an enterprise is deriving from its storage asset. Care is required in calculating this ratio to ensure that it accurately indicates utilization of the storage environment. Since the result of this ratio is often used in business cases and receives wide attention, it must be both logical and defendable.

To these, I would add another intermediate and optional set of virtualization metrics and ratios for environments with storage or server virtualization. One could also presumably add a higher-level set of application efficiency ratios as well.

In the paper, Jon and I also proposed three best practices to improve storage utilization:

1. Drive Array Utilization (Array Usable to Array Used) to greater than 90% (a storage administration responsibility)
2. Drive Allocation Efficiency: Bring Host Usable to be as close to Array Used as possible (a joint responsibility)
3. Drive Filesystem Utilization (“Host Usable to Host Used”) above 80% (a systems administration responsibility)

Go read the paper and let me know what you think. Are we still stuck in 2001?

This post can also be found on Gestalt IT: Storage Utilization Remains at 2001 Levels: Low!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Utilization Remains at 2001 Levels: Low!
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Based on Floral Matryoshka by BrokenSphere/Wikimedia Commons

My February 2003 column for Storage magazine focused on the surprising difficulty of measuring storage utilization. I wrote:
 

“A true measurement of utilization would reflect every layer of usage metrics – from raw disk in a shared array to used storage within files. Raw storage for each new frame of reference is contained within the used storage measured above it, so low utilization is compounded as we move deeper into the stack.”

In that column, I suggested that utilization of any resource was based on just three metrics:

1. Raw
2. Usable
3. Used

But this is confounded by the frame of reference being measured. It’s trivially simple to determine the raw, usable, and used capacity for a storage array, server, or database. But what happens when one tries to measure storage utilization all the way through the stack?

When vendors take up this challenge, the discussion tends to get diverted into a cul-de-sac that presents their products most favorably, as was the case of Chuck Hollis’ comparison of his EMC CLARiiON to HP’s and NetApp’s storage products. Was Chuck wrong? Was HP right? Or was it NetApp that has the best utilization? One thing is certain, we’re getting nowhere if we can’t agree on some basic terminology.

Chris Evans' Storage Waterfall

Credit Storage Architect Chris Evans with seeing the problem for what it was. He noticed the matryoshka effect and put together a “waterfall” diagram, showing how low utilization is compounded as we move down the stack. He also notes that complexity rises as we move to the right, something I never called out.

We were both onto the same thing, though, and my study of storage utilization (published in the April 2003 issue) supported his suggestion that the raw to used ratio might be as little as 10:1 on average. At the time, I even put together a similar waterfall chart, but it was never published outside the company I worked for (that I know of).

So I fully and enthusiastically support Chris’ ideas on this topic! Let’s come up with some standard metrics for the various places that storage can be “raw, usable, and used”:

1. Disk drive units often have excess space (raw), and this is especially true of enterprise flash units
2. RAID sets definitely follow this pattern
3. Storage arrays themselves can have unused usable space (as noted by Marc Farley)
4. Storage virtualization can add another layer of utilization loss
5. On the host side, we must consider volume managers which can perform all the functions of an array
6. Filesystems also have raw, usable, and used space
7. As do applications that manage storage like databases
8. Add in capacity management technologies like compression and deduplication to really mess things up
9. Finally, server virtualization can sit above or below these server variables, and virtual machines themselves often have unused space.

Simply put, there are a lot of places for a few unused bytes to hide. Anyone want to bet that 10:1 is optimistic? And we’re only talking about capacity utilization – there are whole other worlds of power efficiency and performance to consider as well…

© sfoskett for Stephen Foskett, Pack Rat, 2008. | The Storage Utilization Waterfall: Raw, Usable, and Used
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I know, it’s just a study, and studies are flawed. I’ve done enough myself to know that it is difficult to ask the right questions to get clear answers. I do question these results though. Especially the “38% unused” number – my own surveys revealed a much higher number than this. But I guess if storage management is increasingly difficult, it’s also just as difficult to get good utilization numbers…

© sfoskett for Stephen Foskett, Pack Rat, 2007. | 50% of UK IT Managers Must Be Crazy!
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.