Thin provisioning is a topic that is interesting to me. Now, I have to warn you ahead of time that I’ve been a critic of thin provisioning for a long, long time. So, you’re really not going to get a big cheer leader “rah rah” session here.

I feel like too many people use technologies like thin provisioning more for political and management problems than technical problems. But recently I have come around to seeing that thin provisioning has good uses. It has some positive aspects. But, I’ve also discovered a lot about how it really is and how it really works, and it’s not quite as wonderful as people like to say.

But, let’s start at the beginning. Why are we doing this at all? Why do we care about thin provisioning? This is why: Storage is not getting cheaper.

If you went to buy a disk ten years ago, you’re going to spend about the same as would today, but you’re going to get a lot more capacity – a lot more capacity!

This is a big fallacy of enterprise storage. I used to call it the CompUSA problem, but then they went out of business. So, I guess I’ll have to call it the Best Buy problem. The idea that I can go into a store and buy two terabytes of disk for two $100 bills and have change left over.

Well, that’s an issue for enterprise storage, right? Because what happens when you try to actually do real enterprise storage provisioning in a corporation? You end up with people pushing back on you saying, “Hey, wait! How come you’re charging me so much? For the price that I’m paying per month in internal chargebacks I can just go buy this capacity. And I have to pay that every single month?”

I hear this again and again from a lot of the pundits (people who don’t actually do anything in storage) who say, “ah, it’s just a spinning disk. All the enterprise guys they’re bending you over and taking advantage of you, and they are taking too much money out of your pocket for all this.”

Well, the truth is that probably a lot of them are overpricing. But, in my opinion, the cost of capacity is basically irrelevant. That doesn’t mean it’s zero, but it’s basically irrelevant. The reason is that everything else about storage is growing so rapidly that the dropping cost of disk capacity really isn’t helping.

The fact is that we have terrible utilization of enterprise resources anyway, and it’s not getting any better. In fact, most companies today have about the same storage utilization that they did ten years ago. Somewhere between 20 and 30 percent, if you look soup to nuts.

It hasn’t improved because they are “doing storage” the same way. If you’ve got file systems and you’ve got SCSI and you’ve got block storage and all that kind of stuff, guess what? Your utilization is going to be just as bad now as it was then because the technology just isn’t there.

And often it’s the business processes: There is a big problem with how provisioning, forecasting, purchasing, and the rest is done. So, storage costs just keep rising, and the dropping cost of disk capacity is only a minimal component.

Let’s talk about real-world cost. This is a graph that I made for one customer, and it doesn’t necessarily represent everybody. I’m not IDC. I don’t have some kind of universal numbers. I don’t know if they have the universal numbers either, but I definitely don’t.

This is what I found. With hardware and software, it was less than 10 percent of the entire cost of delivering storage services.

What does that mean? That means that if you reduce the cost and hardware and software by 50 percent (which would be a tremendous win by anybody’s standards) you’ve then affected your cost of storage by just five percent.

That’s basically irrelevant. You work really hard to do all this stuff and you’re really not getting anything.

For the next segment of this discussion, we’ll take a look at storage utilization.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Storage is Not Getting Cheaper
This post was categorized as Computer History, Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

You’re Holding It Wrong

How can I make this fantastic offer without knowing anything about your environment and without including any asterisks? Simply because when it comes to data storage, in the immortal words of Steve Jobs, you’re holding it wrong. In 15 years of enterprise storage consulting, I have never seen a single environment using anything close to their total usable capacity. In fact, I’ve never seen an environment that was using even half of its usable capacity.

This makes my job easy. If you have a half dozen storage arrays with a total of 500 TB of raw storage, about 60% of that capacity will be usable (once you take RAID, spares, and other overhead into account). Of the remaining 300 TB of capacity, you’re probably only storing 70 TB of data if you’re like the average enterprise shop. So I can just hand you a report that says “write 35 TB more” and walk out. My job will be done.

But let’s say you’re better than average. Let’s say you run a really tight ship and don’t waste expensive capacity like most people. I bet you’ve still got slack capacity you don’t know about. Maybe a project manager demanded a 30 TB LUN for his new database and won’t let you run your monitoring tools to see what he’s really using. Or perhaps another project never got off the ground but they won’t share the disk space “they paid for.” Then there is that other system that was turned off without you knowing, so the storage is still allocated. There’s always plenty of perfectly-good free usable primary storage capacity.

Wringing Out the Slack

See my post, Use Process Solutions For Process Problems, Technical Solutions For Technical Ones

Despite what the warring vendors might say, the issue isn’t the equipment or software you’re using, the issue is the way you’re using it. Storage isn’t bought as an integrated piece of a compute environment these days, and it isn’t managed that way, either. Enterprise storage arrays are purchased in fits and starts, a little here and a lot there, according to the whims of the budget and project planning process. It’s not at all unusual to see tight storage constraints delaying projects even as a new and totally unused array sits idle in the corner.

The root cause lies with how capacity is purchased, configured, allocated, and charged to projects, not with the technical capabilities of the platform. Nearly every modern array can be shared by many servers, and nearly every environment has ample storage networking potential. Are Fibre Channel directors and HBAs to expensive? Switch to iSCSI or NAS! Every server has a spare gigabit Ethernet port or two, and I bet your networking guys have a decent switch you could use.

All this applies mainly to primary storage, but backups are an equal opportunity. Most daily incremental backup tapes are left half-empty due to job scheduling, connectivity, and inappropriate manual media assignments. And those jam-packed weekly full tapes are probably a waste of time and capacity, too. How about re-thinking your backup process with fewer fulls, virtual tape, elimination of useless data, or even snapshots? I bet my friend W. Curtis Preston could offer some great advice there!

Stephen’s Stance

You don’t need to get crazy to wring out a bit more storage capacity. Deduplication and data optimization sound great, but what’s the point if you’ve got ample unused capacity already? Aren’t all these guarantees just an attempt to grab more business, more money, and sell more gear?

The leading cause of poor storage capacity utilization is failure to use storage capacity!

I’m serious about the offer. I’ve done exactly this kind of work before and have the resources to do it for you, too. Bring me in and I’ll give you a plan to write 50% more primary data. Guaranteed success or you don’t pay. But I bet you could do the same thing without me!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Stephen Foskett’s 50% Free Capacity Guarantee!
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• Capacity increases exponentially based on the formula for the area of a disc: π times radius squared
• Sequential performance increases algebraically based on the formula for the circumference of a disc: π times diameter

Therefore, sequential performance grows smoothly with disk density, but capacity increases much faster. Double the density of disk media and you can read twice as many bits in the same amount of time, but the disk now contains four times as much data. Iterate this a dozen times, a miracle performed regularly by hard disk drive manufacturers, and you have a serious bottleneck to both performance and reliability.

Disk capacity has outpaced performance over the last decade

Back in 2004, I gave this metric a name: Flush time. It is a simple calculation to answer the question, how long would it take to read the entire content of a hard disk drive? Let’s look at some real-world examples:

• In 2000, a 45 GB Western Digital 450AA disk could stream data at 25.4 MB/s, requiring 30 minutes to flush every byte out its UDMA/66 interface. This was a massive and slow drive at the time – enterprise disks were much faster. A 2000 Quantum Atlas 10K II SCSI drive (36 GB and 31 MB/s) could flush in 19 minutes!
• A 2004-era Seagate Barracuda 7200.7 boasted 160 GB ad averaged 44.5 MB/s, requiring about an hour for a full flush.
• By 2007, high-performance drives like the Hitachi 15K450 had hit 450 GB and about 100 MB/s in sustained throughput, but flush times were well over an hour.
• Today’s enterprise drives can push 200 MB/s and average 160 MB/s across the entire 600 GB of capacity. But this is still about an hour for a flush. But large-capacity SATA drives are much more popular for bulk storage. The Samsung Spinpoint F2 EcoGreen drive I use in my Drobo only delivers about 110 MB/s, requiring almost four hours to flush at 1.5 TB of capacity! Think this is unusual? Check out Hitachi’s popular E7K1000, which needs 2.5 hours at 1 TB and 118 MB/s.

What will happen to flush time over the next half decade if density continues to increase? How about 16 TB drives, 400 MB/s, and RAID rebuilds that last more than half a day?

Since (traditional) RAID rebuilds are directly impacted by flush time, today’s massive disk drives are killing RAID. And flush time is only the minimum required time – most RAID rebuilds take much longer! Then there is the issue of media reliability!

Note: Yes, I know there are alternative RAID schemes that get around this problem. Far from ignoring that point, I’ll be promoting these in future posts! Stay tuned for more on these topics…

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Flush Time
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Challenges for Enterprise IT

I’ve spent 15 years focused on the business of IT, with a special interest in data storage, since it makes up such a large part of modern information systems and budgets. The current crisis has forced many of us in enterprise IT to focus inward, tackling the tough challenges inside our data centers rather than adding on exciting new capabilities. The priority has shifted from features with a potential for cost avoidance to real current-year cost savings.

Even as budgets are tightening, we continue to pour good money after bad. This happens everywhere, buying too much of what we don’t want or need.

• As the high end marches ever-higher, we find that most applications require lower tiers of capability
• Data center managers are scrambling to control power and cooling costs
• At the same time, utilization of existing resources remains low, especially storage systems

But change is in the air. Enterprise IT is questioning their old assumptions.

• The monolith of IT infrastructure is finally becoming a pyramid, with a small amount of super high-end “tier-0″ capability and a widening reliance on lower tiers of bulk capacity
• Virtualization of servers and storage is having a real impact, enabling mobility of systems, applications, and data across tiers
• Enterprise IT is increasingly offloading non-essential services to outside partners who can deliver focus and economics

A New Challenge for Me

I have always tried to focus my efforts on making a difference in my little corner of the world, so the time has come for me to make a change professionally.

I am pleased to announce that I am joining Nirvanix, where I will be starting a new strategic consulting practice. We will focus on enabling enterprise customers to leverage modern data storage technology for infrastructure transformation. Nirvanix shares my vision of evolved IT infrastructure supported by tiers of applications, servers, and especially storage. Nirvanix moving forward and attacking the storage issues that face enterprise IT head-on!

I pride myself on giving honest and straightforward advice, and this will not change. Nirvanix Consulting will deliver independent and trustworthy advice, and this is exactly what enterprise IT organizations require and demand. Although Nirvanix provides managed offsite storage as a service to enterprise customers, this offering is not the correct answer for every application or business. In discussing this business concept with Nirvanx CEO, Jim Zierick, I stressed that I will certainly sometimes recommend competing solutions, and that the right move for some is reuse of existing assets. Zierick, who was formerly a partner at the respected consultancy, McKinsey, agrees with this independent approach. We must help each client to develop a strategy that is appropriate to their needs.

Although I have a new professional affiliation, my independence and commitment to the storage community will not change. I remain committed to open communication, and will maintain my Pack Rat blog, Gestalt IT, and my other public writing and speaking activities. Now let’s get down to work!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Changing Times Demand Focus
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Disk Inventory X is an amazing tool to zoom into your full disk and figure out what's taking up all the space!

I’m not an inherently tidy person. That’s one reason that I upgraded my MacBook Pro’s hard disk to 320 GB! But as a storage and data management consultant, I generally keep my computers much cleaner than my office.

So I was surprised to see a pop-up window on my Mac telling me that my hard drive was full! It was time to investigate data management applications for OS X. Luckily, I found some great tools, and these helped me isolate the space-wasting items quickly and easily.

This post is part of my series focused on Apple OS X tips and tricks.

• Access NTFS Volumes On Your Mac
• Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
• How To Move OS X Time Machine Backups To A New Disk
• Hallelujah! OS X Can Reduce PDF File Size!
• Custom Drive Icons in Mac OS X
• OS X Custom Drive Icons 2: Boot Camp and NTFS

Who’s To Blame?

When it comes to data storage, out of sight is usually out of mind. The primary culprit in the rampant over-saving of data is a lack of perspective and visualization: We tend to focus on the items we think are at fault rather than the true space wasters.

Consider your PC or Mac: Do you really know what is taking up all of your disk space? Unless you have a good tool, the answer is probably not. You might think that your “Documents” folder takes up most of the room, since you use it all the time and it has so many files in it. But even the bloated files produced by Microsoft Office pale in comparison to multimedia photo, music, and video files. And it is usually the folders that you don’t actively manage that are the worst space-wasters!

All operating systems have their secret corners that fill up with cruft, and many applications do a terrible job of cleaning up after themselves. In my PC days, I used to use Glary Utilities to clean out the corners of Windows and the awesome treemapping visualizations in WinDirStat to find where my storage was going. But since switching to my Mac I wasn’t aware of similar utilities.

Mapping the Tree

Let’s back up for a minute. Treemapping is an incredibly slick way to quickly zoom in on capacity problems. It divides up a rectangular region into color-coded blocks sized according to some factor for easy visualization. In this case, every file on a hard disk is represented according to its size and colorized by type. See the image at the top of the post for an example, using data from my laptop’s OS X drive.

I created this map with the awesome free Disk Inventory X application, which scans any drive connected to an OS X machine (even a NFS or CIFS share) and creates an interactive treemap. GrandPerspective is a similar app, but I like DI better after trying both.

One glance at the treemap and the eye is drawn to the large aqua file (my iPhoto library, all 25 GB of it) and the multitude of blue and red video files. File types are listed on another inspector, ordered in descending order of size, allowing me to see that I had more than 100 GB of TV shows and movies on my drive, and these made up the bulk of my “missing” space!

Click on a block, and DI displays the file as well as its location in the tree. We can then navigate up and down to locate other areas of wasted space, and DI highlights the current file or folder in the treemap to give a visual representation of what it contains.

Cruft

I found lots of junk (cruft) stored on my Mac:

• The Downloads folder seems like a junk magnet. I tend to open downloaded items from Safari’s Downloads window rather than using Finder, but when you select Remove From List there, it leaves the file in the Downloads folder. Even if you try to police your Safari download junk you can miss a few files. So this has to be the first place to look for cruft in OS X!
• I love how OS X installs apps - in many cases with a simple drag and drop to the Applications folder. But regardless of the install method, old dmg, iso, and pkg files seem to find their way onto my disk. Most of them are in Downloads, but I have found ‘em in lots of places.
• iTunes is particularly egregious, keeping every downloaded iPhone update package in ~/Library/iTunes/iPhone Software Updates. I had 1.3 GB of files sitting there! It’s a good idea to keep the current one around in case you need to restore your phone, but the old ones are just junk. This applies equally in Windows, by the way!
• When you double-click on a zip file in Finder, it expands the whole thing wherever you are, unlike Windows which lets you browse the zip without expanding it. I found quite a few of these weird folders all over my Documents folder, where I had opened a zip to have a look and left behind the entire contents.
• I keep my desktop clean, but I bet most people have at least a few gigs of junk hanging out there!
• VMware, Parallels, and VirtualBox create massive virtual disk files, and these can often auto-grow, becoming space vampires without your knowledge. Be careful if you’re using these apps!
• I’ve got two complete sets of my mail folders created by OS X Mail. I’m not sure why, but am shy to actually delete one.
• Don’t forget your external USB and FireWire drives, too! I deleted over 500 GB of redundant data from four external drives with a total capacity of just 1.3 TB!

Like any computer, Macs have to be periodically cleaned up. I’m keeping my eyes on the above areas using Disk Inventory X. What works for you?

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The good news is that this is no longer a technical problem: Modern virtualized and networked servers ought to have decent utilization of storage capacity, and technology is improving all the time. Consider the compounded impact of modern technology on storage capacity utilization:

• Shared storage (SAN and NAS) allows different servers to share a common pool of storage, reducing the likelihood that excess capacity will be stranded in isolated “puddles”. Pervasive use of NAS technology, and the rise of simple and inexpensive iSCSI SANs, means that every system in the modern data center can use shared storage.
• Organizational and architectural optimization allows storage to be provisioned from a common pool rather than building “stovepipe systems” with their own resources. Quicker provisioning also helps reduce over-provisioning.
• Network connectivity allows servers to share resources, including storage, on a peer-to-peer or client-server basis, ultimately resulting in things like cloud computing.
• Managed and utility services reduce the impact of low utilization, potentially focusing on efficiency or perhaps passing the buck to a service provider.
• Thin provisioning might help certain systems to keep less storage in reserve.

So why don’t things get better? It’s hard to be sure why people don’t use these pervasive tools to improve storage utilization, but I do have some ideas…

• Storage utilization might not be a priority. Utilization isn’t often in the critical path of performance or availability, so overtaxed IT departments aren’t going to focus on it.
• Incentives can be lacking. With the cost of storage constantly falling, the effort required to improve the efficiency of already-allocated storage can be just as easily spent migrating to a newer, cheaper storage platform.
• Virtualization has perversely harmed the efficiency of allocation. One might think that the ease and flexibility of virtual disks would improve things, but it hasn’t. Server and storage virtualization just adds another place to hide unused storage.
• Metrics remain a problem, since everyone gets all balled up trying even to talk about capacity utilization.

I think this last point is something we in the industry really ought to do something about. We say “utilization” but what do we mean? Chris Evans has proposed a set of metrics for the “storage waterfall“, and I mentioned back in October that this all boils down to three key metrics: Raw, usable, and used. The key question is where to apply them!

Way back before the 2001 bubble-burst, I managed professional services for a company called StorageNetworks. At that time, I was quite aggressive in pushing this same idea, even co-writing a whitepaper on the topic titled Measuring and Improving Storage Utilization. My co-author (Jonathan Lunt) and I recently reminisced about that paper, and we both agreed that everything in it still stands today, apart from the high dollar cost per gigabyte.

Each ratio in the storage waterfall can be diagnosed and improved

I suggest that the following key storage utilization ratios (taken directly from this paper) make just as much sense today as they did then:

• Array Overhead is the percentage of installed storage capacity that is not usable. Dividing Array Usable by Array Raw and subtracting that number from 100% yields the percent of overhead. Overhead here is usually due to the desired level of data protection (e.g. RAID, mirroring) rather than to poor management.
• Array Utilization is the percentage of usable array capacity that is allocated to hosts. It indicates the efficiency of storage deployment operations.
• Allocation Efficiency reflects the ratio of storage presented or allocated to hosts to the amount actually seen by them. In many mature environments this ratio is near 100% (i.e. all the storage allocated is being seen), but this ratio can be extremely difficult to determine. It relies on accurate measurements of both Array Used storage and Host Raw.
• Host Overhead reflects the amount of storage configured for use versus the amount the host can see. Since the Host Raw metric is a function of the storage administration team and the Host Usable a function of the systems administration team, this metric is a useful measurement of how well the two functions are cooperating. Data for this classification is collected from the host.
• File System Utilization is the amount of available file system space that actually contains data. File system utilization is familiar to most systems administrators. This metric is often shown in simple system commands like “df” on UNIX or “dir” on Windows. Data for this classification is collected from the host.
• Total Storage Utilization summarizes how well a company manages its storage assets across the entire business. This ratio is the default storage utilization metric used in publications and reflects the actual value an enterprise is deriving from its storage asset. Care is required in calculating this ratio to ensure that it accurately indicates utilization of the storage environment. Since the result of this ratio is often used in business cases and receives wide attention, it must be both logical and defendable.

To these, I would add another intermediate and optional set of virtualization metrics and ratios for environments with storage or server virtualization. One could also presumably add a higher-level set of application efficiency ratios as well.

In the paper, Jon and I also proposed three best practices to improve storage utilization:

1. Drive Array Utilization (Array Usable to Array Used) to greater than 90% (a storage administration responsibility)
2. Drive Allocation Efficiency: Bring Host Usable to be as close to Array Used as possible (a joint responsibility)
3. Drive Filesystem Utilization (“Host Usable to Host Used”) above 80% (a systems administration responsibility)

Go read the paper and let me know what you think. Are we still stuck in 2001?

This post can also be found on Gestalt IT: Storage Utilization Remains at 2001 Levels: Low!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Utilization Remains at 2001 Levels: Low!
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.