• Skip to main content
  • Skip to primary sidebar
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Computer History / Monitoring Filesystem Metadata For Thin Provisioning

Monitoring Filesystem Metadata For Thin Provisioning

January 3, 2011 By Stephen Leave a Comment

One of the topics I've often written and spoken about is thin provisioning. This series of 11 articles is an edited version of my thin provisioning presentation from Interop New York 2010. I hope you enjoy it!

I began by introducing the core problem: Storage isn’t getting any cheaper due to storage utilization and provisioning problems. Thin provisioning isn’t all it’s cracked up to be, since the telephone game makes de-allocation a challenge. So now let’s talk about how to make thin provisioning actually work.

There are 100 different ways of solving the de-allocation problem, some of which have gained some prominence. They all boil down to two options:

  1. Make the server super-smart and have it communicate better
  2. Make the storage super-smart and have it make educated guesses

There’s only a few ways that the server-side option can be implemented, and we’ll get to that. But first, let’s take a look at a sort of hybrid approach that relies on known server usage patterns: Metadata monitoring.

It’s really hard for the storage to really understand what the server is doing. The best example that I know of is the Drobo sitting under my desk.

I love this little black box. When I got it, I configured it as eight terabytes and I put a 160-gig disk in it. That’s thin provisioning. And over time, I’m swapping out the disks and I’m doing all my stuff, and it still looks like eight terabytes. Add data, delete it, swap disks, and it always just works.

Not a lot of people know how the Drobo works, though. One of the things that people have complained about is that it only supports certain file systems and partition schemes. The reason for this is a “magical” thing it’s doing that relates very, very closely to the topic of this discussion. The Drobo is the first thin provisioning box that I know of that directly monitors the file system.

What the Drobo does is this: It knows where the supported filesystems (HFS+, NTFS, EXT3, and FAT) keep the record of what’s been deleted. So the Drobo it watches that spot and when you delete something, it reclaims that space. No enterprise storage system can do this, and yet this little box under my desk does it all day long.

This is basically the super, ultimate smarts of storage. But, of course, it’s very limited. It faces a real challenge in an enterprise setting because there is much more variety. We have all these layers of virtualization and weird file systems and things like that to worry about. We just can’t expect a product like this to accommodate everybody, so we just can’t expect this kind of smarts to be put everywhere.

Instead, we have a variety of semaphores sent from the server to the storage array that attempt to solve the telephone game. That’s what we’re talking about next.

You might also want to read these other posts...

  • Electric Car Over the Internet: My Experience Buying…
  • Ranting and Raving About the 2018 iPad Pro
  • GPS Time Rollover Failures Keep Happening (But…
  • Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!
  • Introducing Rabbit: I Bought a Cloud!

Filed Under: Computer History, Enterprise storage, Everything, Virtual Storage Tagged With: data robotics, Drobo, FAT, filesystem, HFS, Metadata, metadata monitoring, NTFS, thin provisioning

Primary Sidebar

If you wish to make an apple pie from scratch, you must first invent the universe

Carl Sagan

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!

May 29, 2020

What You See and What You Get When You Follow Me

May 28, 2019

GPS Time Rollover Failures Keep Happening (But They’re Almost Done)

April 6, 2019

Symbolic Links

    Featured Posts

    The End of Unlimited Data – Part 1: The Buffet

    June 2, 2010

    On the Death of Innovation, or “These Kids These Days!”

    May 21, 2012

    Marketers: Fudging the Meaning of Buzzwords Matters (To You!)

    December 2, 2015

    FCoE vs. iSCSI – Making the Choice

    May 20, 2011

    The iPhone Revolution 10 Years Later

    January 9, 2017

    GPS Time Rollover Failures Keep Happening (But They’re Almost Done)

    April 6, 2019

    From Kipling’s Dirigibles to the Jet Age

    May 13, 2012

    What More Could Alan Turing Have Accomplished?

    October 7, 2012

    It’s Fine To Mount Hard Drives On Their Side Or Even Upside-Down

    August 13, 2016

    A Complete List of VMware VAAI Primitives

    November 10, 2011

    Copyright © 2021 · Log in