• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Enterprise storage / How I Learned to Stop Worrying and Love Storage Automation

How I Learned to Stop Worrying and Love Storage Automation

February 9, 2009 By Stephen 10 Comments

The first storage performance horseman is spindles: If you don’t have enough disk units, performance will suffer. I have been laying out storage on enterprise arrays since the dark ages, and one of the first lessons I learned was allocating data to avoid hotspots. I remember spending hours back in the 1990’s hunched over custom Excel spreadsheets trying to get my storage layout just right, balancing the workload across every available disk.

This is how we used to avoid hotspots in 1998: Carefully planning every detail of the storage layout.
This is how we used to avoid hotspots in 1998: Carefully planning every detail of the storage layout.

Each disk drive consists of a spindle of spinning platters with read/write heads move back and forth. Each time you access a piece of data that’s not in cache, the drive moves its arm over the platter to access the correct piece of data. Since each drive can only access one piece of data at once, and since caches can only hold so much data, tuning a system to minimize the number of requests per drive is essential.

Manual storage array layout was an art, but we never fooled ourselves into thinking our designs were optimal. There were just too many intractable problems, so we had to compromise at every turn:

  • We usually had no performance data to base our layout decisions on, so we had to rely on guesses and rules of thumb
  • Workloads tend to change over time and manual layouts are painful to modify
  • The smallest unit of allocation was an entire LUN or drive, so even the best disk layout mixed hot and rarely-accessed data everywhere
  • Much of the allocated space was unused, so we used expensive disks to store nothing

One might think that, 10 years later, advances in technology would have solved these basic issues. But for many people using many of the so-called modern mainstream enterprise storage systems, these problems remain.

Like all good systems administrators, I’m a natural control freak. I am uncomfortable letting the system manage itself, having been burned too many times by computers (well, software really) making stupid decisions. It’s analogous to the backlash against anti-lock brakes, traction control, and automated transmissions among racing enthusiasts.

Do we allow technology to help us get better performance, or do we try to micro-manage everything?
Do we allow technology to help us get better performance, or do we try to micro-manage everything? Photo by ClearInnerVision

But the time has come to let go. We don’t have to micro-manage storage anymore, and we have much to gain by letting the array do the work:

  • Just as traction control can manage each wheel independently, something a driver could never do, modern virtualized storage systems can allocate small “chunks” to the optimal drive type, creating a better layout than anyone could manage with LUNs
  • Dynamic optimization technology can move these chunks around, adapting as loads change
  • Thin provisioning can go a step further, not wasting drive capacity for unused space
  • Wide striping and post-RAID storage systems have a higher threshold before performance suffers due to spindle hotspots
  • Widespread availability of tiered storage, including advanced caches, solid state drives, high-performance SAS and FC, and cheap bulk disks, gives us many more options

As I mentioned, not all systems have these capabilities, and not all implementations are created equal. I’m concerned about misuse of thin provisioning, for example, but it’s hard to argue with its effectiveness in many circumstances. Find out how granular your system’s allocation is – some remain LUN-only, while others are much more effective, using tiny chunks.

These new storage automation technologies really become essential once high-dollar flash storage is added to the mix. If you’re paying 30 times more for a flash drive, you want to make sure you’re making the best use of it that you can! Look at IBM’s recently-announced SAN Volume Controller (SVC) and solid state drive (SSD) combination, for example: It will almost certainly have fine-grained thin provisioning of SSDs, and should be able to dynamically move data between flash and disk storage and even between different storage arrays, but I still have questions on how granular this capability will be. HDS can do similar things with their USP-V. NetApp’s V-Series NAS systems will do dynamic allocation, thin provisioning, and data deduplication to enable a better return on the flash drive investment. I’d love to see 3PAR, Compellent, Dell/EqualLogic, and HP/LeftHand apply their solid dynamic allocation tech to solid state storage as well!

Then there’s the 800 lb gorilla: EMC. More enterprise SSD has probably been shipped out of Hopkinton than every other vendor combined, and both the CX and DMX support (optional/expensive) “virtual provisioning” (aka, thin provisioning) of flash storage. But EMC’s Optimizer is not widely used, and only migrates entire LUNs based on user input – hardly the kind of dynamic and granular technology needed to optimally use all of that flash storage. I’m sure the company is working on addressing this issue, though. Perhaps it will appear in the DMX-5 announcement we are all expecting this year?

This article can also be found on Gestalt IT: How I Learned to Stop Worrying and Love Storage Automation

You might also want to read these other posts...

  • Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!
  • Electric Car Over the Internet: My Experience Buying From…
  • How To Connect Everything From Everywhere with ZeroTier
  • How To Install ZeroTier on TrueNAS 12
  • Introducing Rabbit: I Bought a Cloud!

Filed Under: Enterprise storage, Gestalt IT, Virtual Storage Tagged With: 3PAR, Compellent, CX, deduplication, Dell, DMX, DMX-5, EMC, EqualLogic, flash, HDS, HP, IBM, LeftHand, NetApp, optimization, Optimizer, performance, RAID, SSD, SVC, thin provisioning, tiered storage, USP-V, V-Series, virtual provisioning, wide striping

Primary Sidebar

The movie never changes. It can’t change; but every time you see it, it seems different because you’re different. You see different things. – 12 Monkeys

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

How To Install ZeroTier on TrueNAS 12

February 3, 2022

Scam Alert: Fake DMCA Takedown for Link Insertion

January 24, 2022

How To Connect Everything From Everywhere with ZeroTier

January 14, 2022

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Symbolic Links

    Featured Posts

    Ten Terrible Apple Products

    June 14, 2012

    Hands-On Review: Unicomp Spacesaver M Keyboard for Mac

    July 3, 2012

    Virtualized and Distributed Storage: This Time For Sure!

    September 2, 2014

    How To Sync Your iPad With Your Exchange Server

    April 3, 2010

    Defining Failure: What Is MTTR, MTTF, and MTBF?

    July 6, 2011

    Follow the Yellow Brick Road to the Software-Defined Future

    November 29, 2012

    The Ideal pfSense Platform: Netgate RCC-VE 2440

    September 21, 2015

    What More Could Alan Turing Have Accomplished?

    October 7, 2012

    10 Mysteries The Lost Finale Definitively Settled

    May 25, 2010

    It’s Time To Move Beyond Passwords (Especially On Web Sites)

    January 8, 2016

    Footer

    Legalese

    Copyright © 2022 · Log in