• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Computer History / Granularity: The Hidden Challenge of Storage Management

Granularity: The Hidden Challenge of Storage Management

April 19, 2011 By Stephen 2 Comments

How granular is your storage? This question is just as relevant today as when I first published this article in 2008!

Many storage challenges focus on correlating high-level uses of data (such as applications) with the nuts and bolts of storage infrastructure. These discussions often revolve around the conflict between data management, which demands an ever-smaller unit of management, and storage management, which benefits most from consolidation. Developing data management capability that is both granular enough for applications and scalable enough for storage is one key to the future of storage.

Storage Management: Scaling Up

As I discussed in a previous Sunday Series piece, Turning the Page on RAID, the data storage industry has traditionally focused on reducing granularity. Disk capacity has expanded, and RAID technology has multiplied this by combining multiple physical drive mechanisms into a single virtual one. Storage virtualization technologies, from the SAN to the server, have also often been touted primarily as a mechanism to reduce heterogeneity. From a technical perspective, therefore, granularity has been an obstacle to overcome.

The core organizational best practice for storage management is the reduction of complexity and the enforcement of standardization. Consolidation of storage arrays and file servers is a common goal, as IT seeks to benefit from economies of scale. The goal of both initiatives is the creation of a storage utility or managed storage service. This mirrors efforts on the server and network sides to consolidate and virtualize hardware.

Although both technological and organizational factors have traditionally driven granularity out of storage, this does not have to be the case. Virtual pools of storage are ideal for providing storage on demand, as disk-focused RAID groups give way to more flexible sub-disk storage arrangements. And an operational focus on standardized storage service offerings has the potential to enable scalable management of these smaller units.

Filing Service

File-based protocols would seem to have more potential for granular storage management, but they have been undermined by the hierarchical nature of modern file storage. Whether the connection to a file server uses NFS, CIFS, or AFP, the key unit of management is actually the shared directory, not the file. All files in the share \\firefly\backups would be located on the same server and would be managed as a unit.

NAS virtualization can change this somewhat, as can more specialized NAS servers. Although Microsoft DFS enables consolidation and virtualization of NAS shares, it does not allow subdivision of shares below the directory level – all files in a directory must be placed on the same server. Tricks like stubbing and links allow for some movement, but these do not solve the core issue. Specialized virtual NAS devices from F5 (the ARX, nee Acopia), NetApp, BlueArc, Symantec, and others have the ability to move files individually, providing as much a virtualized storage environment as any block-focused enterprise array. Avere is also beginning to talk about granular file management.

But even an ideal virtualized file server lacks the kind of granularity demanded by users. They care about data, not files, and most applications consolidate their data storage into a few files. Consider a database, for example, where users want each record treated uniquely but storage devices see just a few much larger files. We need a storage revolution, where someone creates an ideal storage platform in which each individual record or object includes custom metadata and is managed independently. This would truly be a massive change, however, and it is not clear that all applications will follow the object storage model of Google and Amazon.

Small is Beautiful

Barring a revolution in data management, our best hope is to allow greater granularity in storage management. As mentioned above, virtualization technology has the potential to enable management and protection of any unit of storage, right down to the individual block or record. But the reality of storage virtualization has not matched its promise.

What is needed is greater integration. Each layer of virtualization (file system, volume manager, hypervisor, network, array, and RAID) also hides necessary details from lower layers. Consider the case of a virtual server snapshot: The application and filesystem must be in a quiesced state to allow a snapshot to be taken at the storage level, but the storage array has no intrinsic information about how its capacity is used. A given LUN might contain dozens of servers on a shared VMFS volume, so all must be snapped together.

Integration can be enabled by sharing more information through APIs. VMware leverages Microsoft Volume Shadow Copy Service (VSS) integration for shared storage so a VMFS snapshot can call the operating system and even applications (Windows Server 2003 only, for now) to prepare the data. Similarly, VSS can communicate directly with supported iSCSI and Fibre Channel arrays, calling a snapshot at the right moment. And Microsoft is, no doubt, enhancing VSS as we speak.

As virtualization technology matures, expect this type of integration to improve. We hope to see more APIs exposed by VMware and Microsoft, allowing communication up and down the stack to break through the information barrier. Imagine a future where a standard API like VSS can pass a message through VMware, Xen, and Hyper-V to the underlying storage array to initiate a snap. I predict that this kind of integration-enabled granularity is not too far off.

You might also want to read these other posts...

  • How To Connect Everything From Everywhere with ZeroTier
  • Electric Car Over the Internet: My Experience Buying From…
  • Introducing Rabbit: I Bought a Cloud!
  • Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!
  • Tortoise or Hare? Nvidia Jetson TK1

Filed Under: Computer History, Enterprise storage, Virtual Storage Tagged With: Acopia, AFP, Amazon, BlueArc, CIFS, data management, DFS, F5, Google, Hyper-V, Microsoft, NAS, NetApp, network storage, NFS, ONStor, RAID, SAN, storage area network, storage management, storage virtualization, Sunday series, updated, VMFS, VMware, VSS, Xen

Primary Sidebar

The same thing can be identified by many different terms, and the same term may mean many different things.

Douglas John Foskett

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

How To Install ZeroTier on TrueNAS 12

February 3, 2022

Scam Alert: Fake DMCA Takedown for Link Insertion

January 24, 2022

How To Connect Everything From Everywhere with ZeroTier

January 14, 2022

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Symbolic Links

    Featured Posts

    The End of Unlimited Data – Part 1: The Buffet

    June 2, 2010

    Go Get a ProtonMail Account and Protect Your Online Life!

    July 19, 2017

    How To Connect Everything From Everywhere with ZeroTier

    January 14, 2022

    VMware’s Hardware Partner Strategy: Heads I Win, Tales You Lose

    February 11, 2013

    Nimble Storage Rolls Out an All-Flash Array

    February 24, 2016

    Begun, the Patent Wars Have

    July 9, 2012

    Making a Case For (and Against) Software-Defined Storage

    January 9, 2014

    Storage Changes in VMware vSphere 5

    July 16, 2011

    How to Get Me to Write about Your Company or Product

    March 15, 2012

    Fasting to Mitigate Jet Lag: Surprise! It Works!

    February 11, 2013

    Footer

    Legalese

    Copyright © 2022 · Log in