• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Enterprise storage / Scaling Storage In Conventional Arrays

Scaling Storage In Conventional Arrays

November 19, 2013 By Stephen 3 Comments

Clustering sounds great, but it’s awfully taxing to keep all the nodes consistent!

It is amazing that something as simple-sounding as making an array get bigger can be so complex, yet scaling storage is notoriously difficult. Our storage protocols just weren’t designed with scaling in mind, and they lack the flexibility needed to dynamically address multiple nodes. Data protection is extremely difficult and data movement is always time-consuming.

This is part of a series on “Scale-Out” Storage Field Day 4

Three Ways To Scale Storage

Traditionally, there have been three ways to scale storage, each with its own pros and cons:

  1. Scale-up array: Add more storage behind a single controller “head”, which acts as the termination point for client I/O. Since anything behind the head is invisible to the client anyway, scaling up can be non-disruptive. But there are limits to the amount of data a single head can control, and performance can suffer as the front- and back-end controllers, CPU, and memory become saturated.
  2. Scale-out cluster:  As the array scales, the controller “head” becomes increasingly critical, leading most vendors to cluster arrays for high availability. Most use a “shared-everything” cluster design, with each controller able to “peek into” every other controller’s RAM and caches, ensuring data consistency. But it’s difficult to maintain this “mind meld” between clustered heads beyond a handful of members.
  3. Scale-out gateway: Recently, many storage vendors have adopted a two-tier architecture with a true scale-out object store on the back end and one or more protocol gateways in the middle. The client talks to the gateway using a conventional protocol like iSCSI or NFS; the gateway handles data distribution; and the back end provides scale, data protection, and consistency.

All three of these traditional scale-out architectures have proven their worth in production over the past decade, and all are actively in development by “next-generation” storage vendors.

Four Examples From Storage Field Day 4

At Storage Field Day 4, the delegates heard about scale-out clusters from CloudByte, Overland Storage, and Nimble Storage, all of which can scale up as well as out.

There are many other scale-out cluster solutions, with market leaders like NetApp, EMC Isilon, and Dell’s Compellent and EqualLogic serving as familiar examples.

CloudByte

CloudByte’s scale-out architecture joins multiple nodes into a cluster, sharing storage in the cluster using ZFS. But CloudByte adds an additional layer of abstraction via what they call a tenant storage machine (TSM), which can be moved from cluster to cluster on demand. In this way, they transcend traditional clustering limitations, but data locality and client access is limited to a single cluster within the larger pool.

Overland Storage

Overland Storage scales out using a gateway driver running on their scale-up SnapScale storage nodes. Client I/O is wide striped across “peer sets” of disks located on nodes throughout the cluster. This novel approach allows the SnapScale cluster to grow while balancing storage across every node in the cluster, though it is not clear to me how they rebalance data as the cluster grows. Client I/O is balanced using round-robin DNS to distribute client connections, a simple but inflexible approach.

Nimble Storage

Nimble Storage also scales up and out by slicing all data and distributing it equally across the storage in a pool of nodes. Data is rebalanced in the background as the cluster grows, using what I would call a “lazy” algorithm to avoid performance impact. Currently, Nimble only supports four arrays in a single pool, but they promise that this number will grow over time. Since Nimble uses the iSCSI protocol exclusively, they rely on a host-side MPIO driver to allow parallel and highly-available client access across nodes.

Avere Systems and Cleversafe

Storage Field Day 4 delegates also learned about the scale-out gateway offering from Avere Systems and Cleversafe, who are working together to deliver such a solution. In the past, Storage Field Day 3 saw the launch of Exablox, which sells an integrated solution which includes both a scale-out object store and NAS gateway, but Avere and Cleversafe are focused solely on the gateway and object store, respectively. The Cleversafe distributed storage net (dsNet) platform is a massively-scalable object quite unlike any conventional scale-out storage array. Placing an Avere NAS gateway in front of the dsNet allows high performance NFS and SMB access to the scalable, distributed object store.

Note that Avere’s gateways can be used in front of any NFS or SMB storage system, so this concept isn’t limited to Cleversafe.

Coho Data

There was one more conventional-protocol scalable storage system at Storage Field Day 4: Coho Data. Coho pushes the scaling work into the network in a rather clever way. I’ll cover that in more detail in the future, but for now here’s their tech presentation so you can see how it works!

Stephen’s Stance

Scaling storage is hard. Really hard. Especially when you’re not able to change the client driver or protocol. So my hat is off to these companies and others who have come up with clever ways to maintain compatibility while scaling out beyond the bounds of a single storage array. Next time I’ll write about another approach: Scaling storage by changing the client or protocol!

You might also want to read these other posts...

  • How To Connect Everything From Everywhere with ZeroTier
  • Electric Car Over the Internet: My Experience Buying From…
  • Introducing Rabbit: I Bought a Cloud!
  • Scam Alert: Fake DMCA Takedown for Link Insertion
  • Tortoise or Hare? Nvidia Jetson TK1

Filed Under: Enterprise storage, Features, Virtual Storage Tagged With: Avere, Cleversafe, CloudByte, Coho Data, gateway, Nimble Storage, Overland Storage, scale-out, Scale-up, Storage Field Day, Storage Field Day 4, Tech Field Day

Primary Sidebar

My favorite things in life don’t cost any money. It’s really clear that the most precious resource we all have is time.

Steve Jobs

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

How To Install ZeroTier on TrueNAS 12

February 3, 2022

Scam Alert: Fake DMCA Takedown for Link Insertion

January 24, 2022

How To Connect Everything From Everywhere with ZeroTier

January 14, 2022

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Symbolic Links

    Featured Posts

    How Fast Is It? A Storage Infographic

    October 29, 2010

    Cisco’s Trojan Horse

    September 15, 2014

    Why Are PCIe SSDs So Fast?

    June 12, 2013

    Ranting and Raving About the 2018 iPad Pro

    November 11, 2018

    On the Death of Innovation, or “These Kids These Days!”

    May 21, 2012

    We Live in the Future: Robotic Cat Litter Boxes!

    May 8, 2010

    Thoughts on the Modern Miracle of 3D Printing

    July 28, 2015

    Why Buy a NEX-7? Why Sony NEX At All?

    October 17, 2011

    Frequent Flier Kung Fu for Novices

    March 12, 2012

    vSphere 6: NFS 4.1 Finally Has a Use?

    February 3, 2015

    Footer

    Legalese

    Copyright © 2022 · Log in