• Skip to main content
  • Skip to primary sidebar
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Enterprise storage / Despite Appearances, SpectraLogic’s BlackPearl Isn’t An Amazon Glacier Clone

Despite Appearances, SpectraLogic’s BlackPearl Isn’t An Amazon Glacier Clone

October 17, 2013 By Stephen Leave a Comment

How’d you like a petabyte of S3-like web app storage?

This week, SpectraLogic announced DS3 and “BlackPearl”, an innovative product for tape storage using a cloud API. Although BlackPearl sounds like an Amazon Glacier clone, it’s really nothing of the sort. BlackPearl extends the S3 API for tape storage but this “DS3” API requires well-behaved clients and disciplined access. BlackPearl is exciting, it’s novel, and it’s useful. But it’s not S3 or Glacier, despite what some initial coverage may say.

There’s lots of great detail on DS3 and BlackPearl in this post by Ray Lucchesi

Amazon designed S3 to be massive, flexible, and always online. Typical users keep multiple buckets “open” all the time – reading and writing to a bucket per application or app profile and always assuming they’ll be available. This is one of the core values of S3, and in fact I use it to host all the images you see on this very blog!

SpectraLogic’s S3 protocol extension, known as “DS3” (“Deep Simple Storage Services”), extends the basic S3 protocol with a bunch of commands intended to make it work with tape. In addition to the standard S3 CRUD (“create, read, update, delete”), DS3 includes commands to load and unload buckets (tape sets) and put and get (read and write) bulk objects. This last bit is critical, since it’s important to keep tape drives streaming data.

Spectra also introduced an appliance instantiation of DS3. Known as “BlackPearl” (CamelCase, no space), this front-end for a Spectra library allows a DS3 (or even plain S3) client to read and write to tape.

BlackPearl has some flash storage inside, but this is a buffer rather than a cache or tier. Let me explain:

  • A storage tier is the final landing spot for data. It can be permanent or actively managed.
  • A cache is an alternate location for data that is stored elsewhere.
  • A buffer is really ephemeral – it just serves to hold data momentarily while the real storage is unavailable.

In BlackPearl, the flash buffer allows data to be ordered for streaming to tape. It never holds data longer than necessary, and mainly serves to “coalesce” bulk writes. No writes are “committed” until the data is safely written to tape, and nothing is stored in the buffer after this, so this definitely isn’t a cache.

As a new product, BlackPearl is necessarily limited. It can’t handle more than a few open buckets at once – since each bucket maps to one or more tapes, and the appliance can’t address more than 4 drives, the current maximum number of open buckets is 4. And it can’t handle too many active reads or writes at once, either, since they have to be buffered and written immediately. In fact, BlackPearl has a clever mechanism to keep unserviceable PUT and GET operations alive while waiting for tape loads – it just issues a 300 redirect to itself every 30 seconds or so!

Although BlackPearl implements DS3, and DS3 is a superset of S3, one shouldn’t just go out and hitting Spectra’s new baby with S3 applications. They will probably function (after all, the S3 protocol “just works”) but probably not in a satisfactory manner. Nearly every S3 application I know of would choke at the limited availability of buckets and concurrent I/O operations, though they’d likely manage to keep alive.

1,000 LTO-5 tapes = 1.5 petabytes of storage. No kidding.

Stephen’s Stance

DS3 and BlackPearl is intended as an entirely new frontier for web application storage. It shouldn’t take too much effort and time to develop applications with real DS3 integration assuming the developer can get over the concept of buckets as tape sets. Used properly, BlackPearl will offer massive (sequential) performance, exabyte-level scalability, and eye-popping cost savings over disk.

I would love to see an open-source Glacier alternative built around BlackPearl. Heck, Amazon should just rewrite Glacier internally to use a Spectra library rather than the disks we presume they’re using today! This is exciting stuff!

You might also want to read these other posts...

  • Electric Car Over the Internet: My Experience Buying…
  • GPS Time Rollover Failures Keep Happening (But…
  • Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!
  • What You See and What You Get When You Follow Me
  • Introducing Rabbit: I Bought a Cloud!

Filed Under: Enterprise storage, Virtual Storage Tagged With: Amazon, Amazon Glacier, Amazon S3, BlackPearl, CRUD, DS3, S3, SpectraLogic, tape

Primary Sidebar

A man provided with paper, pencil, and rubber, and subject to strict discipline, is in effect a universal machine.

Alan Turing

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!

May 29, 2020

What You See and What You Get When You Follow Me

May 28, 2019

GPS Time Rollover Failures Keep Happening (But They’re Almost Done)

April 6, 2019

Symbolic Links

    Featured Posts

    Why I Am Biased Against FCoE

    October 21, 2011

    Scaling Storage At The Client

    November 25, 2013

    Review: American Standard’s Champion 4 Toilet Flushes Almost Anything

    July 31, 2012

    How To Sync Your iPad With Your Exchange Server

    April 3, 2010

    Introducing Rabbit: I Bought a Cloud!

    September 10, 2020

    It’s Time To Speak Out Against Sexism In IT Recruiting

    May 6, 2013

    How to Get Me to Write about Your Company or Product

    March 15, 2012

    Cisco’s Trojan Horse

    September 15, 2014

    Top VMware Blogs 2014: How I Voted

    February 25, 2014

    Faster Ethernet Gets Weird

    June 19, 2015

    Copyright © 2021 · Log in