• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Enterprise storage / Despite Appearances, SpectraLogic’s BlackPearl Isn’t An Amazon Glacier Clone

Despite Appearances, SpectraLogic’s BlackPearl Isn’t An Amazon Glacier Clone

October 17, 2013 By Stephen Leave a Comment

How’d you like a petabyte of S3-like web app storage?

This week, SpectraLogic announced DS3 and “BlackPearl”, an innovative product for tape storage using a cloud API. Although BlackPearl sounds like an Amazon Glacier clone, it’s really nothing of the sort. BlackPearl extends the S3 API for tape storage but this “DS3” API requires well-behaved clients and disciplined access. BlackPearl is exciting, it’s novel, and it’s useful. But it’s not S3 or Glacier, despite what some initial coverage may say.

There’s lots of great detail on DS3 and BlackPearl in this post by Ray Lucchesi

Amazon designed S3 to be massive, flexible, and always online. Typical users keep multiple buckets “open” all the time – reading and writing to a bucket per application or app profile and always assuming they’ll be available. This is one of the core values of S3, and in fact I use it to host all the images you see on this very blog!

SpectraLogic’s S3 protocol extension, known as “DS3” (“Deep Simple Storage Services”), extends the basic S3 protocol with a bunch of commands intended to make it work with tape. In addition to the standard S3 CRUD (“create, read, update, delete”), DS3 includes commands to load and unload buckets (tape sets) and put and get (read and write) bulk objects. This last bit is critical, since it’s important to keep tape drives streaming data.

Spectra also introduced an appliance instantiation of DS3. Known as “BlackPearl” (CamelCase, no space), this front-end for a Spectra library allows a DS3 (or even plain S3) client to read and write to tape.

BlackPearl has some flash storage inside, but this is a buffer rather than a cache or tier. Let me explain:

  • A storage tier is the final landing spot for data. It can be permanent or actively managed.
  • A cache is an alternate location for data that is stored elsewhere.
  • A buffer is really ephemeral – it just serves to hold data momentarily while the real storage is unavailable.

In BlackPearl, the flash buffer allows data to be ordered for streaming to tape. It never holds data longer than necessary, and mainly serves to “coalesce” bulk writes. No writes are “committed” until the data is safely written to tape, and nothing is stored in the buffer after this, so this definitely isn’t a cache.

As a new product, BlackPearl is necessarily limited. It can’t handle more than a few open buckets at once – since each bucket maps to one or more tapes, and the appliance can’t address more than 4 drives, the current maximum number of open buckets is 4. And it can’t handle too many active reads or writes at once, either, since they have to be buffered and written immediately. In fact, BlackPearl has a clever mechanism to keep unserviceable PUT and GET operations alive while waiting for tape loads – it just issues a 300 redirect to itself every 30 seconds or so!

Although BlackPearl implements DS3, and DS3 is a superset of S3, one shouldn’t just go out and hitting Spectra’s new baby with S3 applications. They will probably function (after all, the S3 protocol “just works”) but probably not in a satisfactory manner. Nearly every S3 application I know of would choke at the limited availability of buckets and concurrent I/O operations, though they’d likely manage to keep alive.

1,000 LTO-5 tapes = 1.5 petabytes of storage. No kidding.

Stephen’s Stance

DS3 and BlackPearl is intended as an entirely new frontier for web application storage. It shouldn’t take too much effort and time to develop applications with real DS3 integration assuming the developer can get over the concept of buckets as tape sets. Used properly, BlackPearl will offer massive (sequential) performance, exabyte-level scalability, and eye-popping cost savings over disk.

I would love to see an open-source Glacier alternative built around BlackPearl. Heck, Amazon should just rewrite Glacier internally to use a Spectra library rather than the disks we presume they’re using today! This is exciting stuff!

You might also want to read these other posts...

  • Electric Car Over the Internet: My Experience Buying From…
  • How To Connect Everything From Everywhere with ZeroTier
  • Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!
  • Introducing Rabbit: I Bought a Cloud!
  • How To Install ZeroTier on TrueNAS 12

Filed Under: Enterprise storage, Virtual Storage Tagged With: Amazon, Amazon Glacier, Amazon S3, BlackPearl, CRUD, DS3, S3, SpectraLogic, tape

Primary Sidebar

If you wish to make an apple pie from scratch, you must first invent the universe

Carl Sagan

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

How To Install ZeroTier on TrueNAS 12

February 3, 2022

Scam Alert: Fake DMCA Takedown for Link Insertion

January 24, 2022

How To Connect Everything From Everywhere with ZeroTier

January 14, 2022

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Symbolic Links

    Featured Posts

    Regarding My Symbolic Links and Good Reads

    April 16, 2015

    What’s (Still) Wrong With Dropbox For Business

    April 17, 2013

    How Fast Is It? A Storage Infographic

    October 29, 2010

    A High-Tech Water Heater? Yep! Introducing the A. O. Smith Vertex

    November 15, 2012

    Are You a Hypervisor Hugger or a Storage Stalwart?

    November 14, 2011

    Datacenter History: Through the Ages in Lego

    October 22, 2013

    Why You Should Never Again Utter The Word, “CIFS”

    February 16, 2012

    The Ideal pfSense Platform: Netgate RCC-VE 2440

    September 21, 2015

    Put that camera away and enjoy the view!

    April 11, 2012

    The Prime Directive of Storage: Do Not Lose Data

    December 12, 2014

    Footer

    Legalese

    Copyright © 2022 ยท Log in