• Skip to main content
  • Skip to primary sidebar
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Apple / How to Use Mac OS X Sparse Bundle Disk Images

How to Use Mac OS X Sparse Bundle Disk Images

July 22, 2015 By Stephen 10 Comments

I’m a big fan of “sparse bundle” disk images in Mac OS X. They allow me to create encrypted repositories for valuable data that can efficiently be rsync-ed between disks and don’t waste a lot of space. So I thought I’d write up a bit on what they are and how they can be used.

What’s a Disk Image?

I suppose I should start with a bit of background info on why I love sparse bundles so much. Here’s the low-down!

We’re all used to dealing with hard disk drives and thumb drives: They offer raw “block” storage that is formatted with a file system and used by the operating system, applications, and us users. Buy a 4 TB hard disk drive and you can format it and store (about) 4 TB of stuff on it.

Disk images are a little less familiar to average folks, but they work pretty much the same way. A disk image is a file on a disk that acts like a separate disk. It could be a virtual hard drive for a virtual machine, a copy of a DVD or Blu-Ray disc, or an archive for an application that wants to use an entire disk.

My favorite use of disk images is as a secure, encrypted drive for important data. Create a disk image with encryption and you can move it around from drive to drive or machine to machine without having to worry that someone else will get their hands on the content. Sure, you could encrypt your whole drive but this isn’t always desirable for removable media (portable hard disk drives and thumb drives) since you often want to have some “wide open” space, too.

When you create a disk image, you must specify the size of the virtual disk drive, and this space is typically consumed immediately regardless of how much data you actually write to it. So a 1 TB disk image will take up 1 TB of actual capacity on whatever drive you write it to. And the entire image is treated as a single huge file, so it’s not efficient to update a copy of it after you’ve changed something.

What’s a Sparse Bundle?

Happily, Mac OS X supports a “sparse bundle” disk image that solves these issues. Sparse bundles are thin provisioned, meaning they grow as you add data. And they consist of many (many!) “bands” of data, each stored in a separate file. So it’s very efficient to keep sparse bundles images in sync between media using a command like rsync.

Create a 1 TB sparse bundle disk image and it will take up only a few megabytes of physical space until you add some data to it. Then it will grow as you write to it, creating new few-MB files one after another to handle your data. The size of the bands is determined by the size of the total image, and might be 1, 2, 4, or 8 MB.

You can create a sparse bundle using Disk Utility in Mac OS X, as shown below.

Launch Disk Utility and select File->New->Disk Image
Launch Disk Utility and select File->New->Blank Disk Image…
Select the location to write the image, then choose "sparsebundle disk image" before setting the size. You can add encryption here, too, and the tool will ask for a passphrase when you click "Create".
Select the location to write the image, then choose “sparse bundle disk image” before setting the size. You can add encryption here, too, and the tool will ask for a passphrase when you click “Create”.

If you’re curious about the sparse bundle format, you can examine it from the command line. It consists of a “bundle” in Mac OS X parlance, which is a directory which is treated as a single file by Finder. 1 Inside this directory, you’ll find a few reference files as well as a subdirectory called “bands” with the files of actual data. As you add data to the sparse bundle, it will create more bands to store it.

My empty 1 TB sparse bundle takes up 672 MB on disk because of the 86 "band" files used to store the filesystem structure
My empty 1 TB sparse bundle takes up 672 MB on disk because of the 86 “band” files used to store the filesystem data structures

You can also examine a sparse bundle by right-clicking in Finder and selecting “Show Package Contents”. But in everyday use you will see the bundle as single large file. Double-click on it in Finder and it will mount as a new drive that you can use just like any other.

Efficiently Moving Sparse Bundle Disk Images

Let’s say you wanted to create a sparse bundle on a thumb drive to hold some important data. I recommend formatting smaller flash drives using the exFAT filesystem so they can be read on both Windows and Mac OS X machines. exFAT is better than FAT32 and Mac OS X is happy to write a sparse bundle there, but it’ll work on FAT32 or regular HFS+ too.

The quickest way to move data in Mac OS X is usually dragging-and-dropping in Finder. 2 And since Finder treats a sparse bundle as a single file, you can easily drag and drop your newly-created bundle to another drive.

But things get more complicated once you’ve started using the drive. As you add files, you have to be careful to keep both the original and copy in perfect sync or you will corrupt the bundle and lose the data. That’s bad.

I like to use rsync to keep bundles in sync across drives or machines. It’s purpose-built to do this, included in Mac OS X by default, and works wonderfully with sparse bundle bands!

For example, let’s say “Theon.sparsebundle” was an encrypted sparse bundle on a thumb drive called “Winterfell” and you wanted to keep it in sync with another drive called “Pike”. Here’s the rsync command you would use:

rsync -hav --progress --delete /Volumes/Winterfell/Theon.sparsebundle /Volumes/Pike

Enter that on the command line in Terminal and rsync will examine each file and synchronize everything that changed. The “–delete” part is important since it will remove anything on Pike that’s not also on Winterfell: A disk image must have only one master; as soon as it is split it will become corrupted! 3

Let me reiterate that last point very clearly: Decide which image you will treat as the master and always only write to that one! Do not try to modify the same image in multiple locations or you will lose track of which is which and you will lose data! In this example, you could use the Theon image on Winterfell for regular reading and writing and keep the copy on Pike as a backup in case you lose Winterfell, but don’t try using Theon on Pike, too, or he’ll get very confused! 4

I like to create a simple shell script to store my rsync operation so it’s executed the same way every time. Just write the rsync command to a file ending in “.sh” and “chmod +x” the file to make it executable.

Taking Out the Trash: Compress Your Sparse Bundle

One aspect of sparse bundles that is not handled well is trash collection. Just like a normal Mac disk drive, deleted files will be stored in the “trash can” rather than deleted for good. But when you empty the trash, the sparse bundle won’t get any smaller!

Although sparse bundles are thin provisioned, they don’t have any built-in un-provisioning mechanism. Once a band is created, it continues to exist. It can be re-used by new data, but your “sparse” bundle will eventually grow to the total size you created at the start, and this can be problematic.

Thankfully, there is a command line function to “compact” a sparse bundle! The Mac OS X command “hdiutil” can do lots of great stuff with disk images, including mounting, un-mounting, and compacting them. It can even create them if you’re so inclined.

First, delete what you don’t want anymore and empty the trash. This tells the file system on the sparse bundle what is and is not being used.

Next, un-mount the sparse bundle but don’t pull out the drive:

hdiutil eject /Volumes/Theon

Now you’re ready to reclaim space and remove any un-used bands:

hdiutil compact /Volumes/Winterfell/Theon.sparsebundle

And we’re good! The hdiutil command will remove any un-needed bands and return the sparse bundle to the minimum size possible. Then you can re-mount it and use it as normal. Just compact it again once you’ve deleted a bunch of data and feel it’s gotten too big.

Stephen’s Stance

If you’re careful about keeping them in sync, sparse bundle disk images can be a wonderfully useful tool in Mac OS X. You can have a secure repository for sensitive data on thumb drives or even in the cloud! But make sure to treat only one as the “master” and use rsync carefully to avoid losing data.

  1. Bundles are also used by Photos, Final Cut Pro, and other Mac applications. ↩
  2. I’ve timed command line cp, tar, and rsync and Finder is typically quicker. ↩
  3. See what I did there, George R. R. Martin? ↩
  4. Yeah, I did it again… ↩

You might also want to read these other posts...

  • Electric Car Over the Internet: My Experience Buying…
  • Powering Rabbits: The Mean Well LRS-350-12 Power Supply
  • What You See and What You Get When You Follow Me
  • Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!
  • GPS Time Rollover Failures Keep Happening (But…

Filed Under: Apple Tagged With: exFAT, FAT32, Game of Thrones, image, Mac OS X, rsync, SparseBundle

Primary Sidebar

Don’t start an argument with somebody who has a microphone when you don’t. They’ll make you look like chopped liver.

Harlan Ellison

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Liberate Wi-Fi Smart Bulbs and Switches with Tasmota!

May 29, 2020

What You See and What You Get When You Follow Me

May 28, 2019

GPS Time Rollover Failures Keep Happening (But They’re Almost Done)

April 6, 2019

Symbolic Links

    Featured Posts

    Regarding My Symbolic Links and Good Reads

    April 16, 2015

    Ranting and Raving About the 2018 iPad Pro

    November 11, 2018

    Not All 802.11n Networks Are Alike

    July 2, 2011

    Review: Blue Snowball USB Microphone

    March 31, 2010

    Making a Case For (and Against) Software-Defined Storage

    January 9, 2014

    Aerobie AeroPress Review: The Hacker Coffee Maker

    February 7, 2011

    A Complete List of VMware VAAI Primitives

    November 10, 2011

    FCoE vs. iSCSI – Making the Choice

    May 20, 2011

    The Fat Middle: Today’s Enterprise Storage Array

    August 31, 2014

    A High-Tech Water Heater? Yep! Introducing the A. O. Smith Vertex

    November 15, 2012

    Copyright © 2021 · Log in