Migrating Data With ZFS Send and Receive

I’m a huge fan of rsync as a migration tool, but FreeNAS is ZFS-centric so I decided to take a shot at using some of the native tools to move data. I’m not sold on it for daily use, but ZFS Send and Receive is awfully useful for “internal” maintenance tasks like moving datasets and rebuilding pools. Since this kind of migration isn’t well-documented online, I figured I would make my notes public here.

ZFS Send and Receive

ZFS is a volume manager, a file system, and a set of data management tools all bundled together.¹ One of the interesting features of ZFS is the ability to backup and restore data natively using the “send” and “receive” (or “recv”) commands.

Like rsync, tar, and other UNIX backup utilities, ZFS Send and Receive is a two-part process:

1. “zfs send” reads snapshot data and outputs a datastream to stdout
2. “zfs recv” reads a stream and restores data to a snapshot

You’ll note that I said “snapshot” there: In the interest of consistency, ZFS Send and Receive mainly operates on ZFS snapshots. This is a huge advantage over other utilities since it gives a point-in-time copy rather than reading some data at one time and other at a different time.

It’s also important to see that ZFS Send and Receive respectively output or input a stream of data. You can redirect this to a file or send it over an ssh connection, but you can also use it locally with a simple pipe. This means that ZFS Send and Receive can be used to move data internally to a system as well as for backup or remote replication.

Copying a ZFS Dataset

Let’s say you have a dataset called gang/scooby and wanted to create a perfect copy, gang/scrappy. ZFS Send and Receive can do this quickly and easily, guaranteeing a perfect and consistent copy! Here’s how:

1. Try to quiet gang/scooby (turn off Samba, stop applications, etc)
2. Make a snapshot: zfs snap gang/[email protected]
3. Send that snapshot to overwrite gang/scrappy: zfs send -R gang/[email protected] | zfs recv -F gang/scrappy²
4. Check out the results: zfs list -t snapshot -r gang
5. Promote gang/scrappy’s new snapshot to become the dataset’s data: zfs rollback gang/[email protected]

Note that this will totally blow away any existing data in the target dataset. If this is not what you want to do, you have been warned! But no one likes Scrappy Doo anyway…

Wait For It…

You can also redirect ZFS Send to a file and tell ZFS Receive to read from a file. This is handy when you need to rebuild a pool as well as for backup and replication.

In this example, we will send gang/scooby to a file and then restore that file later.

1. Try to quiet gang/scooby
2. Make a snapshot: zfs snap gang/[email protected]
3. Send that snapshot to a file: zfs send -R gang/[email protected] > gzip /tmp/ghost.gz
4. Do what you need to gang/scooby
5. Restore the data to gang/scooby: gzcat /tmp/ghost.gz | zfs recv -F gang/scooby
6. Promote gang/scooby’s new snapshot to become the dataset’s data: zfs rollback gang/[email protected]

Incremental Improvements

You can also send incremental updates of ZFS datasets. This is handy for more-active data that can’t be offline for the hours it might take to move data from place to place.

In this example, we’ll send an initial snapshot using ZFS Send and Receive, followed by an incremental snapshot of the data that changed.

1. Make a snapshot: zfs snap gang/[email protected]
2. Send that snapshot to overwrite gang/scrappy: zfs send -R gang/[email protected] | zfs recv -F gang/scrappy
3. Check out the results: zfs list -t snapshot -r gang
4. Try to quiet gang/scooby
5. Make another snapshot: zfs snap gang/[email protected]
6. Send the incremental snapshot to gang/scrappy: zfs send -i -R initial gang/[email protected] | zfs recv gang/scrappy
7. Note that both gang/scooby and gang/scrappy now have both @initial and @incremental snapshots: zfs list -t snapshot -r gang
8. Promote the latest gang/scrappy snapshot to become the dataset’s data: zfs rollback gang/[email protected]

You could script this to make sure it happens as quickly as possible. Of course it’s best not to try to do something like this on an active dataset, but it could still be useful on an active server: You only need to truly quiet the source application before the incremental snapshot is taken, and this will hopefully be much quicker than the initial data transfer.

Stephen’s Stance

I like ZFS Send and Receive, but I’m not totally sold on it. I’ve used rsync for decades, so I’m not giving it up anytime soon. Even so, I can see the value of ZFS Send and Receive for local migration and data management tasks as well as the backup and replication tasks that are typically talked about.

Update: Fixed a few typos. You need to add -R to zfs send! Thanks!

Here’s my entire FreeNAS series so far:

1. Hello FreeNAS! Goodbye Drobo and Iomega…
2. My FreeNAS Build: Supermicro X10SL7, Intel Haswell Xeon, ECC RAM
3. 14 Drives For 14 Ports: A Case For FreeNAS
4. It’s Fine To Mount Hard Drives On Their Side Or Even Upside-Down
5. Migrating Data With ZFS Send and Receive
6. FreeNAS First Impressions

Image credit: “Shipping” by Constance Abram, CC-by-NC-ND