I’m a huge fan of rsync as a migration tool, but FreeNAS is ZFS-centric so I decided to take a shot at using some of the native tools to move data. I’m not sold on it for daily use, but ZFS Send and Receive is awfully useful for “internal” maintenance tasks like moving datasets and rebuilding pools. Since this kind of migration isn’t well-documented online, I figured I would make my notes public here.
ZFS Send and Receive
ZFS is a volume manager, a file system, and a set of data management tools all bundled together.1 One of the interesting features of ZFS is the ability to backup and restore data natively using the “send” and “receive” (or “recv”) commands.
Like rsync, tar, and other UNIX backup utilities, ZFS Send and Receive is a two-part process:
- “zfs send” reads snapshot data and outputs a datastream to stdout
- “zfs recv” reads a stream and restores data to a snapshot
You’ll note that I said “snapshot” there: In the interest of consistency, ZFS Send and Receive mainly operates on ZFS snapshots. This is a huge advantage over other utilities since it gives a point-in-time copy rather than reading some data at one time and other at a different time.
It’s also important to see that ZFS Send and Receive respectively output or input a stream of data. You can redirect this to a file or send it over an ssh connection, but you can also use it locally with a simple pipe. This means that ZFS Send and Receive can be used to move data internally to a system as well as for backup or remote replication.
Copying a ZFS Dataset
Let’s say you have a dataset called gang/scooby and wanted to create a perfect copy, gang/scrappy. ZFS Send and Receive can do this quickly and easily, guaranteeing a perfect and consistent copy! Here’s how:
- Try to quiet gang/scooby (turn off Samba, stop applications, etc)
- Make a snapshot: zfs snap gang/scooby@migrate
- Send that snapshot to overwrite gang/scrappy: zfs send -R gang/scooby@migrate | zfs recv -F gang/scrappy2
- Check out the results: zfs list -t snapshot -r gang
- Promote gang/scrappy’s new snapshot to become the dataset’s data: zfs rollback gang/scrappy@migrate
Note that this will totally blow away any existing data in the target dataset. If this is not what you want to do, you have been warned! But no one likes Scrappy Doo anyway…
Wait For It…
You can also redirect ZFS Send to a file and tell ZFS Receive to read from a file. This is handy when you need to rebuild a pool as well as for backup and replication.
In this example, we will send gang/scooby to a file and then restore that file later.
- Try to quiet gang/scooby
- Make a snapshot: zfs snap gang/scooby@ghost
- Send that snapshot to a file: zfs send -R gang/scooby@ghost > gzip /tmp/ghost.gz
- Do what you need to gang/scooby
- Restore the data to gang/scooby: gzcat /tmp/ghost.gz | zfs recv -F gang/scooby
- Promote gang/scooby’s new snapshot to become the dataset’s data: zfs rollback gang/scooby@ghost
Incremental Improvements
You can also send incremental updates of ZFS datasets. This is handy for more-active data that can’t be offline for the hours it might take to move data from place to place.
In this example, we’ll send an initial snapshot using ZFS Send and Receive, followed by an incremental snapshot of the data that changed.
- Make a snapshot: zfs snap gang/scooby@initial
- Send that snapshot to overwrite gang/scrappy: zfs send -R gang/scooby@initial | zfs recv -F gang/scrappy
- Check out the results: zfs list -t snapshot -r gang
- Try to quiet gang/scooby
- Make another snapshot: zfs snap gang/scooby@incremental
- Send the incremental snapshot to gang/scrappy: zfs send -i -R initial gang/scooby@incremental | zfs recv gang/scrappy
- Note that both gang/scooby and gang/scrappy now have both @initial and @incremental snapshots: zfs list -t snapshot -r gang
- Promote the latest gang/scrappy snapshot to become the dataset’s data: zfs rollback gang/scrappy@incremental
You could script this to make sure it happens as quickly as possible. Of course it’s best not to try to do something like this on an active dataset, but it could still be useful on an active server: You only need to truly quiet the source application before the incremental snapshot is taken, and this will hopefully be much quicker than the initial data transfer.
Stephen’s Stance
I like ZFS Send and Receive, but I’m not totally sold on it. I’ve used rsync for decades, so I’m not giving it up anytime soon. Even so, I can see the value of ZFS Send and Receive for local migration and data management tasks as well as the backup and replication tasks that are typically talked about.
Update: Fixed a few typos. You need to add -R to zfs send! Thanks!
Here’s my entire FreeNAS series so far:
Image credit: “Shipping” by Constance Abram, CC-by-NC-ND
I think there is a typo here. Step 3 in the “Wait for it” section… you have: zfs send gang/scooby@migrate > gzip /tmp/ghost.gz…. I think you’re wanting to use the snapshot you just took (ghost), not the one from the previous example (migrate). So you should run: zfs send gang/scooby@ghost > gzip /tmp/ghost.gz
Also you shouldn’t need to “quiet” anything. Due to the nature of COW filesystems, and zfs in particular, you can take a snapshot in a very active fs at any moment. The resulting snapshot will have all the data present at that precise moment.
Found a typo. Opening sentence:
“I’m a huge fan or rsync as a migration tool…”
“or” should be “of”.
But if it is not fs sync’d then you may not have a snapshot with the data you think you have. Depends on your goial..
I think a snapshot syncs all pending operations related to a dataset. Transactions blocks in zfs are related to datasets, so I think it is safe to assume that all data not committed will be committed during the snapshot command.
If you have like 1-10 TB of data to move from one ZFS to another ZFS this can take some time and if executed from SSH the cursor just sits there waiting for the command to finish – I mean this could take hours.
To see progress in your SSH window, FYI=I use Putty SSH, add pv switch to authors command.
Here is the full syntax with added pv switch
Send that snapshot to overwrite gang/scrappy WITH REAL TIME PROGRESS (just added pv) below is the syntax to use (I am referencing Step 3 at top of this article):
zfs send gang/scooby@migrate | pv | zfs recv -F gang/scrappy
This will show MiB/s data movement and total data transferred in GiB in real time – so you can see how fast it is working and estimate how long it will take and see it is actually working.
I know the author mentioned being a fan of rsync but rsync has a real problem with speed e.g. in rsync you can copy the files with all sorts of switches but the speed is a major issue – on my system rsync will only copy the files @ 100MiB/s max while the zfs send/revc will perform from 400MiB/s to 700MiB/s. Also, I have used many switches in rsync in attempt to speed it up e.g. –inplace –no-compress, etc. and still the max is around 100MiB/s (I was testing rsync copy internally from one ZFS dataset to another ZFS dataset on high performance hardware- I just mention this as FYI
@disqus_6etNXOTLI6:disqus don’t run rsync over SSH then rsync can run as a daemon on its on port. without ssh it would get line speed like ZFS send
I feel like you are missing the -R parameter in zfs send. Most people will have nested containers and your command does nothing for me unless I add that. Good article anyway.
Small typo I picked up after trying to use this 🙂
6. Send the incremental snapshot to gang/scrappy:
zfs send -i -R initial gang/scooby@incremental | zfs recv gang/scrappy
needs to be:
zfs send -R -i initial gang/scooby@incremental | zfs recv gang/scrappy
this is because “initial” is a parameter of -i
Hi there,
I’ve been searching for hours now and all examples I find are doing a zfs send pool/dataset@snapshot – is there no way to specify a dataset with all its recursive snapshots as a source instead of a specific snapshot?