You might also want to read How Google and Dropbox Revolutionized My Laptop Migration

The Challenge of Synchronization

Keeping multiple machines “in sync” has always been a challenge. Ever since struggling with LapLink way back in the 1990′s, I’ve been looking for a better way.

Once I moved to Mac OS X, with its UNIX underpinnings, I began using rsync to move data back and forth. But rsync needs to be pointed in the right direction: It’s great for making sure the latest version of “this” is “over there” but not much help when trying to determine which is newer.

Early last year, I switched to the excellent open-source application, Unison. Like rsync, Unison optimizes the movement of data between multiple machines. But it detects which data is newer on a file-by-file basis and prompts before acting.

But Unison only runs on demand and requires interaction and attention. I also didn’t feel comfortable exposing it to the Internet, so both computers had to be on the same LAN.

Introducing Dropbox

I never expected Dropbox to be part of my dream synchronization solution, but that’s what it’s become. Dropbox mirrors local data to the cloud, allowing files to be made available through a web browser or mobile app. It can share a folder between multiple computers, and even multiple users.

By default, Dropbox syncs a cache folder with the cloud, rather than data in other locations

But data in Dropbox “lives” in Dropbox. Drag a file into your Dropbox folder and it now “lives” in the Dropbox cache and cloud. If the cache exists on the same drive as the original location, the file will be moved; it it is on another disk, a copy will be made. Either way, the original location is no longer relevant: Only the copy in the Dropbox folder will be synchronized.

Symlink to Sync

But I didn’t want to live out of the cache folder. I wanted my data to remain in my Documents folder. It’s not obvious how to do this with Dropbox, but a few FAQs and tools exist to make it happen.

Put a symbolic link in the Dropbox cache folder and it will sync files from any location!

The Dropbox application will follow symbolic links in the cache to discover data to synchronize, and this is the secret. Simply create a symbolic link (“symlink” for short) to another folder and you can leave your data wherever you want to!

There are two essential tricks to make this happen:

1. The folder symlinks in the Dropbox cache folder must have the same name
2. Dropbox must be turned off on all but one computer when initially setting all this up

A great little app called MacDropAny even exists to facilitate this process, and it’s not too difficult if you follow these rules.

Step By Step Instructions

Make sure you follow these instructions to the letter. Failure to turn off Dropbox, for example, can result in data loss! Good thing Dropbox keeps old copies available for recovery!

These instructions assume you have two Apple Macintosh computers, named “Computer 1″ and “Computer 2″, and that the Dropbox client and MacDropAny application are installed on both. A similar method will presumably work with Windows computers, but that’s left as an exercise for the reader!

Note: You must start with exactly-identical folders on both computers. The easiest way to ensure this is illustrated below, but there are other methods (rsync, for example).

1. In Finder on Computer 1, drag a folder from any location and drop it on the MacDropAny application in the Applications folder. For example, I dropped the folder, “~/Documents/Example”.
2. MacDropAny will ask where to create the link. Select a location in your Dropbox cache. For example, I created a folder called “~/Dropbox/Documents” and selected that.
3. MacDropAny will ask what to name this symlink. It is critical that this name be the same on both sides, and I suggest using the default name, that of the source folder. In my example, I called my symlink “Example” so the full path is “~/Dropbox/Documents/Example”.
4. Dropbox will discover, index, and synchronize this folder on both computers. You must wait until this process is finished before proceeding to the next step!
5. Did you wait until it’s finished? Really? Did you check both computers? Click the Dropbox icon on both, just to be sure it’s not still synchronizing.
6. Quit Dropbox on Computer 2. This is very important. You can leave it running on Computer 1, but it must not be running when you perform the next step!
7. On Computer 2, open the Dropbox cache folder. Note that the symlink will show up as a real folder full of files here. Drag-and-drop this folder to the desired location. In my example, I dragged the folder “Example” from “~/Dropbox/Documents” to “~/Documents”.
8. Still on Computer 2, drag this new folder and drop it on MacDropAny.
9. Select the same folder in the Dropbox cache and give it the same name. This is absolutely critical. In my example, the symlink went back in “~/Dropbox/Documents” and was called “~/Dropbox/Documents/Example”.
10. Check to make sure both computers have identically-named symlinks in the exact same location in both Dropbox cache folders.
11. If you’re really sure, start the Dropbox application on Computer 2 and let it discover, re-index, and sync the folder. The synchronization should be very quick since no data has to be moved.

That’s it! Now when you edit a file in “~/Documents/Example” on one computer, it will automatically and nearly-instantly show up on the other computer, as long as both are connected to the Internet. And if you do it offline, these changes will queue until you reconnect. You’ll also have access to these files online and through the mobile Dropbox app. Nice!

Caution!

This really isn’t difficult, but make sure you don’t take any shortcuts. Dropbox will automatically delete the whole folder if you forget to turn it off while creating the symlink on the second computer, and it will delete any missing files if you try to use an incomplete copy. Monitor the “Events” tab in your Dropbox web page to make sure you didn’t lose any data, and restore any accidentally-deleted files from there as well.

This makes it well-suited for keeping multiple “Documents” folders in sync, but isn’t always desirable. Some applications (notably iPhoto) don’t want to sync individual files, they want an entire bundle of files to be consistent. So this isn’t a perfect Mac-syncing solution. It does work for individual files usually found in the Documents folder, but be cautious. Don’t open the same file on two machines simultaneously, for one thing.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Keep Multiple Macs in Sync with Dropbox
This post was categorized as Apple, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

On The Extinction Of Ramdisks

System RAM is not typically used for storage anymore

Ramdisks were a cheat to make slow system I/O tolerable, but it was never an easy decision. These days, computers are fast enough that ramdisks are rare. Every part of the computer has gotten quicker and cheaper over the last two decades, including hard disk drives, RAM, and CPUs. A typical laptop like my MacBook Pro might have 4 GB of system RAM, 320 GB of hard disk capacity, and dual 2 GHz CPUs.Those specs would have seemed inconceivable just a few years ago: I still have a Dell laptop with a 20 MHz CPU and two expensive upgrades: 1 MB of RAM and a 20 MB hard drive!

Far more impressive, though, are the numerous performance optimizations that have appeared. I/O channels are quicker than ever, and caches have appeared at most performance bottlenecks. It’s hard to believe, but computers used to rely much more on raw storage. Many hobbyists recall waiting for data to load from cassette tape drives, or that one key performance advance in Intel’s 80486 chip was its on-chip cache. Today’s CPUs have three levels of cache, and operating systems have gotten much better at caching data internally as well.

Hard disk drives have sprouted their own caches, too, making I/O seem quicker than it is. My 1997 Toshiba laptop used an un-cached 4200 rpm hard disk, but even laptop drives today sport 16 MB or more and spin at 5400 rpm or faster. Although RAM was scarce back in the day, the performance gained from a ramdisk was worth the effort. Faster I/O and strategic caching has largely eliminated this need. Increasing operating system intelligence also plays a major part – witness the failure of hybrid hard drives.

The Modern Use Case

A long-standing argument in the field of computer system performance revolves around whether to manually reserve resources and place data or to let the system dynamically manage resources on its own. Computers do a decent job if allowed to, often adapting quicker and more efficiently to changing demands. But some jobs are harder to automate, and buffers and caches don’t always catch the right data.

Purely temporary data can be written to a ramdisk as a high performance scratch space. There is no need to store this on disk at all, and merely placing it there is likely to “pollute” the buffers, pushing out real useful data.

A ramdisk can also increase security somewhat by never allowing certain data to be written to disk. Ramdisks are sure to be flushed on the next reboot, but data on disk can linger and be discovered later.

One could even argue that today’s high-performance solid state disk is simply an evolution of the old ramdisk concept. NAND flash and RAM both offer the high random performance of the ramdisks of yore, and many analysts expect them to displace high-performance hard disk drives in the coming years. So perhaps ramdisks are not extinct after all!

Ultra-performance storage systems like this Violin array use RAM for storage

Using Ramdisks In Mac OS X

Mac OS X has the ability natively to create and use ramdisks. Here’s the simple procedure:

1. Create the ramdisk with hdiutil – This terminal command will create a 1 GB ram disk and report the name used. The number following the “ram://” statement is a number of 512 KB disk blocks – multiply the desired ram size in MB by 2048 (that’s 1024 times 2) to derive other sizes.
   hdiutil attach -nomount ram://2097152
2. Format the ramdisk with diskutil – The next command will format the newly-created ramdisk using the diskutil tool. We will use /dev/disk3, but you should specify whatever disk device hdiutil reported from step 1. You can also specify a different filesystem or disk name.
   diskutil eraseVolume "HFS+" "ramdisk" /dev/disk3
3. Your ramdisk will now appear – OS X will now mount the newly-created ramdisk and it will appear on your desktop. Note that the contents will be destroyed if it is unmounted or if the operating system is rebooted!
4. Delete your ramdisk – It’s worth saying once again: The ramdisk will disappear and all data will be deleted when you eject it or power off your computer. Be sure to back up any important data from it, then eject it with the following command:
   hdiutil eject /Volumes/ramdisk

Script It!

My backup script called for the creation of a tar file, compression, encryption, and rsync copying. I decided that I could perform the first three functions on the ramdisk to increase performance and reduce the chance that an unencrypted version would remain on the disk if the script failed.

A simple command line combination of the above creates the necessary ramdisk:

diskutil eraseVolume "HFS+" "ramdisk" `hdiutil attach -nomount ram://2097152`

This failed once I put it in a shell script, however. The issue was the finicky nature of diskutil – hdiutil’s output included a number of whitespace characters, and diskutil failed in the script whenever it encountered this. So I used the cut command to grab just the first field of the output, like so:

diskutil eraseVolume "HFS+" "ramdisk" `hdiutil attach -nomount ram://2097152 | cut -d " " -f 1`

This solved my problem, and I was able to tar and compress to /Volumes/ramdisk in my script. I added the hdiutil eject command to the script right after the encryption step to wipe out the unencrypted data. Although this isn’t foolproof security by any means, it’s better than nothing.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Ramdisks: Back From the Brink of Extinction
This post was categorized as Apple, Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Iomega ix4-200d is a sleek 4-drive SOHO RAID system that does just about everything, from NAS to Time Machine to iSCSI for a list price right around $700

Iomega is well into its second coming as EMC’s entry-level storage division. First, they applied EMC’s compact and full-featured LifeLine home storage software to existing gear, giving birth to the Home Media Network Hard Drive, StorCenter ix2, and StorCenter Pro ix4-100. Then they wooed the small-business community with the rack-mount StorCenter ix4-200r, adding iSCSI target support and VMware compatibility.

Today, they are back with the new ix4-200d, probably Iomega’s best product yet. It includes every feature of the rack-mount ix4-200r, including NAS and iSCSI target mode plus great new stuff like one-touch synchronization. All of this is packaged in a Drobo-like desktop system with a starting list price of just $700, or less than half the cost of a comparable ix4-200r!

Desktop Storage

External desktop storage products, exemplified by Western Digital’s wildly successful My Book series, have been a huge retail hit. Priced just over $100, these drives pack a terabyte or more and offer plug and play simplicity. I recently visited a small business with a WD or Seagate USB enclosure on every single desk. I’ve purchased five USB- or FireWire-connected hard drives myself over the last two years!

But these single-drive desktop wonders are a disaster waiting to happen:

• They fail frequently (like my Maxtor 3200), instantly wiping out the data they contained
• They are targets for thieves, so data loss prevention (DLP) experts warn against their use
• They aren’t shareable natively, so most people resort to sneakernet swapping rather than fight with Windows to present them as a network share
• When they’re full, they’re full, forcing the purchase of a whole new drive

Many vendors sell grown-up versions that address some or all of these concerns with multiple drives, network connections, and encryption, but these have been slow to catch on. Since they contain redundant drives and extra hardware and software, they are much more expensive than their little cousins. Iomega has done battle in this arena with their original StorCenter ix2 and the ix4-100, but these have yet to catch on. Even Drobo, with their fanatical user-friendly focus, has failed to convince many buyers.

Then there is the world of business storage. Way down at the bottom of the enterprise storage pyramid lies the realm of small 4- and 8-drive storage arrays. These SMB storage arrays offer a lot of capacity and reliability for the money but very little in the way of features. Iomega’s StorCenter Pro ix4-200r, in contrast, wowed the techies with a full iSCSI target stack that was certified for VMware ESX. But the price, over $1500, definitely limited sales to the home hobbyist.

“Just Right” Storage

Although the name is similar to the StorCenter Pro ix4-200r launched this Spring, Iomega went back to the drawing board for the ix4-200d. They built an entirely new device that could offer the impressive features of their StorCenter Pro line at a price closer to consumer storage offerings. The result boasts everything the Pro has and more at less than half the price:

• You want connectivity? The ix4 sports dual gigabit Ethernet ports that can be teamed up for performance or split off for redundancy.
• You want NAS? The ix4 supports NFS, SMB, and even Apple’s AFP, plus it’s Active Directory compatible.
• How about iSCSI? It’s a full-featured iSCSI target, certified for Microsoft Server 2003 and 2008.
• Want to host virtual machines? The ix4 is certified with VMware ESX 4 vSphere using both NFS and iSCSI, on the Xen HCL, and that Microsoft logo means it will work with Hyper-V as well.
• Need backup? The ix4 comes with EMC’s Retrospect and supports OS X Time Machine over AFP just like an Apple Time Capsule.
• Looking for weird features? How about support for up to 5 Axis network cameras, BlueTooth Picture Transfer Protocol, and UPnP/DLNA media service!
• Iomega also added a new feature, QuickTransfer, to synchronize files between devices.

This is one seriously feature-rich storage system. In fact, this glut of features is the ix4′s Achilles heel: How do you effectively communicate the value of a device that does so much? Most of the buying public has never heard of most of these features, so the price remains hard for some to justify.

Every ix4 configuration comes loaded with four hard disk drives

Another hurdle for the ix4 is Iomega’s decision to fill it with hard drives. One cannot buy an empty ix4-200d, and both the 2 TB and 4 TB configurations come loaded with four hard disk drives. This raises the price of entry and scares off the very techies who might be interested in the device. The majority of Drobos are purchased with no drives at all, and storage geeks like me love the idea that disks can be added as-needed in the future. Although Iomega is open to users swapping out the drives in their own StorCenter device, this is not its intended use case. Iomega’s decision to sell the ix4-200d as a loaded appliance seems counter to the price sensitivity and flexibility needs of buyers.

QuickTransfer: Data Synchronization for Everything

One nifty new feature included with the ix4-200d is QuickTransfer, a one-touch data synchronization capability. Leveraging rsync technology, but hiding this complexity with a more-friendly wizard-based GUI, QuickTransfer allows users to set up synchronization jobs between the ix4 and a variety of targets:

• USB drives can be plugged into one of the three USB 2.0 ports and synchronized with a subset of the content of the ix4. For example, a portable USB drive could be “recharged” with the latest set of data before one heads out of the office.
• Two Iomega ix4′s, or other NAS systems for that matter, can be synchronized over the Ethernet/IP LAN. This would provide a robust and bandwidth-friendly remote office backup or data replication solution.
• A PC or Mac can also be synchronized over a network share, providing a simple alternative to the bundled backup software.

QuickTransfer is exclusive to the ix4-200d for now, but Iomega assured me that it would be included in future StorCenter products and added to the StorCenter Pro ix4-200r in the near future. It is unknown if or when other existing StorCenter devices will get QuickTransfer, however.

Iomega’s Next Move

What will Iomega do next? They must be ready to announce their vSphere 4 and Microsoft Windows Server certification soon, since both company’s web sites already list the device in their compatibility lists. Iomega ought to try to take advantage of the interest among VMware users with a big VMworld splash. They will be there, but it is awfully hard to get noticed at such a large event. I am looking forward to the event to get a hands-on test.

The ix4 should begin showing up for sale at online stores very quickly. Amazon listed the rackmount product within days of its release, and we expect the same this time. But will Iomega offer this cheaper device in retail stores? It would be great to have it available at Staples and Best Buy, but shelf space for a storage system this expensive would be hard to get. Instead, expect it at specialty outlets like Fry’s and perhaps Micro Center.

How much does the StorCenter ix4-200d cost?

• The 2 TB model (SKU# 34546 with four 500 GB drives) lists at $699.99
• The 4 TB model (SKU# 34549 with four 1 TB drives) lists at $899.99
• The 8 TB model (SKU# 34563 with four 2 TB drives) lists at $1,899.99

It also seems likely that the 2-bay product is up for a refresh in the near future. The ix2 can’t be said to be very attractive, so hopefully the company will do something about this with the next-generation product. It is unclear whether the inexpensive 2-bay device will get iSCSI support, but I suspect it will. Beyond this, might Iomega move further upmarket with an 8-drive unit? We shall see!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Iomega’s ix4-200d: A Killer Desktop Storage Array
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Does data encryption throw storage efficiency out the window? Not always!

One of the great ironies of storage technology is the inverse relationship between efficiency and security: Adding performance or reducing storage requirements almost always results in reducing the confidentiality, integrity, or availability of a system.

Many of the advances in capacity utilization put into production over the last few years rely on deduplication of data. This key technology has moved from basic compression tools to take on challenges in the fields of replication and archiving, and is even moving into primary storage. At the same time, interconnectedness and the digital revolution has made security a greater challenge, with focus and attention turning to encryption and authentication to prevent identity theft or worse crimes. The only problem is, most encryption schemes are incompatible with compression or deduplication of data!

Incompatibility of Encryption and Compression

Consider a basic lossless compression algorithm: We take an input file consisting of binary data and replace all repeating patterns with a unique code. If a file contained the sequence, “101110″ eight hundred times in a row, we could replace the whole 4800-bit sequence with a much smaller sequence that says “repeat this eight hundred times”. In fact, this is exactly what I did (using English) in the previous sentence! This basic concept, called run-length encoding, illustrates how most modern compression technology functions.

Replace the sequence of identical bits with a larger block of data or an entire file and you have deduplication and single-instance storage! In fact, as the compression technology gains access to the underlying data, it can become more and more efficient. The software from Ocarina, for example, actually decompresses jpg and pdf files before recompressing them, resulting in astonishing capacity gains!

Now let’s look at compression’s secretive cousin, encryption. It’s only a small intellectual leap to use similar ideas to hide the contents of a file, rather than just squashing it. But encryption algorithms are constantly under attack, so some very smart minds have come up with some incredibly clever methods to hide data. One of the most important advances was public-key cryptography, where two different keys are used: A public key used for writing, and a private key to read data. This same technique can be used to authenticate identity, since only the designated reader would (in theory) have the key required.

Cryptography has become exceedingly complicated lately in response to repeated attacks. Most compression and encryption algorithms are deterministic, meaning that identical input always yields the same output. This is unacceptable for strong encryption, since a known plaintext attack can be used with the public key to reveal the contents. Much work has focused on eliminating residues of the original data from the encrypted version, as illustrated brilliantly on Wikipedia with the classic Linux “tux” image. The goal is to make the encrypted data indistinguishable from random “noise”.

What happens when we mix these powerful technologies? Deduplication and encryption defeat each other! Deduplication must have access to repeating, deterministic data, and encryption must not allow this to happen. The most common solution (apart from skipping the encryption) is to place the deduplication technology first, allowing it access to the raw data before sending it on to be encrypted. But this leaves the data unprotected longer, and limits the possible locations where encryption technology can be applied. For example, an archive platform would have to encrypt data internally, since many now include deduplication as an integral component.

Why do we prefer compression to encryption? Simply because that’s where the money is! If we can cut down on storage space or WAN bandwidth, we see cost avoidance or even real cost savings! But if we “waste” space by encrypting data, we only save money in the case of a security breach.

A Glimmer of Hope

I had long thought this was an intractable problem, but a glimmer of hope recently presented itself. My hosting provider allows users to back up their files to a special repository using the rsync protocol. This is pretty handy, as you can imagine, but I was concerned about the security of this service. What happens if someone gains access to all of my data by hacking their servers?

At first, I only stored non-sensitive data on the backup site, but this limited its appeal. So I went looking for something that would allow me to encrypt my data before uploading it, and I discovered two interesting concepts: rsyncrypto and gzip-rsyncable.

rsync is a solid protocol, reducing network demands by only sending the changed blocks of a file. But, as noted, compression and encryption tools change the whole file even if only a tiny bit has been altered. A few years back, the folks behind rsync (who also happen to be the minds behind the Samba CIFS server) developed a patch for gzip which causes it to compress files in chunks rather than in their entirety. This patch, called gzip-rsyncable, hasn’t been added to the main source even after a dozen years, but yields amazing results in accelerating rsync performance.

The same technique was then applied to RSA and AES cryptography to create rsyncrypto. This open source encryption tool makes a simple tweak to the standard CBC encryption schema (reusing the initialization vector) to allow encrypted files to be sent more efficiently over rsync. In fact, it relies on gzip-rsyncable to work its magic. Of course, the resulting file is somewhat less secure, but it is probably more than enough to keep a casual snooper at bay.

Both of these tools are similar to modern deduplication techniques in that they chop files up into smaller, variable-sized blocks before working their magic. And the result is awesome: I modified a single word in a large word document that I had previously encrypted and stored at the backup site and was able to transfer just a single block of the new file in an instant rather than a few minutes. My only real issue is the lack of integration of all of these tools: I had to write a bash script to encrypt  my files to a temporary directory before rsyncing them. I wish they could be integrated with the main gzip and rsync sources!

If you are interested in trying out these tools for yourself, and if you use a Mac, you are in luck: Macports offers both tools as simple downloads! Just install macports, type “sudo port install gzip +rsyncable” to install gzip with the –rsyncable flag, then type “sudo port install rsyncrypto” and you’re done! I’ll post more details here if there is interest.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Compression, Encryption, Deduplication, and Replication: Strange Bedfellows
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.