Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
This post was categorized as Enterprise storage, Everything, Gestalt IT, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My State of the Art Thin Provisioning series
  • Storage is Not Getting Cheaper
  • Thin Provisioning: Attacking Storage Utilization
  • De-Allocating is the Core Issue for Thin Provisioning
  • Thin Provisioning: Playing the Telephone Game
  • Monitoring Filesystem Metadata For Thin Provisioning
  • Zero Page Reclaim: Savior of Thin Provisioning?
  • What is WRITE_SAME? Green Eggs and Ham!
  • The Bridge: Veritas Thin (Provisioning) API
  • Is TRIM Useful For Thin Provisioning?
• The Raising the Floor podcast series
  • Raising the Floor 1: Cloud Storage Gateways
  • Talking Cloud Storage Gateways With Nasuni and Cirtas
  • Raising the Floor: How the Cloud Impacts Enterprise IT
• Tech Field Day links
  • Data Robotics Is First Three-Time Tech Field Day Presenter
  • The Nerd’s Going to Have a Field Day! | The Networking Nerd
• And more links!
  • I wrote this piece for SearchStorage: Is PCIe SSD right for you? Deploying PCI Express SSD devices
  • I was interviewed for another thin story: Thin provisioning driven by technology advances, best practices
  • My friend Erin Collopy from HP is starting a blog: A new year, a fresh slate « Social Swimmer
  • Bob Plankers always impresses with his technical writing: How To Create and Measure NTFS & ext3 Disk Fragmentation
  • I loved this idea from Freedom to Tinker: Monitoring all the electrical and hydraulic appliances in your house
  • Chris Mellor spills the beans on EMC’s big announcement next week: Storage Virtualization, Unified and the VNX
  • Chris also posted a great piece about InPhase: How I watched a holographic storage company implode
  • Then there’s this Winextra bit on Microsoft’s Biggest Mistake of 2010
  • A new EMCer blog, Storage Savvy, has a nice pNFS piece: Why pNFS can be a big deal even if NFS4.1 isn’t…
  • Martin Glassborow is always worth reading: Reading the Ruins
  • Floyd Strimling makes a go at prognostication: Fun Alert: Insane 2011 Predictions That May Come True
  • Another great Bill Hill post on cloud: Cloud Crossroads
  • And Ian Foster isn’t happy with Amazon: Amazon – Masters of marketing or perfecting the art of customer dissatisfaction?

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, January 7, 2011
This post was categorized as Computer History, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I began by introducing the core problem: Storage isn’t getting any cheaper due to storage utilization and provisioning problems. Thin provisioning isn’t all it’s cracked up to be, since the telephone game makes de-allocation a challenge. So now let’s talk about how to make thin provisioning actually work.

There are 100 different ways of solving the de-allocation problem, some of which have gained some prominence. They all boil down to two options:

1. Make the server super-smart and have it communicate better
2. Make the storage super-smart and have it make educated guesses

There’s only a few ways that the server-side option can be implemented, and we’ll get to that. But first, let’s take a look at a sort of hybrid approach that relies on known server usage patterns: Metadata monitoring.

It’s really hard for the storage to really understand what the server is doing. The best example that I know of is the Drobo sitting under my desk.

I love this little black box. When I got it, I configured it as eight terabytes and I put a 160-gig disk in it. That’s thin provisioning. And over time, I’m swapping out the disks and I’m doing all my stuff, and it still looks like eight terabytes. Add data, delete it, swap disks, and it always just works.

Not a lot of people know how the Drobo works, though. One of the things that people have complained about is that it only supports certain file systems and partition schemes. The reason for this is a “magical” thing it’s doing that relates very, very closely to the topic of this discussion. The Drobo is the first thin provisioning box that I know of that directly monitors the file system.

What the Drobo does is this: It knows where the supported filesystems (HFS+, NTFS, EXT3, and FAT) keep the record of what’s been deleted. So the Drobo it watches that spot and when you delete something, it reclaims that space. No enterprise storage system can do this, and yet this little box under my desk does it all day long.

This is basically the super, ultimate smarts of storage. But, of course, it’s very limited. It faces a real challenge in an enterprise setting because there is much more variety. We have all these layers of virtualization and weird file systems and things like that to worry about. We just can’t expect a product like this to accommodate everybody, so we just can’t expect this kind of smarts to be put everywhere.

Instead, we have a variety of semaphores sent from the server to the storage array that attempt to solve the telephone game. That’s what we’re talking about next.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Monitoring Filesystem Metadata For Thin Provisioning
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing
  • Cloud storage gateway eases transition to cloud storage services
  • Eleven Tech Trends To Watch In 2011
  • How To Write To Windows NTFS Drives In Mac OS X 10.6 “Snow Leopard”
  • Introducing “Raising the Floor”, Our Datacenter Podcast
• Enterprise tech
  • NetApp Snapshot: News On NetApp – Ep 4
  • December #SANchat – 11 Trends for 2011
  • Fujitsu Teams up with NetApp to Design a Storage Blade
  • The Scaling Limitations of Etherchannel -Or- Why 1+1 Does Not Equal 2
• Other geeky stuff:
  • The Curse of Beta Hardware
  • Full Body Scanners: What’s Next?
  • Useful applications for the PC-to-iPad convert
  • CradlePoint CTR-500 Review

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Back From the Pile: Interesting Links, December 3, 2010
This post was categorized as Enterprise storage, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Introducing NTFS

A filesystem is the basic scheme for organizing data on a hard disk drive. The common FAT filesystem, supported by just about every operating system, is just too basic for modern operating systems. Microsoft introduced the advanced NTFS filesystem with their Windows NT operating system in 1993. Borrowing from OS/2′s HPFS and VMS’ Files-11, NTFS is a journalling filesystem with many features, including POSIX links, alternate data streams, sparse files, compression, and encryption.

Although just about everything Microsoft produces is maligned by UNIX and Mac geeks, this hatred is unwarranted in the case of the many storage advances coming out of Redmond. Microsoft may be fairly criticized for producing expensive, proprietary systems, but they have dome some amazing things with disks, and NTFS is arguably one of the most advanced filesystems in wide usage. Much of this credit goes to Windows NT architect Dave Cutler, who also led development of DEC VMS, the RISC concepts that became Alpha, and Microsoft’s Azure cloud platform.

Although every version of Windows supports evolutions of the old FAT filesystem, modern iterations increasingly require NTFS. Windows XP allowed either FAT32 or NTFS for booting; Windows Vista could be forced to boot from FAT32; Windows 7 requires NTFS. Therefore, most PC hard disk drives, including external USB drives, now come formatted with NTFS by default.

NTFS on Mac OS X

This leads to issues for Mac users, especially when they dual-boot with Boot Camp or buy external hard disk drives. As mentioned, all versions of Mac OS X since 10.3 “Panther” have included read-only NTFS support, but those wanting full access had two options:

1. Paragon Software’s NTFS for Mac OS X is a full and supported read/write solution. Although available for purchase separately, many hard disk drives like the Seagate GoFlex now include a free copy of Paragon NTFS.
2. The open source NTFS-3G driver has also been ported to Mac OS X, allowing read/write support for free.

Mac OS X 10.6 “Snow Leopard” adds a third alternative: Native read/write support for NTFS. But it’s not as simple as that. Although early betas enabled read/write support by default, it was limited once again to read-only in the final releases.

It is possible and even simple) to enable read/write support with the native Snow Leopard drivers, but this is definitely an “at your own risk” proposition. Users have reported kernel panics when using these methods, and although I have not heard of data corruption, it is possible as well.

The Manual Method: mount_ntfs

Making disks available is called “mounting” in UNIX geek-speak, and mount commands in modern operating systems are like ogres: They have many layers. Plug in a drive, and Mac OS X makes it available for you using a layer cake of commands, but the bottom-most command is “mount”. This simple command-line utility takes a few arguments and attaches a raw disk device to a point in the unified filesystem tree.

When presented an NTFS drive, Mac OS X calls a utility called mount_ntfs that resides in the /sbin directory, and we can call it, too. Although OS X always tells mount_ntfs to mount NTFS drives read-only, we can tell it to enable writing as well by using the “-o rw” flag.

There’s just one problem: Where should we connect the NTFS file system? Mac OS X automatically creates a new directory (called a “mount point” in UNIX-speak) in /Volumes with the name of the filesystem. For example, the GoFlex drive I bought yesterday mounts as “/Volumes/FreeAgent GoFlex Drive” by default. But mount_ntfs won’t create this mount point, so we have to do it.

Here are the steps to mount an NTFS drive read/write from the command line:

1. Plug in the drive and let Mac OS X mount it read-only.
2. Look in Finder to see the name of the drive. For example, my drive mounts with the name “FreeAgent GoFlex Drive” and this is what shows up on the desktop and in Frinder under “Devices”.
3. Open the Terminal application to get command-line access. We’ll use “sudo” which executes commands as the administrator instead of a regular user.
4. Type the following command in Terminal to get necessary information about the drive. Use the drive name you see in Finder in place of “drive name”, and make sure to use quotation marks around it since Terminal treats un-quoted spaces as separators rather than part of the name.

   diskutil info "/Volumes/drive name"

5. Record the “Device Node” entry, which will look something like “/dev/disk4s1″. This is where the raw partition is mapped by the disk subsystem, and where the filesystem driver will read its data.


The diskutil command gives us the "Node Name", which we'll need momentarily

6. Eject the drive, using option-click in Finder or with the hdiutil command in Terminal:

   hdiutil eject "/Volumes/drive name"

7. Now you must create the mount point again. Type the following:

   sudo mkdir "/Volumes/drive name"

8. Finally, use mount_ntfs to mount the drive read/write using the following command. Use the node name you spotted above and the mount point you just created:

   sudo mount_ntfs -o rw /dev/disk?s? "/Volumes/drive name"

9. When you’re done, you can use the Finder, the hdiutil command, or just about anything else to eject the drive. Make sure your mount point is deleted as well!

This method is great for occasional NTFS use, which is frankly all Mac users should need. If you intend to use a drive frequently, I highly recommend reformatting it with the native HFS+ filesystem since it will be fully-supported by the operating system.

The Per-Drive Method

Let’s say you have an NTFS drive you frequently need to mount read/write and find the above method too much of a hassle. Maybe it’s your Boot Camp drive, or perhaps it’s a USB hard disk drive you share with a Windows-using colleague. It is possible to tell Mac OS X to always mount a given NTFS drive read/write using its native driver using the following commands. However, I recommend investing in the Paragon driver for this use case, since it’s fully-supported!

Like all UNIX systems, Mac OS X uses a configuration file to determine what it should do with certain drives. This file, called “fstab” and located in “/etc”, is not present by default but will override the “read-only” flag on a per-drive basis if found. This allows a drive to always be mounted read/write without any hassle.

1. Get the drive name as in step 2 above.
2. Open Terminal and type the following command to edit the /etc/fstab file:

   sudo nano /etc/fstab



Adding a line in /etc/fstab will make Mac OS X always mount a single drive in read/write mode

3. “Nano” is a command-line editing program. Add the following line to the file (and don’t forget the quotes around the drive name if it contains spaces!)

   LABEL="drive name" none ntfs rw

4. Now type control-x to save and exit, type y for yes, and hit enter.
5. After a reboot, Mac OS X should automatically mount any drive with that label as read/write

Note that this can also be done using the NTFS UUID of the drive, but this isn’t present on every drive to LABEL works better. Also, some have reported needing multiple reboots to make it work. I have no idea why.

The Permanent Universal Method

It is also possible to permanently modify how Mac OS X calls the mount_ntfs command to always mount every NTFS drive in read/write mode. Again, I will point out that this isn’t necessarily a great idea since Apple doesn’t support using the driver in this way. But here it is!

1. Open Terminal. We won’t need the drive name or Node Name since this applies to every NTFS drive.
2. Rename the mount_ntfs command to something else by typing the following:

   sudo mv /sbin/mount_ntfs /sbin/mount_ntfs_orig

3. Create a shell script called mount_ntfs using the Nano editor:

   sudo nano /sbin/mount_ntfs

4. This script will simply call mount_ntfs with the “-o rw” flag every time it’s used. Type the following in Nano:

   #!/bin/sh
   /sbin/mount_ntfs_orig -o rw "$@"

5. Type control-x to save and exit, type y for yes, and hit enter.
6. Now we’ll fix the permissions on this new file so it’s owned by root and group wheel:

   sudo chown root:wheel /sbin/mount_ntfs

7. Finally, we make the shell script executable:

   sudo chmod 755 /sbin/mount_ntfs

8. Now every time OS X encounters an NTFS drive, it will mount it read/write by calling this script rather than the “mount_ntfs” executable it thinks it’s using.

Note that future OS updates might “break” this method by replacing the /sbin/mount_ntfs command, or it could cause other issues. But it works for now and doesn’t even require a reboot.

Stephen’s Stance

I’m hesitant to recommend that casual Mac users employ any of these methods to access NTFS drives. Each requires superuser access and careful typing. You can easily mess up your system this way, and unsupported uses of the NTFS driver might result in data loss.

If you aren’t comfortable with this kind of mucking-about or intend to frequently write to NTFS drives, I highly suggest installing NTFS-3G or Paragon’s NTFS driver. In fact, the best method is simply to go buy a Seagate GoFlex drive and use the Paragon driver that comes with it!

But hackers and geeks like me want to know how to do this. This is why I decided to document it here.

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | How To Write To Windows NTFS Drives In Mac OS X 10.6 “Snow Leopard”
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Today's super-sized drives need something better than FAT. Introducing exFAT, the next-generation portable filesystem.

Microsoft already gave the world FAT and NTFS, and both have become common in the non-Windows world thanks to flash drives, SD cards, and portable disks. But the folks from Redmond are now introducing a new filesystem, exFAT. Do we really need a new filesystem?

The Rise and Fall of FAT

Although computing is a volatile world, standards tend to stick around way past their prime. Nowhere is this more apparent than the ubiquitous FAT filesystem. Although it has been modified and updated, the basic FAT structure pre-dates DOS: It was designed for Microsoft’s first real product, Standalone Basic, and bears the fingerprints of the company’s first employee, Marc McDonald, as well as some other programmer called Bill Gates.

Appealing in its simplicity, FAT was adopted by Tim Paterson at Seattle Computer Products for his QDOS operating system, which found its way back to Microsoft as IBM’s PC-DOS. FAT was improved with 16-bit cluster addresses for the 1984 IBM PC-AT’s hard disk drive, and enhanced again to match growing disk sizes as FAT32 in Windows 95. Along the way, VFAT added 32-bit file access and long file names.

Because it was simple in concept, FAT was reverse-engineered and adopted as a standard filesystem for data interchange between DOS/Windows PCs and other operating systems, from Apple to Atari to Amiga. Today, FAT16 is the dominant format for SD cards, and most thumb drives and SDHC cards use FAT32. But FAT is not an open standard: Microsoft successfully patented VFAT and FAT32 and specifically requires digital camera and flash drive companies to obtain a license for its use.

Although widely dismissed as archaic, many of the limits of FAT (especially FAT32) are due to factors outside the filesystem, particularly the MBR partition scheme and implementation-related issues. FAT32 is capable on disks up to 2 terabytes, which matches the maximum size of the next-generation SDXC flash card specification. And, despite Microsoft’s patents, FAT is widely implemented and stable on a variety of platforms and embedded devices.

Why exFAT?

Hard disk drives long since switched to NTFS or other operating system-specific formats, so FAT is mainly used on flash media today. Sadly, FAT isn’t all that well-suited for applications like these. Thumb drives don’t really push the envelope for FAT since they tend to hold reasonable numbers of smaller files, but flash media for digital cameras is a different story.

SD cards have taken over the digital camera world: Even holdouts like Sony now support SD and its successors, SDHC and SDXC. And just about every SD and SDHC card sold today uses FAT16 and FAT32, respectively. It works reasonably well for still images, but video poses real issues. FAT32 is limited to 4 GB per file, and HD camcorders and digital cameras can easily push past this size limit. Most models today split files or limit recording time to avoid this limitation, but that’s not optimal.

exFAT enables file sizes up to 16 EB and also adds nifty features like a free space bitmap allowing better use of flash memory and eliminating the read/erase/write issue of NAND. It also allows more-granular timestamps (10 ms, rather than 2 seconds on FAT) which better-match high-speed shooting capabilities of today’s digital cameras.

exFAT Limitations

But not everything is rosy with exFAT. Although it seems poised to take off thanks to strong support from the SDcard Association, exFAT remains encumbered by patents and licensing fees from Microsoft. Memory Stick XC requires the use of exFAT, but SDXC merely suggests it and there is no guarantee of third-party support. Digital camera makers may decide to stick with FAT32 rather than switch, though Microsoft makes it more attractive for embedded uses with a flat $300k fee per manufacturer.

There is also one odd technical limitation to exFAT. Because removable media is often physically ejected at inopportune moments, the FAT filesystem was designed with alternating file allocation tables. Should writing of the table be interrupted, FAT can always fall back on the previous iteration. exFAT curiously uses just one allocation table and free space bitmap, making it more likely to become corrupt in precisely the portable devices it was intended for. A follow-on enhancement, Transactional exFAT or TexFAT, adds redundant tables and bitmaps but may not be implemented.

Unfortunately, the old, inflexible MBR partition scheme remains coupled to exFAT. It would have been nice to ditch MBR in favor of the open GPT format, but the static nature of flash media made that less of a priority. Still, MBR is limited to 2 TB, and this will become a roadblock for flash media at some point.

You might also want to read Introducing SDXC and exFAT in Apple Mac OS X

Stephen’s Stance

Adoption of exFAT seems likely, especially for the SDXC and MSXC flash cards used in digital cameras. Because it is not backwards-compatible with devices, media, operating systems, and card readers, however, consumer confusion is likely to result. Even systems with an SDXC card reader might not be capable of reading the exFAT filesystem contained on the card, and vice versa.

Microsoft, being the proud parent of exFAT, supports the new filesystem in its modern Windows 7 and Server 2008 operating systems and has moved quickly to add it to Windows XP, Server 2003, and Vista. A few exFAT drivers exist for Linux and Android, and Apple is likely to add support for exFAT in Snow Leopard 10.6.5, since the SDXC-capable mid-July iMac and Mac Mini computers already have it in 10.6.4.

So it looks like we’ve got a new filesystem on our hands, at least for camera cards. But exFAT will not likely replace FAT32 on thumb drives any time soon. Their portability requires broad compatibility, and FAT32 has the widest support today. Although exFAT will likely be used in embedded applications, it there is no compelling reason for it to displace NTFS and other operating system filesystems like (Apple) HFS+ and (Linux) ext3/ext4 any time soon.

Image credit: Scale Face by Playingwithbrushes

© sfoskett for Stephen Foskett, Pack Rat, 2010. | An Introduction To exFAT
This post was categorized as Apple, Computer History, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It's hard to take a product called "Robocopy" seriously!

Anyone doing much storage work on Microsoft Windows machines is familiar with Robocopy. It’s the best tool to move data between NTFS filesystems, since Robocopy maintains permissions and file attributes. It also tolerates dropped connections, resuming where it left off, and can throttle operations over slow networks.

But Robocopy was never very quick, especially when dealing with large data sets. It was single-threaded, hurting performance on high-latency networks, and startup was painfully slow on deep directory structures.

Robocopy XXVII

Windows Vista, Windows Server 2008, and later versions include a new version of Robocopy with performance tweaks designed to overcome these limitations. Right-click on the executable in Windows\System32 and make sure you are using version XP027, 5.1.10.1027.

Update: Oops! Only the “6.1″ versions of Microsoft Windows (Windows 7 and Windows Server 2008 R2) include multi-threaded robocopy!

The big deal here is multi-threading:

• The application can now run in multi-threaded mode using the /MT option. This defaults to 8 threads, but users can specify up to 128 if desired. For example, the following command would use 16 threads:

robocopy c:\ d:\ /MT:16

• Initial directory enumeration is also multi-threaded, so deep directory structures are examined much more quickly.
• XP027 also added the /EFSRAW parameter, allowing one to copy files from EFS using RAW mode. However neither this nor /IPG (inter-packet gap) can be combined with the multi-threaded option mentioned above.

Dead or alive, you’re coming with me!

I tried out the new /MT option on a Core 2 Duo laptop and was surprised by the dramatic improvement in copy performance. Copying my entire “Program Files” directory took well over a minute without multi-threading, but simply specifying “/MT” at the end of the command reduced a second copy to a different directory to about 20 seconds. Using “/MT:32″ was blazing fast – easily less than 15 seconds. I repeated the first single-threaded test again and watched it dawdle along, taking over a minute again to finish.

Microsoft suggests that multi-threaded Robocopy helps with network throughput as well. I tried a series of copies between two Windows 7 machines over Wi-Fi, to simulate a slow network. Although single-threaded Robocopy was able to saturate the network with large files, it really slowed down (thanks to latency) once it hits a patch of smaller files. Multi-threaded Robocopy was more capable of maintaining high throughput once smaller files were encountered, with a 32-thread test keeping the link at maximum pretty much the entire time. Again, a noticeable improvement.

Your Move, Creep

If you’re copying lots of Windows data on a machine running Windows 7 Vista or newer, I can confidently say that the /MT switch will speed things up dramatically. Whether you should stick with the default 8 threads or up it to 16 or 32 depends on the capabilities of your CPU, but it’s worth a try. One more tip: Use the /LOG switch or pipe the output to NULL to speed up copying even more. Displaying all that text delays the whole process!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Robocopy: Better, Faster, Stronger
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

1. Use the excellent and free NTFS-3G driver
2. Buy Paragon Software’s Paragon NTFS

I’ve been using the NTFS-3G driver myself for quite some time. It works as advertised: Install it and you can read and write to NTFS-formatted drives with ease. These drives are integrated right into the Finder, appearing just like Mac-standard HFS+ disks. This is the method I used to assign custom drive icon for my Boot Camp volume and USB drives, for example.

Note: Snow Leopard’s NTFS driver is read/write capable. See my post, How To Write To Windows NTFS Drives In Mac OS X 10.6 “Snow Leopard”

This post is part of my series focused on Apple OS X tips and tricks.

• Access NTFS Volumes On Your Mac
• Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
• How To Move OS X Time Machine Backups To A New Disk
• Hallelujah! OS X Can Reduce PDF File Size!
• Custom Drive Icons in Mac OS X
• OS X Custom Drive Icons 2: Boot Camp and NTFS

NTFS is a userspace (FUSE) driver, so installing it requires the installation of mac-fuse as well. It’s been nicely packaged for install over at MacNTFS-3G, or you can install it with MacPorts, along with lots of other great UNIX software.

Paragon NTFS functions similarly to NTFS-3G. The latest versions also include an HFS+ driver for Windows, allowing your Boot Camp instance to read your Mac drive, which is pretty cool. I was initially interested in Paragon’s software, thinking it would be more stable and supported than the open source NTFS-3G. But at $40, it’s hard to justify the cost when a free alternative is available. Once I tried the free version from Giveaway-of-the-Day, I know I would have felt cheated if I had spent the money on it. The MacNTFS-3G package is so nicely integrated, there is really no need to fear it. And I’ve had no trouble with it in over a year of use.

The Paragon Giveaway didn’t include the HFS+ driver for Windows, but reports are that Apple will introduce a similar driver with Snow Leopard. I think I’m liking free and bundled software more and more!

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Access NTFS Volumes On Your Mac
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Yup, my Boot Camp drive icon is customized, too!

In my last post, I discussed the simple but confusing steps required to add custom icons to Mac OS X drives. I mentioned that there were some tricks to getting custom icons on some drives, however, including Boot Camp and NTFS disks. This week, I’ll show how to customize these as well.

There are two core problems with customizing drive icons for certain volumes:

1. OS X can’t directly write to anything but HFS+ and FAT, and this includes writing icons
2. OS X needs a special resource on the drive to indicate that the custom icon should be used, and only HFS+ supports this

But it’s easy enough to get around these limitations. Apple takes care of number 2, and we can cover number 1 in one of two ways.

This post is part of my series focused on Apple OS X tips and tricks.

• Access NTFS Volumes On Your Mac
• Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
• How To Move OS X Time Machine Backups To A New Disk
• Hallelujah! OS X Can Reduce PDF File Size!
• Custom Drive Icons in Mac OS X
• OS X Custom Drive Icons 2: Boot Camp and NTFS

Techie Bits

Booooooring generic USB drive icon in OS X

Old versions of Mac OS stored lots of data (aka “resources”) in the resource fork of a file, including icon graphics. However, this became a problem with the advent of networking and cross-platform computing, since no other popular personal computing platform supported two-fork files. OS X moves most resources into the data fork – although resource forks still exist, they are rarely used.

However, it appears to me that although the “icns” data for a drive icon is stored in the data fork of the .VolumeIcon.icns file, Apple still uses a pointer in the resource fork of the root directory of the drive to tell OS X to use that icon. In other words, when a drive is mounted, OS X seems to look in the resource fork of “.” for a flag that says “use the icon in .VolumeIcon.icns” before it will use the icon. I’m not sure this is the case, but I do know that .VolumeIcon.icns has no resource fork (even on HFS+), but it requires a special resource file on other filesystems.

Since only HFS and HFS+ support resource forks, this is a problem. Apple gets around this by saving resource fork data in files with “._” prepended to their name. So the resource fork of “index.html” would be “._index.html” on a FAT drive. Every FAT drive I’ve added a custom icon to includes a file called “._.”, which sounds like the root directory’s resource to me, and this is required for the icon to work.

Getting It Done

Aah, that's a much better icon for my Maxtor 3200!

So you need two files on a non-HFS+ drive to get the custom icon to work:
 

1. .VolumeIcon.icns – the same exact file as on an HFS+ volume, containing the icon data in the data fork
2. ._. – a special file (resource fork, perhaps?) which seems to tell OS X to use the icon

As illustrated previously, adding a custom icon to a FAT drive is exactly the same as on an HFS+ drive – create the icon resource and copy and paste it in the drive’s Info inspector. If we move these two files over to any other OS X-readable drive (including NTFS-formatted Boot Camp partitions), it will use the icon!

OS X can’t write to NTFS, so there are two ways to get these files onto an NTFS drive:

1. Make OS X write to NTFS by installing NTFS-3G
2. Copy the two files over in an OS like XP or Vista that can write to NTFS

[ad#vmware]The first method is simple, but not one I wanted to try. NTFS-3G is great – it’s a reverse-engineered open-source read/write driver for NTFS, and works on lots of platforms. But NTFS is finicky, and I’m not too keen on risking the integrity of the drive with an unsupported driver. If you want to use this method, though, customizing the icons is as simple as it is with FAT, once the driver is installed.

I used method 2 – relying on VMware Fusion to allow me to write the icon data to my NTFS drives and using a FAT-formatted flash drive as an intermediary between OS X and Vista. I created my icons, applied them to my flash drive, then booted up Vista in Fusion and copied the two files over in a DOS window. They’re hidden by default, but that shouldn’t slow you down.

Note that this only affects the icons in OS X – it will not allow you to customize the boot icons.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | OS X Custom Drive Icons 2: Boot Camp and NTFS
This post was categorized as Apple, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What’s Wrong With Thin Provisioning?

As I have explained in my storage seminars, thin provisioning is the opposite of what storage management professionals should be doing: Instead of managing usage, we just throw up our hands and say “you want 500 GB? Fine, you’ve got it!” while all the while only provisioning a fraction of that space. It’s a lie, and is thus bound to catch up with us sooner or later, and probably at just the wrong time.

People use disk space like money – their needs tend to expand to use up all they can get. Tell the users that you just added another 8 TB to the file server and watch their usage spike. Tell a database manager that they need to buy 20 TB and watch as their tablespaces magically start using 19. It’s human nature, and fighting this impulse to consume is precisely what management is all about. Traditional thin provisioning (or “virtual provisioning” in EMC-speak) does exactly this – it “tells” the downstream users of a storage resource that they have more capacity than is actually assigned to them and then grows capacity as it is used. To say that it is controversial is an understatement.

In certain instances, including Drobo and VMware growable disks, this can be beneficial since it’s a pain for most end-user OS configurations to resize a volume after the fact. In these special cases, I concede that thin provisioning is the right way to go. The same could be said for deduplicating or compressed storage – these simply have to be thin provisioned, since the actual allocation is completely abstracted by the compression algorithm. Thin provisioning can also help (slightly) for the OS volumes of virtual servers. But mainstream enterprise users have storage, server, and application managers, so they shouldn’t resort to “tricks” like thin provisioning – instead, they should manage their storage!

But the worst thing about thin provisioning is that it can’t un-provision storage. Let’s say a user uses your thin-provisioned file server as a temporary landing zone while switching to a new laptop. Or your database folks load their LUNs up with SQL dumps after an outage. Or your application folks fill up their test servers prior to going into production. Predictably, that thin-provisioned storage will expand, using up real disk capacity, to take the load (presuming enough capacity is available). The problem arises when they delete this temporary data – the storage array has no way of knowing that those blocks are no longer in use, so it cannot un-provision them. Suddenly your 500 GB thin-provisioned LUN is really taking up 400 GB even though it only has 20 GB of actual data on it, and you feel like a chump. Time to go manage your storage…

Zero-detection helps a little

Now back to 3PAR. Their new T-class InServ storage array has a special ASIC designed to attack a small chunk of the un-provisioning problem. It scans allocated storage, looking for blocks filled with zeros, and de-provisions them. This is nice – it’s a great tool to convert traditional storage to thin-provisioned storage. It’s also the first practical un-provisioning approach I’ve heard about, and might yield some capacity improvements for already-provisioned LUNs in certain special cases, though I’m not sure 3PAR is aiming for this market.

See, it’ll only work for zeroed-out storage, which is sadly extremely rare in the world of storage. It will detect capacity that has never been used, but most filesystems simply change their pointers when a file is deleted – leaving the data just where it was. 3PAR’s effort won’t work in this case. Even decommissioned servers often leave their LUNs full of old data, a security risk to be sure, and not a case that 3PAR could deal with, either.

The only way to make this work for already-used storage would be to add another step to the decommissioning process – zero out LUNs that are no longer in use as a way to send a signal to the storage array that it can un-provision that storage. But of course, we could also just send an email to the storage administrator to de-allocate the LUNs, leaving us in a much better position since we no longer have unused LUNs sitting on the storage array. Maybe we could modify the filesystem to zero out unused storage. Anyone have the source code for NTFS?

Seriously, though, this is a practical step in the right direction. We need better communication between applications, operating systems, and storage in order to enable lots of beneficial features. 3PAR is trying to enable some communication, and I applaud them for that. Just don’t expect too much.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | 3PAR’s Thin Un-Provisioning is Slightly Less Bad
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.