I began by introducing the core problem: Storage isn’t getting any cheaper due to storage utilization and provisioning problems. Thin provisioning isn’t all it’s cracked up to be, since the telephone game makes de-allocation a challenge. So now let’s talk about how to make thin provisioning actually work.

There are 100 different ways of solving the de-allocation problem, some of which have gained some prominence. They all boil down to two options:

1. Make the server super-smart and have it communicate better
2. Make the storage super-smart and have it make educated guesses

There’s only a few ways that the server-side option can be implemented, and we’ll get to that. But first, let’s take a look at a sort of hybrid approach that relies on known server usage patterns: Metadata monitoring.

It’s really hard for the storage to really understand what the server is doing. The best example that I know of is the Drobo sitting under my desk.

I love this little black box. When I got it, I configured it as eight terabytes and I put a 160-gig disk in it. That’s thin provisioning. And over time, I’m swapping out the disks and I’m doing all my stuff, and it still looks like eight terabytes. Add data, delete it, swap disks, and it always just works.

Not a lot of people know how the Drobo works, though. One of the things that people have complained about is that it only supports certain file systems and partition schemes. The reason for this is a “magical” thing it’s doing that relates very, very closely to the topic of this discussion. The Drobo is the first thin provisioning box that I know of that directly monitors the file system.

What the Drobo does is this: It knows where the supported filesystems (HFS+, NTFS, EXT3, and FAT) keep the record of what’s been deleted. So the Drobo it watches that spot and when you delete something, it reclaims that space. No enterprise storage system can do this, and yet this little box under my desk does it all day long.

This is basically the super, ultimate smarts of storage. But, of course, it’s very limited. It faces a real challenge in an enterprise setting because there is much more variety. We have all these layers of virtualization and weird file systems and things like that to worry about. We just can’t expect a product like this to accommodate everybody, so we just can’t expect this kind of smarts to be put everywhere.

Instead, we have a variety of semaphores sent from the server to the storage array that attempt to solve the telephone game. That’s what we’re talking about next.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Monitoring Filesystem Metadata For Thin Provisioning
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Today's super-sized drives need something better than FAT. Introducing exFAT, the next-generation portable filesystem.

Microsoft already gave the world FAT and NTFS, and both have become common in the non-Windows world thanks to flash drives, SD cards, and portable disks. But the folks from Redmond are now introducing a new filesystem, exFAT. Do we really need a new filesystem?

The Rise and Fall of FAT

Although computing is a volatile world, standards tend to stick around way past their prime. Nowhere is this more apparent than the ubiquitous FAT filesystem. Although it has been modified and updated, the basic FAT structure pre-dates DOS: It was designed for Microsoft’s first real product, Standalone Basic, and bears the fingerprints of the company’s first employee, Marc McDonald, as well as some other programmer called Bill Gates.

Appealing in its simplicity, FAT was adopted by Tim Paterson at Seattle Computer Products for his QDOS operating system, which found its way back to Microsoft as IBM’s PC-DOS. FAT was improved with 16-bit cluster addresses for the 1984 IBM PC-AT’s hard disk drive, and enhanced again to match growing disk sizes as FAT32 in Windows 95. Along the way, VFAT added 32-bit file access and long file names.

Because it was simple in concept, FAT was reverse-engineered and adopted as a standard filesystem for data interchange between DOS/Windows PCs and other operating systems, from Apple to Atari to Amiga. Today, FAT16 is the dominant format for SD cards, and most thumb drives and SDHC cards use FAT32. But FAT is not an open standard: Microsoft successfully patented VFAT and FAT32 and specifically requires digital camera and flash drive companies to obtain a license for its use.

Although widely dismissed as archaic, many of the limits of FAT (especially FAT32) are due to factors outside the filesystem, particularly the MBR partition scheme and implementation-related issues. FAT32 is capable on disks up to 2 terabytes, which matches the maximum size of the next-generation SDXC flash card specification. And, despite Microsoft’s patents, FAT is widely implemented and stable on a variety of platforms and embedded devices.

Why exFAT?

Hard disk drives long since switched to NTFS or other operating system-specific formats, so FAT is mainly used on flash media today. Sadly, FAT isn’t all that well-suited for applications like these. Thumb drives don’t really push the envelope for FAT since they tend to hold reasonable numbers of smaller files, but flash media for digital cameras is a different story.

SD cards have taken over the digital camera world: Even holdouts like Sony now support SD and its successors, SDHC and SDXC. And just about every SD and SDHC card sold today uses FAT16 and FAT32, respectively. It works reasonably well for still images, but video poses real issues. FAT32 is limited to 4 GB per file, and HD camcorders and digital cameras can easily push past this size limit. Most models today split files or limit recording time to avoid this limitation, but that’s not optimal.

exFAT enables file sizes up to 16 EB and also adds nifty features like a free space bitmap allowing better use of flash memory and eliminating the read/erase/write issue of NAND. It also allows more-granular timestamps (10 ms, rather than 2 seconds on FAT) which better-match high-speed shooting capabilities of today’s digital cameras.

exFAT Limitations

But not everything is rosy with exFAT. Although it seems poised to take off thanks to strong support from the SDcard Association, exFAT remains encumbered by patents and licensing fees from Microsoft. Memory Stick XC requires the use of exFAT, but SDXC merely suggests it and there is no guarantee of third-party support. Digital camera makers may decide to stick with FAT32 rather than switch, though Microsoft makes it more attractive for embedded uses with a flat $300k fee per manufacturer.

There is also one odd technical limitation to exFAT. Because removable media is often physically ejected at inopportune moments, the FAT filesystem was designed with alternating file allocation tables. Should writing of the table be interrupted, FAT can always fall back on the previous iteration. exFAT curiously uses just one allocation table and free space bitmap, making it more likely to become corrupt in precisely the portable devices it was intended for. A follow-on enhancement, Transactional exFAT or TexFAT, adds redundant tables and bitmaps but may not be implemented.

Unfortunately, the old, inflexible MBR partition scheme remains coupled to exFAT. It would have been nice to ditch MBR in favor of the open GPT format, but the static nature of flash media made that less of a priority. Still, MBR is limited to 2 TB, and this will become a roadblock for flash media at some point.

You might also want to read Introducing SDXC and exFAT in Apple Mac OS X

Stephen’s Stance

Adoption of exFAT seems likely, especially for the SDXC and MSXC flash cards used in digital cameras. Because it is not backwards-compatible with devices, media, operating systems, and card readers, however, consumer confusion is likely to result. Even systems with an SDXC card reader might not be capable of reading the exFAT filesystem contained on the card, and vice versa.

Microsoft, being the proud parent of exFAT, supports the new filesystem in its modern Windows 7 and Server 2008 operating systems and has moved quickly to add it to Windows XP, Server 2003, and Vista. A few exFAT drivers exist for Linux and Android, and Apple is likely to add support for exFAT in Snow Leopard 10.6.5, since the SDXC-capable mid-July iMac and Mac Mini computers already have it in 10.6.4.

So it looks like we’ve got a new filesystem on our hands, at least for camera cards. But exFAT will not likely replace FAT32 on thumb drives any time soon. Their portability requires broad compatibility, and FAT32 has the widest support today. Although exFAT will likely be used in embedded applications, it there is no compelling reason for it to displace NTFS and other operating system filesystems like (Apple) HFS+ and (Linux) ext3/ext4 any time soon.

Image credit: Scale Face by Playingwithbrushes

© sfoskett for Stephen Foskett, Pack Rat, 2010. | An Introduction To exFAT
This post was categorized as Apple, Computer History, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Yup, my Boot Camp drive icon is customized, too!

In my last post, I discussed the simple but confusing steps required to add custom icons to Mac OS X drives. I mentioned that there were some tricks to getting custom icons on some drives, however, including Boot Camp and NTFS disks. This week, I’ll show how to customize these as well.

There are two core problems with customizing drive icons for certain volumes:

1. OS X can’t directly write to anything but HFS+ and FAT, and this includes writing icons
2. OS X needs a special resource on the drive to indicate that the custom icon should be used, and only HFS+ supports this

But it’s easy enough to get around these limitations. Apple takes care of number 2, and we can cover number 1 in one of two ways.

This post is part of my series focused on Apple OS X tips and tricks.

• Access NTFS Volumes On Your Mac
• Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
• How To Move OS X Time Machine Backups To A New Disk
• Hallelujah! OS X Can Reduce PDF File Size!
• Custom Drive Icons in Mac OS X
• OS X Custom Drive Icons 2: Boot Camp and NTFS

Techie Bits

Booooooring generic USB drive icon in OS X

Old versions of Mac OS stored lots of data (aka “resources”) in the resource fork of a file, including icon graphics. However, this became a problem with the advent of networking and cross-platform computing, since no other popular personal computing platform supported two-fork files. OS X moves most resources into the data fork – although resource forks still exist, they are rarely used.

However, it appears to me that although the “icns” data for a drive icon is stored in the data fork of the .VolumeIcon.icns file, Apple still uses a pointer in the resource fork of the root directory of the drive to tell OS X to use that icon. In other words, when a drive is mounted, OS X seems to look in the resource fork of “.” for a flag that says “use the icon in .VolumeIcon.icns” before it will use the icon. I’m not sure this is the case, but I do know that .VolumeIcon.icns has no resource fork (even on HFS+), but it requires a special resource file on other filesystems.

Since only HFS and HFS+ support resource forks, this is a problem. Apple gets around this by saving resource fork data in files with “._” prepended to their name. So the resource fork of “index.html” would be “._index.html” on a FAT drive. Every FAT drive I’ve added a custom icon to includes a file called “._.”, which sounds like the root directory’s resource to me, and this is required for the icon to work.

Getting It Done

Aah, that's a much better icon for my Maxtor 3200!

So you need two files on a non-HFS+ drive to get the custom icon to work:
 

1. .VolumeIcon.icns – the same exact file as on an HFS+ volume, containing the icon data in the data fork
2. ._. – a special file (resource fork, perhaps?) which seems to tell OS X to use the icon

As illustrated previously, adding a custom icon to a FAT drive is exactly the same as on an HFS+ drive – create the icon resource and copy and paste it in the drive’s Info inspector. If we move these two files over to any other OS X-readable drive (including NTFS-formatted Boot Camp partitions), it will use the icon!

OS X can’t write to NTFS, so there are two ways to get these files onto an NTFS drive:

1. Make OS X write to NTFS by installing NTFS-3G
2. Copy the two files over in an OS like XP or Vista that can write to NTFS

[ad#vmware]The first method is simple, but not one I wanted to try. NTFS-3G is great – it’s a reverse-engineered open-source read/write driver for NTFS, and works on lots of platforms. But NTFS is finicky, and I’m not too keen on risking the integrity of the drive with an unsupported driver. If you want to use this method, though, customizing the icons is as simple as it is with FAT, once the driver is installed.

I used method 2 – relying on VMware Fusion to allow me to write the icon data to my NTFS drives and using a FAT-formatted flash drive as an intermediary between OS X and Vista. I created my icons, applied them to my flash drive, then booted up Vista in Fusion and copied the two files over in a DOS window. They’re hidden by default, but that shouldn’t slow you down.

Note that this only affects the icons in OS X – it will not allow you to customize the boot icons.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | OS X Custom Drive Icons 2: Boot Camp and NTFS
This post was categorized as Apple, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.