Mac OS X 10.7 "Lion" lacks many of the storage features we've long hoped for, including ZFS, iSCSI, and USB 3.0

Last week, at WWDC, Apple introduced many of the features found in their next operating system, OS X “Lion”. At that time, I posted an article about the storage features found in this new release, including integrated revision control, a major update to the FileVault encryption package, and additional enterprise storage protocol support. But, like Snow Leopard, Lion still lacks many storage related features, and it doesn’t look like Apple will get around to adding these anytime soon.

A Better Filesystem (ZFS, Please)

See also ZFS: Super File System! and Mac OS X Lion Adds CoreStorage, a Volume Manager (Finally!)

During the development of Mac OS 10.5, “Leopard”, Apple spent a great deal of time working to replace their legacy HFS+ filesystem with ZFS, a much more advanced option. For starters, ZFS would have given Mac OS better reliability and flexibility, and it has been extended to include advanced features for security and capacity optimization.

But the battle between Sun and NetApp over patents related to the development of ZFS cast a shadow over the long needed replacement of HFS+. With Oracle buying Sun and focusing away from infrastructure products like ZFS, Apple seems to have lost interest in replacing their crufty old filesystem.

Instead of adding an advanced filesystem like ZFS, Mac OS X Lion extends HFS+ with versioning and enhanced security. The new Core Data incremental storage technology in Lion would probably have been easier to implement on ZFS, but Apple was able to add it to HFS+, and it will be a lifesaver in the guise of autosave, versions, and resume. The same goes for encryption, with FileVault 2 boasting background full disk encryption, remote wipe, and external drive support.

Those hoping for the integration of ZFS with Mac OS X appear to be out of luck. All components were removed from Snow Leopard, and Lion is moving forward without it. Sadly, this means that Mac OS X still lacks a flexible volume manager, something even Microsoft Windows boasts.

Perhaps the next version of Mac OS X will include friendly volume management features, but it is more likely that Apple will focus away from the filesystem and direct application developers toward the iCloud Storage API. And cloud truly is next-generation storage, making this a leapfrog approach and leaving ZFS in the dust.

Update: Lion does indeed include a full logical volume manager! See Mac OS X Lion Adds CoreStorage, a Volume Manager (Finally!)

USB 3.0: Still AWOL

See also Will OS X 10.7 “Lion” Bring USB 3.0 To The Mac? and my Thunderbolt series

Although Apple may have simply overlooked or neglected to mention it, USB 3.0 apparently made no appearance at WWDC. Thunderbolt is an impressive technology to be sure, and I am bullish on its future application and performance. But “SuperSpeed” USB 3.0 seems poised to seize the baton and become the ubiquitous next-generation interconnect for every day peripherals.

Thunderbolt is a strategic protocol for Apple, and I expect it to rapidly spread across the entire Mac product range. Rather than simply a high-speed interconnect, Thunderbolt will soon enable advanced docking features, as envisioned in my recent post about the iMac as a Thunderbolt peripheral. It will also enable changes to the physical size and shape of laptop and desktop computers and servers, with many suggesting that the next-generation MacBook Air will become the standard Apple laptop.

In contrast, USB 3.0 is simply a performance bump for USB. It is likely that Apple will support USB 3.0 sooner or later, and third-party vendors are already rolling out Mac OS support. CalDigit recently shipped their third USB 3.0 controller for the Mac, and LaCie sells their own “walled garden” card and peripherals as well. I heard rumors that a few vendors are working on Thunderbolt to USB 3.0 bridges and breakout boxes as well.

USB 3.0 will come to the Mac sooner or later, but Thunderbolt is here to stay.

Enterprise iSCSI Support

See also Will Snow Leopard Finally Bring iSCSI To The Mac?

Another technology that Apple has flirted with in Leopard but never delivered is a software initiator for iSCSI, the block storage protocol that runs over Ethernet. We have not heard anything further about iSCSI since 2007, and there was no mention in the Lion introduction either.

Interestingly, Apple did rollout other new enterprise storage protocol options, including NFSv4, DFS, and even integration of Xsan, the Fibre Channel filesystem. Xsan also added ALUA compatible multipathing, a real surprise for storage geeks like me. But iSCSI was nowhere to be found.

This is a real shame, since iSCSI is becoming increasingly common in enterprise storage circles. Convergence on Ethernet is a hot topic right now, and iSCSI for Mac would give exceptional flexibility and interoperability and fit right into the “prosumer” Mac market niche.

Instead, end-users are stuck working with third-party iSCSI initiators, Fibre Channel and Xsan, or NFS. Although I am a fan of their free globalSAN product, Studio Network Solutions does not offer enterprise support for third-party arrays. The other major option for Mac iSCSI is ATTO’s Xtend, which is supported but somewhat expensive. Drobo also offers an iSCSI client for use with their storage arrays, but it is severely limited. An integrated Apple solution would be a welcome addition, both for consumers and enterprise systems administrators.

Stephen’s Stance

Apple is not in enterprise storage company to be sure, and news from WWDC dashed any hopes we had for ZFS and iSCSI support. USB 3.0 seems a foregone conclusion, but Apple seems intent on ignoring it as long as possible. Although I welcome the new storage features included in Lion, it is disappointing that these were left out.

Note that TRIM support was also not mentioned at WWDC, but it is likely included.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Three Key Storage Features Missing in Mac OS X “Lion”
This post was categorized as Apple, Enterprise storage, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Mac OS X 10.7 "Lion" is coming, and it's bringing a few storage features to the table

Apple’s not an enterprise company or a storage company. In fact, they’re rapidly jettisoning both – consider the sad fate of the Xserve RAID. But Apple does have enterprise features and storage features in their operating systems. And Mac OS X 10.7 “Lion” is a great case in point. From Versions to Time Machine Local Snapshots to AirDrop, Lion brings some storage love, and iCloud’s Storage API could be game-changing. Let’s look at what’s new and key in terms of storage in the latest version of Mac OS X.

New and Updated Storage Features in Lion

Lion is strong on consumer-oriented features, of course. And Apple is leading the industry in pushing user-friendly storage features for data protection and sharing. OS features like Core Data incremental storage enable Auto Save and Versions, Time Machine gets local snapshots, and FileVault is updated into a whole-disk encryption (WDE) tool.

Update: Probably the most important storage feature, CoreStorage, went un-covered! Read more at Mac OS X Lion Adds CoreStorage, a Volume Manager (Finally!)

Auto Save, Versions, and Resume

AutoSave and Versions leverage advances in HFS+ and a new Core API

Like iOS, Lion enhances the “back where you were” nature of computing with Auto Save, Versions, and system Resume. These completely change the end-user computing experience: Applications don’t have temporal “use once” interfaces but have lasting, historical state. And the ability to move through time (à la Time Machine)

Of course, lots of applications have had auto-save in the past. But Lion adds OS-level interfaces and APIs to enable applications to save data in a standard way. And these will be integrated with Resume (see below) for a very iOS-like experience.

Lion enables all this by enhancing the old, familiar HFS+ filesystem with Core Data incremental storage, a snapshot-like interface to save and recover multiple point-in-time instances of a single document. This is a delta differencing system, probably on a block level, in the filesystem.

Versions are accessed through the title bar document name, as well as a Time Machine-like interface in some applications. Applications can open up multiple versions of the same document at once, and you can cut and paste between them.

Resume is very cool. Applications using the new Lion APIs can save their state, even through reboots! This is what computers should have always done, but no one ever implemented it. Why should a reboot wipe out where you were last? Although not really a storage feature, resume relies on Auto Save and new APIs to store application state.

Time Machine Local Snapshots

Time Machine in Mac OS X "Lion" includes local snapshots as well as storage of backups on external disks

See also Local Snapshots in Mac OS X Lion Time Machine: Is It A Good Idea?

And speaking of Time Machine, it’s been enhanced with Local Snapshots, a mechanism for storing data on the local drive in addition to an external Time Machine drive. The Time Machine interface combines local snapshots and the backup drive or Time Capsule into a single timeline when browsing.

This appears to use Core Data incremental storage as well, though it’s not definite. And it’s unclear how local snapshots will interact with Versions.

AirDrop

See also Snooping on AirDrop in Apple’s Mac OS X 10.7 “Lion”

AirDrop is a new wireless file sharing protocol. Although it smacks of Wi-Fi Direct or Bluetooth, it’s neither. Instead, AirDrop is a newly-developed proprietary mechanism of sharing data between two Macs over a Wi-Fi link. It’ll be speedy and simple but incompatible with the vast majority of systems out there.

FileVault 2

Face it, FileVault was little-used and unfriendly. That’s all changed with FileVault 2 in Lion. It’s encryption done right and might just push average Apple users to protect their data. Kind of like what Time Machine did for backup. That’s a good thing.

What’s new in FileVault 2?

• Full-disk encryption, rather than an image-based hack.
• Encrypt in place, in the background, while you work. No more waiting and losing your computer for hours or days while encryption is installed. And the encryption process will resume even after a reboot (or two)!
• Encrypt external drives, too. Very sweet, especially for Time Machine backups!
• Remote wipe service, probably through iCloud
• XTS-AES 128 support

FileVault 2 is full-disk only, but supports multiple users (each with their own encrypted home area) by storing the full-disk key in each user’s keychain. They access it by using their login password, just like they always did. Apple will offer an option to store the encryption key in iCloud, but this does not appear to be the default condition.

Of course, users with poor passwords, or those vulnerable to social engineering, will still be vulnerable. But some encryption beats no encryption any day! Lion uses the login window for sleep and screen savers, so FileVault 2 protects in those states as well.

NFSv4

Lion adds support for NFS version 4, which is a welcome update. But it doesn’t appear to support version 4.1, or parallel NFS.

SMB with DFS

DFS is a technology in Microsoft Windows that virtualizes file server shares, improving flexibility. Lion’s SMB client supports DFS when connecting to Windows file servers.

Xsan Built In

Mac OS X Server will now be an optional extra to add to the standard Lion install, rather than a separate SKU. Part of this shift is the integration of Xsan, Apple’s OEM version of Quantum’s StorNext SAN file system. Xsan has been quite popular in the media space on client machines, and it appears that it does not require a Server license.

Xsan also gets case-insensitive volumes, which brings it more in line with the expectations of users used to HFS+. Finally, Lion gets ALUA-compatible multipathing, supporting a wide variety of storage arrays.

Probable: TRIM Support for SSDs

Apple enabled TRIM in certain versions of 10.6 “Snow Leopard”, but it only supported their OEM SSDs. Lion probably gets a fully-supported TRIM implementation that works with any SSD with TRIM, though this was not mentioned at WWDC or online. This is a welcome update, and one too-long in coming!

Stephen’s Stance

Mac OS X “Lion” doesn’t give storage folks much, but it could be a harbinger of massive changes. Although not part of Lion per se, the iCloud Storage API will likely see much use by application developers. But even Lion’s smaller storage feature list is welcome. Simple full-disk encryption in FileVault and Core Data incremental storage are promising. Let’s just hope TRIM comes along for the ride!

On the enterprise side of things, Lion is improving as a storage client with NFSv4, DFS, and Xsan built in. But there’s no mention of an iSCSI client, suggesting that project is dead. Thank goodness for Studio Network Solutions! And we’re still stuck with HFS+, suggesting the ZFS transition is off the table, too. Too bad – Versions and FileVault would have been much easier to implement on ZFS!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Key Storage Features in Mac OS X 10.7 “Lion”
This post was categorized as Apple, Enterprise storage, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I think this sentiment is just as valid today as when I posted it in 2008!

Although many discussions in the storage industry focus on the relative merits of one protocol or another, the conversation occasionally turns to the core issue at hand: We continue to patch together a system based on outdated concepts. Most storage protocols continue to mimic direct attached storage, and most of our so-called networks act as point to point channels. An ultra-modern virtualized storage infrastructure with all the latest bells and whistles still holds the concepts of block and file at its core. Whenever the storage industry has tried to bring about real storage management they have been stymied by a lack of context for data.

No amount of virtualization, and no new protocol, will fix this. Put simply, we need a storage revolution.

Channels, Blocks, and Files

Most innovation in the 1980s and early 1990s focused on moving storage out of the server. SCSI allowed disk to exist in a separate cabinet, RAID allowed multiple physical disks to become a single virtual one, and these were mixed to become the prototype storage array. Although SCSI allowed one-to-many connectivity, it was never a true peer-to-peer network, even once it was mixed with network concepts in the form of Fibre Channel.

Even today, SAN storage is focused on providing faster, more flexible, and feature-packed direct-attached storage. A modern virtual SAN hides a complex arrangement of caching, data protection, tiered storage, replication, and deduplication, masquerading the lot as a simple, lowly disk drive. It is sad but true that all of our work as an industry has been dedicated to recreating what we started with.

Networked file-based storage is no better. Although NAS devices have all the advanced features of their SAN cousins, they must present a simple file tree to the host to retain compatibility. File virtualization merely presents a larger homogenous tree.

Inside the server, too, features and complexity are hidden to retain a familiar file system format. Volume managers can do anything a virtualization device can, but must present their output as a simple (though virtual) disk drive. File systems, too, have added features but still present a familiar tree of mount points, inodes, and files. Even ZFS, possibly the most advanced combination of volume management and file system technology yet, must present a simple tree of storage to applications.

The Metadata Roadblock

This outdated paradigm, of disks and file trees, is ill-suited to today’s storage challenges. Data must be categorized so actions can be taken to preserve or destroy it based on policies. Data must be searchable so users and applications can find what they want. Data must be flexible so it can be used in new ways. Our antiquated notions are not capable of meeting these challenges.

One simple problem is that we lack context for our data. Most file systems merely assign to a file a name, location, owner, and security attributes. The most advanced can contain extended metadata, but this is rarely seen in practice since many applications cannot agree on how to use this data. Microsoft’s Office suite can store and share extended file attributes, for example, but these live inside the file rather than in the file system. The promise of expanded Office attributes is only realized in conjunction with a content management system like SharePoint which lies above the lowly file system.

What if the storage system could keep this data instead? What if it could logically group files according to project or client, mining keywords and authors, and maintaining revisions? These concepts are not new, having been implemented in content management systems for years, and certain elements appeared in file systems, like Apple’s HFS and VMS’ Files-11, for decades.

Cut Down the Tree

File metadata would allow advanced features, but truly taking advantage of them requires a more fundamental shift in the way applications access files. Rather than sticking to a traditional hierarchy of directories in a tree (which was, after all, simply a primitive metadata system), we should remove the tree altogether. Allow files to become data objects, identified by arbitrary attributes and managed according to an overarching policy.

This future vision is decidedly different from our current notion of storage, but is not so far off. Many organizations now rely on central data warehouses based on SQL-language relational databases. As many storage managers have grumbled, databases tend to ignore storage management concepts entirely, managing their own content independently.

But not all applications need a database back-end, so another initiative seeks to provide generic object storage for wider use. Called content-addressable storage or CAS, these devices have traditionally been used only for archival purposes, since that was their first market application. As vendors break free of proprietary interfaces in favor of open ones like XAM, CAS could transform storage itself by eliminating both file and block storage at once.

Similar concepts are already at work in the so-called Web 2.0 world. Non-traditional databases like Google BigTable, Amazon S3, and Hadoop allow massive scalability for object storage. API-sharing initiatives with many Web 2.0 companies can be seen as similar prototypical object storage frameworks. Any of these could be leveraged to provide a new world of data storage, and many are gaining traction even now.

Stephen’s Stance

Although traditional block storage is here to stay for disk drives, and tree-type file systems are likely to remain the foundation of operating system storage, new object-based concepts could change the world in fundamental ways. As applications become “web aware”, they also become object aware, increasing the likelihood of such a storage revolution. For the majority of applications, this new world would be a welcome one indeed.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | We Need a Storage Revolution
This post was categorized as Apple, Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I began by introducing the core problem: Storage isn’t getting any cheaper due to storage utilization and provisioning problems. Thin provisioning isn’t all it’s cracked up to be, since the telephone game makes de-allocation a challenge. So now let’s talk about how to make thin provisioning actually work.

There are 100 different ways of solving the de-allocation problem, some of which have gained some prominence. They all boil down to two options:

1. Make the server super-smart and have it communicate better
2. Make the storage super-smart and have it make educated guesses

There’s only a few ways that the server-side option can be implemented, and we’ll get to that. But first, let’s take a look at a sort of hybrid approach that relies on known server usage patterns: Metadata monitoring.

It’s really hard for the storage to really understand what the server is doing. The best example that I know of is the Drobo sitting under my desk.

I love this little black box. When I got it, I configured it as eight terabytes and I put a 160-gig disk in it. That’s thin provisioning. And over time, I’m swapping out the disks and I’m doing all my stuff, and it still looks like eight terabytes. Add data, delete it, swap disks, and it always just works.

Not a lot of people know how the Drobo works, though. One of the things that people have complained about is that it only supports certain file systems and partition schemes. The reason for this is a “magical” thing it’s doing that relates very, very closely to the topic of this discussion. The Drobo is the first thin provisioning box that I know of that directly monitors the file system.

What the Drobo does is this: It knows where the supported filesystems (HFS+, NTFS, EXT3, and FAT) keep the record of what’s been deleted. So the Drobo it watches that spot and when you delete something, it reclaims that space. No enterprise storage system can do this, and yet this little box under my desk does it all day long.

This is basically the super, ultimate smarts of storage. But, of course, it’s very limited. It faces a real challenge in an enterprise setting because there is much more variety. We have all these layers of virtualization and weird file systems and things like that to worry about. We just can’t expect a product like this to accommodate everybody, so we just can’t expect this kind of smarts to be put everywhere.

Instead, we have a variety of semaphores sent from the server to the storage array that attempt to solve the telephone game. That’s what we’re talking about next.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Monitoring Filesystem Metadata For Thin Provisioning
This post was categorized as Computer History, Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Today's super-sized drives need something better than FAT. Introducing exFAT, the next-generation portable filesystem.

Microsoft already gave the world FAT and NTFS, and both have become common in the non-Windows world thanks to flash drives, SD cards, and portable disks. But the folks from Redmond are now introducing a new filesystem, exFAT. Do we really need a new filesystem?

The Rise and Fall of FAT

Although computing is a volatile world, standards tend to stick around way past their prime. Nowhere is this more apparent than the ubiquitous FAT filesystem. Although it has been modified and updated, the basic FAT structure pre-dates DOS: It was designed for Microsoft’s first real product, Standalone Basic, and bears the fingerprints of the company’s first employee, Marc McDonald, as well as some other programmer called Bill Gates.

Appealing in its simplicity, FAT was adopted by Tim Paterson at Seattle Computer Products for his QDOS operating system, which found its way back to Microsoft as IBM’s PC-DOS. FAT was improved with 16-bit cluster addresses for the 1984 IBM PC-AT’s hard disk drive, and enhanced again to match growing disk sizes as FAT32 in Windows 95. Along the way, VFAT added 32-bit file access and long file names.

Because it was simple in concept, FAT was reverse-engineered and adopted as a standard filesystem for data interchange between DOS/Windows PCs and other operating systems, from Apple to Atari to Amiga. Today, FAT16 is the dominant format for SD cards, and most thumb drives and SDHC cards use FAT32. But FAT is not an open standard: Microsoft successfully patented VFAT and FAT32 and specifically requires digital camera and flash drive companies to obtain a license for its use.

Although widely dismissed as archaic, many of the limits of FAT (especially FAT32) are due to factors outside the filesystem, particularly the MBR partition scheme and implementation-related issues. FAT32 is capable on disks up to 2 terabytes, which matches the maximum size of the next-generation SDXC flash card specification. And, despite Microsoft’s patents, FAT is widely implemented and stable on a variety of platforms and embedded devices.

Why exFAT?

Hard disk drives long since switched to NTFS or other operating system-specific formats, so FAT is mainly used on flash media today. Sadly, FAT isn’t all that well-suited for applications like these. Thumb drives don’t really push the envelope for FAT since they tend to hold reasonable numbers of smaller files, but flash media for digital cameras is a different story.

SD cards have taken over the digital camera world: Even holdouts like Sony now support SD and its successors, SDHC and SDXC. And just about every SD and SDHC card sold today uses FAT16 and FAT32, respectively. It works reasonably well for still images, but video poses real issues. FAT32 is limited to 4 GB per file, and HD camcorders and digital cameras can easily push past this size limit. Most models today split files or limit recording time to avoid this limitation, but that’s not optimal.

exFAT enables file sizes up to 16 EB and also adds nifty features like a free space bitmap allowing better use of flash memory and eliminating the read/erase/write issue of NAND. It also allows more-granular timestamps (10 ms, rather than 2 seconds on FAT) which better-match high-speed shooting capabilities of today’s digital cameras.

exFAT Limitations

But not everything is rosy with exFAT. Although it seems poised to take off thanks to strong support from the SDcard Association, exFAT remains encumbered by patents and licensing fees from Microsoft. Memory Stick XC requires the use of exFAT, but SDXC merely suggests it and there is no guarantee of third-party support. Digital camera makers may decide to stick with FAT32 rather than switch, though Microsoft makes it more attractive for embedded uses with a flat $300k fee per manufacturer.

There is also one odd technical limitation to exFAT. Because removable media is often physically ejected at inopportune moments, the FAT filesystem was designed with alternating file allocation tables. Should writing of the table be interrupted, FAT can always fall back on the previous iteration. exFAT curiously uses just one allocation table and free space bitmap, making it more likely to become corrupt in precisely the portable devices it was intended for. A follow-on enhancement, Transactional exFAT or TexFAT, adds redundant tables and bitmaps but may not be implemented.

Unfortunately, the old, inflexible MBR partition scheme remains coupled to exFAT. It would have been nice to ditch MBR in favor of the open GPT format, but the static nature of flash media made that less of a priority. Still, MBR is limited to 2 TB, and this will become a roadblock for flash media at some point.

You might also want to read Introducing SDXC and exFAT in Apple Mac OS X

Stephen’s Stance

Adoption of exFAT seems likely, especially for the SDXC and MSXC flash cards used in digital cameras. Because it is not backwards-compatible with devices, media, operating systems, and card readers, however, consumer confusion is likely to result. Even systems with an SDXC card reader might not be capable of reading the exFAT filesystem contained on the card, and vice versa.

Microsoft, being the proud parent of exFAT, supports the new filesystem in its modern Windows 7 and Server 2008 operating systems and has moved quickly to add it to Windows XP, Server 2003, and Vista. A few exFAT drivers exist for Linux and Android, and Apple is likely to add support for exFAT in Snow Leopard 10.6.5, since the SDXC-capable mid-July iMac and Mac Mini computers already have it in 10.6.4.

So it looks like we’ve got a new filesystem on our hands, at least for camera cards. But exFAT will not likely replace FAT32 on thumb drives any time soon. Their portability requires broad compatibility, and FAT32 has the widest support today. Although exFAT will likely be used in embedded applications, it there is no compelling reason for it to displace NTFS and other operating system filesystems like (Apple) HFS+ and (Linux) ext3/ext4 any time soon.

Image credit: Scale Face by Playingwithbrushes

© sfoskett for Stephen Foskett, Pack Rat, 2010. | An Introduction To exFAT
This post was categorized as Apple, Computer History, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Seagate Expansion Portable packs 1 TB into a compact single-cable box, but our initial experience with a DOA unit wasn't positive

The Tech Field Day events I organize generate a massive amount of HD video content, and moving half a terabyte or more of data is a real issue. We had been using luggable desktop drives from Western Digital and Seagate, but preferred a smaller, lighter, USB bus-powered portable solution. The Seagate Expansion Portable USB drive we bought this week packs an amazing 1 TB of capacity, but our experience with the product was mixed at best.

Exploring the Seagate Expansion Portable

“Expansion” is Seagate’s basic (read “inexpensive”) external drive line, designed more for ease of use than portability, performance, or features. The mainstream FreeAgent line includes more bundled software and flexibility to be used in docks, media players, and more. Since I didn’t need any of this, I was happy to stick with the basic Expansion line.

I picked the “Expansion Portable” which is smaller and bus-powered, thanks to its use of a smaller 2.5″ hard disk drive mechanism. 1 TB is an amazing capacity point in a 2.5″ drive (see below), but the expansion is not as slim as other portable drives. Indeed, it’s just plain chunky, despite its curving piano-black case. The drive came in a plain-ish cardboard box (printed, unlike WD’s basic line) and included a short USB-to-mini-USB cable, a fold-out booklet, and not much else.

Trouble: DOA and Format

Our troubles started when plugged the brand new drive into a MacBook Pro to transfer some video from VMworld. Although the drive audibly spun up and the green light came on, it was not visible to the computer at all. We tried it on different ports, with different cables, and on different computers with no luck. It appeared to be working but was unusable. A quick check on Amazon reveals many similar comments. What’s the issue here, Seagate? Sounds like a bad batch of controllers.

After exchanging our defective drive for a working one (thank goodness we bought it at a local Office Depot rather than via mail order) we hit our next snag. The drive comes formatted with NTFS using an MBR partition scheme. Since the source computer was running Mac OS X 10.5, it could not write to NTFS. The videographer couldn’t reformat it to HFS+ due to its size and partitioning scheme. We had to use Disk Utility to repartition it to GPT before we could get it to work. Although not difficult, I can’t imagine most Mac users would think of changing the partition scheme! Admittedly, Seagate does not claim OS X compatibility for this drive: The box includes Windows logos only. It is too bad the Windows world still sticks to MBR when GPT is so much more functional, flexible, and scalable.

The drive seems to work fine now that we’ve done all this, however. It is no speed demon (see below) but packs tons of data into a small physical unit with no bulky power adapter to bring along. Let’s hope the DOA problem is not a sign of unreliability for this unit as well!

1 TB in a 2.5″ Bus-Powered Drive

Packing 1 TB into a 2.5″ form factor hard disk drive is more amazing even than it sounds. Although a few vendors sell these beasts, it’s a stretch to make it work at current platter densities. Rather than sticking to the 1- or 2-platter format found in common 9.5 or 12.5 mm laptop hard disk drives, vendors have had to use 3 platters, each at 333 GB or so, to reach 1 TB. Since 3 platters barely fit in a 12.5 mm case, 1 TB laptop drives are rare.

In Seagate’s case, they didn’t bother minimizing the size of the drive. Instead, they use a comparatively chunky case that is nearly 15 mm high, perhaps enough even for four platters. This is why the Expansion Portable is almost twice as tall as other USB-powered portable drives, and why the hard disk inside isn’t available for purchase as a bare drive. For reference, the drive inside is a Seagate “FreePlay” ST91000430AS mechanism and is accessed by removing two screws under the “feed” at the front of the case.

Performance (or Lack Thereof)

Interested in hard disk performance? Check out my “Four Horsemen of Storage Performance” series!

No USB 2.0 drive is very fast, which is why the storage industry is transitioning to USB 3.0 and eSATA. I have almost a dozen different USB hard drive connectors and none can top 25 MB/s in sequential throughput, a pathetic number when one considers that a mainstream 5400 rpm hard disk can do four to six times that using SATA. FireWire is faster, but SATA and USB 3.0 have them both beat.

Therefore, it is no surprise that the Expansion Portable pokes along at around 22 MB/s during data transfer. This means that 18 minutes of HD video (weighing in at 15 GB) took 12 minutes to copy. We’ll have to budget 8.5 hours to copy all the video we’ll capture at our next two-day Tech Field Day event, but at least the little Seagate can hold this much data!

Although sequential performance is limited by USB 2.0, random performance on these drives is much more telling. The big Seagate was somewhat quicker than an old 160 GB Maxtor drive but rather slower than either a 320 GB Verbatim or 1.5 TB desktop Seagate USB drive in random read and write. This poor showing indicates that the drive just isn’t engineered for performance, and buyers should not expect much from it.

Seagate chose a slow USB chipset for their Expansion Portable drive

Update: Head-to-head tests reveal that Seagate’s choice of USB controllers isn’t helping anything from a performance standpoint, either. It was easily 30% slower than the USB controller used in a Verbatim enclosure, and almost half the speed of the FireWire chip found in that box!

Stephen’s Stance

The Seagate Expansion Portable is a reasonably-priced device with massive 1 TB capacity. It performs adequately, limited more by USB than the hard disk mechanism used, but isn’t exemplary in this respect. It is physically larger than other devices, especially in terms of thickness, but remains portable since it is USB-powered and thus does not require a power brick.

Macintosh users not yet using Snow Leopard will want to repartition and reformat the drive before use. Although reformatting USB drives as HFS+ is familiar to many Mac users, the need to change the partition type is a little more unusual.

Our initial experience was marred by a DOA unit, and online reports indicate that this was not an isolated case. Due to this failure, I cannot recommend the 1 TB Seagate Expansion Portable at this point. The hard disk drive mechanism seems fine, since it spun up without issue, but the drive never even appeared indicating a defective USB controller. Seagate should take steps to improve quality control before these failures damage their reputation!

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Review: 1 TB Seagate Expansion Portable USB Drive (ST910004EXA101-RK)
This post was categorized as Apple, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

See, Snow Leopard’s version of HFS+ allows per-file compression using three very creative filesystem hacks. I’ll let John Siracusa from Ars Technica take the story from here, and I urge you to read John’s complete (and very, very long) Snow Leopard review!

In Snow Leopard, other kinds of files climb on board the compression bandwagon. To give just one example, ninety-seven percent of the executable files in Snow Leopard are compressed. How compressed? Let’s look:

% cd Applications/Mail.app/Contents/MacOS

% ls -l Mail

-rwxr-xr-x@ 1 root wheel 0 Jun 18 19:35 Mail

Boy, that’s, uh, pretty small, huh? Is this really an executable or what? Let’s check our assumptions.

% file Applications/Mail.app/Contents/MacOS/Mail

Applications/Mail.app/Contents/MacOS/Mail: empty

Yikes! What’s going on here? Well, what I didn’t tell you is that the commands shown above were run from a Leopard system looking at a Snow Leopard disk. In fact, all compressed Snow Leopard files appear to contain zero bytes when viewed from a pre-Snow Leopard version of Mac OS X. (They look and act perfectly normal when booted into Snow Leopard, of course.)

So, where’s the data? The little “@” at the end of the permissions string in the ls output above (a feature introduced in Leopard) provides a clue. Though the Mail executable has a zero file size, it does have some extended attributes:

% xattr -l Applications/Mail.app/Contents/MacOS/Mail

com.apple.ResourceFork:

0000 00 00 01 00 00 2C F5 F2 00 2C F4 F2 00 00 00 32 …..,…,…..2

0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….

(184,159 lines snipped)

2CF610 63 6D 70 66 00 00 00 0A 00 01 FF FF 00 00 00 00 cmpf…………

2CF620 00 00 00 00 ….

com.apple.decmpfs:

0000 66 70 6D 63 04 00 00 00 A0 82 72 00 00 00 00 00 fpmc……r…..

Ah, there’s all the data. But wait, it’s in the resource fork? Weren’t those deprecated about eight years ago? Indeed they were. What you’re witnessing here is yet another addition to Apple’s favorite file system hobbyhorse, HFS+.

At the dawn of Mac OS X, Apple added journaling, symbolic links, and hard links. In Tiger, extended attributes and access control lists were incorporated. In Leopard, HFS+ gained support for hard links to directories. In Snow Leopard, HFS+ learns another new trick: per-file compression.

The presence of the com.apple.decmpfs attribute is the first hint that this file is compressed. This attribute is actually hidden from the xattr command when booted into Snow Leopard. But from a Leopard system, which has no knowledge of its special significance, it shows up as plain as day.

Even more information is revealed with the help of Mac OS X Internals guru Amit Singh’s hfsdebug program, which has quietly been updated for Snow Leopard.

% hfsdebug /Applications/Mail.app/Contents/MacOS/Mail

…

compression magic = cmpf

compression type = 4 (resource fork has compressed data)

uncompressed size = 7500336 bytes

And sure enough, as we saw, the resource fork does indeed contain the compressed data. Still, why the resource fork? It’s all part of Apple’s usual, clever backward-compatibility gymnastics. A recent example is the way that hard links to directories show up—and function—as aliases when viewed from a pre-Leopard version of Mac OS X.

In the case of a HFS+ compression, Apple was (understandably) unable to make pre-Snow Leopard systems read and interpret the compressed data, which is stored in ways that did not exist at the time those earlier operating systems were written. But rather than letting applications (and users) running on pre-10.6 systems choke on—or worse, corrupt through modification—the unexpectedly compressed file contents, Apple has chosen to hide the compressed data instead.

And where can the complete contents of a potentially large file be hidden in such a way that pre-Snow Leopard systems can still copy that file without the loss of data? Why, in the resource fork, of course. The Finder has always correctly preserved Mac-specific metadata and both the resource and data forks when moving or duplicating files. In Leopard, even the lowly cp and rsync commands will do the same. So while it may be a little bit spooky to see all those “empty” 0 KB files when looking at a Snow Leopard disk from a pre-Snow Leopard OS, the chance of data loss is small, even if you move or copy one of the files.

The resource fork isn’t the only place where Apple has decided to smuggle compressed data. For smaller files, hfsdebug shows the following:

% hfsdebug /etc/asl.conf

…

compression magic = cmpf

compression type = 3 (xattr has compressed data)

uncompressed size = 860 bytes

Here, the data is small enough to be stored entirely within an extended attribute, albeit in compressed form. And then, the final frontier:

% hfsdebug /Volumes/Snow Time/Applications/Mail.app/Contents/PkgInfo

…

compression magic = cmpf

compression type = 3 (xattr has inline data)

uncompressed size = 8 bytes

That’s right, an entire file’s contents stored uncompressed in an extended attribute. In the case of a standard PkgInfo file like this one, those contents are the four-byte classic Mac OS type and creator codes.

% xattr -l Applications/Mail.app/Contents/PkgInfo

com.apple.decmpfs:

0000 66 70 6D 63 03 00 00 00 08 00 00 00 00 00 00 00 fpmc…………

0010 FF 41 50 50 4C 65 6D 61 6C .APPLemal

There’s still the same “fpmc…” preamble seen in all the earlier examples of the com.apple.decmpfs attribute, but at the end of the value, the expected data appears as plain as day: type code “APPL” (application) and creator code “emal” (for the Mail application—cute, as per classic Mac OS tradition).

You may be wondering, if this is all about data compression, how does storing eight uncompressed bytes plus a 17-byte preamble in an extended attribute save any disk space? The answer to that lies in how HFS+ allocates disk space. When storing information in a data or resource fork, HFS+ allocates space in multiples of the file system’s allocation block size (4 KB, by default). So those eight bytes will take up a minimum of 4,096 bytes if stored in the traditional way. When allocating disk space for extended attributes, however, the allocation block size is not a factor; the data is packed in much more tightly. In the end, the actual space saved by storing those 25 bytes of data in an extended attribute is over 4,000 bytes.

But compression isn’t just about saving disk space. It’s also a classic example of trading CPU cycles for decreased I/O latency and bandwidth. Over the past few decades, CPU performance has gotten better (and computing resources more plentiful—more on that later) at a much faster rate than disk performance has increased. Modern hard disk seek times and rotational delays are still measured in milliseconds. In one millisecond, a 2 GHz CPU goes through two million cycles. And then, of course, there’s still the actual data transfer time to consider.

Granted, several levels of caching throughout the OS and hardware work mightily to hide these delays. But those bits have to come off the disk at some point to fill those caches. Compression means that fewer bits have to be transferred. Given the almost comical glut of CPU resources on a modern multi-core Mac under normal use, the total time needed to transfer a compressed payload from the disk and use the CPU to decompress its contents into memory will still usually be far less than the time it’d take to transfer the data in uncompressed form.

That explains the potential performance benefits of transferring less data, but the use of extended attributes to store file contents can actually make things faster, as well. It all has to do with data locality.

If there’s one thing that slows down a hard disk more than transferring a large amount of data, it’s moving its heads from one part of the disk to another. Every move means time for the head to start moving, then stop, then ensure that it’s correctly positioned over the desired location, then wait for the spinning disk to put the desired bits beneath it. These are all real, physical, moving parts, and it’s amazing that they do their dance as quickly and efficiently as they do, but physics has its limits. These motions are the real performance killers for rotational storage like hard disks.

The HFS+ volume format stores all its information about files—metadata—in two primary locations on disk: the Catalog File, which stores file dates, permissions, ownership, and a host of other things, and the Attributes File, which stores “named forks.”

Extended attributes in HFS+ are implemented as named forks in the Attributes File. But unlike resource forks, which can be very large (up to the maximum file size supported by the file system), extended attributes in HFS+ are stored “inline” in the Attributes File. In practice, this means a limit of about 128 bytes per attribute. But it also means that the disk head doesn’t need to take a trip to another part of the disk to get the actual data.

As you can imagine, the disk blocks that make up the Catalog and Attributes files are frequently accessed, and therefore more likely than most to be in a cache somewhere. All of this conspires to make the complete storage of a file, including both its metadata in its data, within the B-tree-structured Catalog and Attributes files an overall performance win. Even an eight-byte payload that balloons to 25 bytes is not a concern, as long as it’s still less than the allocation block size for normal data storage, and as long as it all fits within a B-tree node in the Attributes File that the OS has to read in its entirety anyway.

There are other significant contributions to Snow Leopard’s reduced disk footprint (e.g., the removal of unnecessary localizations and “designable.nib” files) but HFS+ compression is by far the most technically interesting.

via Mac OS X 10.6 Snow Leopard: the Ars Technica review – Ars Technica.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Bizarre HFS+ Tricks in Mac OS X 10.6 Snow Leopard
This post was categorized as Apple, Computer History. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

1. Use the excellent and free NTFS-3G driver
2. Buy Paragon Software’s Paragon NTFS

I’ve been using the NTFS-3G driver myself for quite some time. It works as advertised: Install it and you can read and write to NTFS-formatted drives with ease. These drives are integrated right into the Finder, appearing just like Mac-standard HFS+ disks. This is the method I used to assign custom drive icon for my Boot Camp volume and USB drives, for example.

Note: Snow Leopard’s NTFS driver is read/write capable. See my post, How To Write To Windows NTFS Drives In Mac OS X 10.6 “Snow Leopard”

This post is part of my series focused on Apple OS X tips and tricks.

• Access NTFS Volumes On Your Mac
• Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
• How To Move OS X Time Machine Backups To A New Disk
• Hallelujah! OS X Can Reduce PDF File Size!
• Custom Drive Icons in Mac OS X
• OS X Custom Drive Icons 2: Boot Camp and NTFS

NTFS is a userspace (FUSE) driver, so installing it requires the installation of mac-fuse as well. It’s been nicely packaged for install over at MacNTFS-3G, or you can install it with MacPorts, along with lots of other great UNIX software.

Paragon NTFS functions similarly to NTFS-3G. The latest versions also include an HFS+ driver for Windows, allowing your Boot Camp instance to read your Mac drive, which is pretty cool. I was initially interested in Paragon’s software, thinking it would be more stable and supported than the open source NTFS-3G. But at $40, it’s hard to justify the cost when a free alternative is available. Once I tried the free version from Giveaway-of-the-Day, I know I would have felt cheated if I had spent the money on it. The MacNTFS-3G package is so nicely integrated, there is really no need to fear it. And I’ve had no trouble with it in over a year of use.

The Paragon Giveaway didn’t include the HFS+ driver for Windows, but reports are that Apple will introduce a similar driver with Snow Leopard. I think I’m liking free and bundled software more and more!

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Access NTFS Volumes On Your Mac
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Yup, my Boot Camp drive icon is customized, too!

In my last post, I discussed the simple but confusing steps required to add custom icons to Mac OS X drives. I mentioned that there were some tricks to getting custom icons on some drives, however, including Boot Camp and NTFS disks. This week, I’ll show how to customize these as well.

There are two core problems with customizing drive icons for certain volumes:

1. OS X can’t directly write to anything but HFS+ and FAT, and this includes writing icons
2. OS X needs a special resource on the drive to indicate that the custom icon should be used, and only HFS+ supports this

But it’s easy enough to get around these limitations. Apple takes care of number 2, and we can cover number 1 in one of two ways.

This post is part of my series focused on Apple OS X tips and tricks.

• Access NTFS Volumes On Your Mac
• Clean Up Your Mac! Essential OS X Tidiness Tools and Techniques
• How To Move OS X Time Machine Backups To A New Disk
• Hallelujah! OS X Can Reduce PDF File Size!
• Custom Drive Icons in Mac OS X
• OS X Custom Drive Icons 2: Boot Camp and NTFS

Techie Bits

Booooooring generic USB drive icon in OS X

Old versions of Mac OS stored lots of data (aka “resources”) in the resource fork of a file, including icon graphics. However, this became a problem with the advent of networking and cross-platform computing, since no other popular personal computing platform supported two-fork files. OS X moves most resources into the data fork – although resource forks still exist, they are rarely used.

However, it appears to me that although the “icns” data for a drive icon is stored in the data fork of the .VolumeIcon.icns file, Apple still uses a pointer in the resource fork of the root directory of the drive to tell OS X to use that icon. In other words, when a drive is mounted, OS X seems to look in the resource fork of “.” for a flag that says “use the icon in .VolumeIcon.icns” before it will use the icon. I’m not sure this is the case, but I do know that .VolumeIcon.icns has no resource fork (even on HFS+), but it requires a special resource file on other filesystems.

Since only HFS and HFS+ support resource forks, this is a problem. Apple gets around this by saving resource fork data in files with “._” prepended to their name. So the resource fork of “index.html” would be “._index.html” on a FAT drive. Every FAT drive I’ve added a custom icon to includes a file called “._.”, which sounds like the root directory’s resource to me, and this is required for the icon to work.

Getting It Done

Aah, that's a much better icon for my Maxtor 3200!

So you need two files on a non-HFS+ drive to get the custom icon to work:
 

1. .VolumeIcon.icns – the same exact file as on an HFS+ volume, containing the icon data in the data fork
2. ._. – a special file (resource fork, perhaps?) which seems to tell OS X to use the icon

As illustrated previously, adding a custom icon to a FAT drive is exactly the same as on an HFS+ drive – create the icon resource and copy and paste it in the drive’s Info inspector. If we move these two files over to any other OS X-readable drive (including NTFS-formatted Boot Camp partitions), it will use the icon!

OS X can’t write to NTFS, so there are two ways to get these files onto an NTFS drive:

1. Make OS X write to NTFS by installing NTFS-3G
2. Copy the two files over in an OS like XP or Vista that can write to NTFS

[ad#vmware]The first method is simple, but not one I wanted to try. NTFS-3G is great – it’s a reverse-engineered open-source read/write driver for NTFS, and works on lots of platforms. But NTFS is finicky, and I’m not too keen on risking the integrity of the drive with an unsupported driver. If you want to use this method, though, customizing the icons is as simple as it is with FAT, once the driver is installed.

I used method 2 – relying on VMware Fusion to allow me to write the icon data to my NTFS drives and using a FAT-formatted flash drive as an intermediary between OS X and Vista. I created my icons, applied them to my flash drive, then booted up Vista in Fusion and copied the two files over in a DOS window. They’re hidden by default, but that shouldn’t slow you down.

Note that this only affects the icons in OS X – it will not allow you to customize the boot icons.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | OS X Custom Drive Icons 2: Boot Camp and NTFS
This post was categorized as Apple, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.