Advanced Format is on the rise, with Western Digital and now Hitachi GST supporting the 4K drive addressing method

Given the IDEMA’s statement that all drive makers will transition to Advanced Format as of January, 2011, this article has been substantially updated.

Another limit is being pushed in computers: The 32-bit LBA addressing mode. Hard disk drives have simply become too big for the 2.1 TB allowed by 32-bit LBA and 512 K sectors. Western Digital was first to answer this challenge with “Advanced Format“, and Seagate took an alternate 48-bit LBA route while also supporting AF. Now Hitachi GST introduced an Advanced Format drive of their own. Will the industry ever adopt 48-bit LBA?

Two Options Past 2.1 TB

As explained in detail in my piece, Taming Monster Disk Drives: 3 TB and Beyond!, the current hard disk addressing scheme tops out around 2.1 TB. There are two simple factors in this equation:

1. There are 4,294,967,296 addressable blocks in a 32-bit “namespace”, and this is all most PCs can handle
2. Each block is 512 bytes in size, and has been since dinosaurs sold the computers

If we multiply 4,294,967,296 times 512, we get 2,147,483,648 KB of capacity. In wacky base-10 storage industry speak, this is 2.15 TB of capacity. No hard disk drive can be larger than this while obeying these limits.

The solution to the problem is simple: Increase one value or the other and the issue vanishes. Although older computers can’t handle a change to either, it’s equally simple to engineer hardware and software that can.

The hard disk drive manufacturers have two roads forward, and they are not mutually-exclusive:

1. Western Digital was first out with an “Advanced Format” drive late in 2009. This addresses the (archaic) block size limit, increasing it from 512 bytes to 4 kilobytes. With each address now eight times larger, we need one-eighth as many addresses for a given drive size. Advanced Format drives can grow to 16 TB without hitting the 32-bit LBA size limit.
2. Seagate attacked the other variable, increasing the address space to the full 48 bits specified in the LBA standard. This gives an amazing 128 petabytes of capacity even with tiny 512 byte blocks.

Both of these solutions have issues with older systems, but many current machines can handle either with ease. Apple Macintosh machines are set, since they use modern EFI firmware and have had capable driver software since OS X 10.4 “Tiger”. PCs running recent versions of Windows can generally use larger disks as long as long as they don’t try to boot from them. External USB or FireWire drives ought to be fine with every computer, too.

Hitachi GST Joins Western Digital With Advanced Format

The 4K “Advanced Format” solution is gaining traction, with both Western Digital and Hitachi GST currently selling drives supporting it. Seagate introduced 48-bit addressing and has also begun to support Advanced Format, quietly introducing the technology in their Momentus line. Toshiba and Samsung are expected to adopt Advanced Format soon as well.

Interestingly, Hitachi GST introduced Advanced Format on a line of 2.5-inch drives that top out at just 750 GB. It is curious that they chose to introduce it in a product line that will not likely breach the 2 TB limit in its lifetime. It is likely that this is a sign of the company’s commitment, however, and that future 3.5-inch products will share the same Advanced Format controller chipsets used here. See the excellent Hitachi Advanced Format brief for more information on the technology.

Stephen’s Stance

I see the innovations of 48-bit LBA and Advanced Format as a welcome sign of change in the hard disk drive industry. Inevitably, both variables must be addressed: 512 bytes is simply too small for modern applications, and the 32-bit LBA limit will inevitably become a constraint regardless. The industry must push both forward sooner or later, and I am glad to see it happening even if only in pieces.

The PC makers’ stubborn support of BIOS and MBR is much more of a concern. Switching to a modern system firmware like EFI would only cost a few pennies, and modern operating systems are ready for the change. Indeed, a few PCs come with EFI already, and Apple adopted it wholesale years ago. Similarly, GPT is ready to take over for MBR if only the PC software makers would stand and support the change.

MBR and BIOS are the real enemies here, whether we are discussing hard disk drive formats or filesystem limitations. Backwards compatibility is nice, but the Wintel PC world will be forced to make the change eventually. Since external hard disk drives work equally well regardless of partition scheme and most PC users will never upgrade their hardware, maintaining BIOS and MBR in PCs is really a question of penny-pinching anyway. It’s time for Microsoft to push PC vendors to make the change.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Hitachi GST Joins WD On The “Advanced Format” Hard Disk Drive Bandwagon
This post was categorized as Apple, Enterprise storage, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

How Big?

Hard disk drive capacity limits are familiar to PC enthusiasts. I remember worrying about the 504 MB barrier but pushing through with a BIOS update. The first greater than 2 GB disk I used required updates to both the BIOS and filesystem. Then there was the 32 GB limit imposed by Windows 95. Each of these limits was caused by two factors: Underestimation of the amount of storage capacity available to future PCs and technical “bitness” limitations.

The evolution of PC hard disk drive interfaces is well documented, and I commented on it in some depth when covering Western Digital’s 4K “Advanced Format” drives. In that piece, I mentioned that most PCs today use the “future-proof” 48-bit LBA addressing method, allowing a maximum of 128 PB of capacity. This wasn’t the whole story, however. Many operating systems still cannot use the full 48-bit address space, and we must also consider partitioning schemes and filesystems.

See also Drobo, XP Users: Beware of 4K “Advanced Format” Drives! and Drobo Adding 4K Drive Support – What About Everyone Else?

When 48 Bits is 32 Bits

LBA now allows 48 bits of block address space, and this should allow modern PCs to address a full 128 PB of capacity. This number is easy to calculate: 48 bits of address space allows us to count to 281,474,976,710,656 in (unsigned) integers. Multiply this by 512 bytes and we get 140,737,488,355,328 KB of storage capacity. Funny base-10 to base-2 disk math translates this to 128 PB of capacity.

But using 48 bits of address space requires a CPU and operating system that can work with 48 bit numbers, and many computer systems remain limited to 32 bits thanks to hardware, software, or both. These systems shrink the address space to 4,294,967,296 times 512 or 2,147,483,648 KB of capacity. This works out to 2048 GB or 2.15 TB in base-10 disk capacity speak.

The 2.1 TB Ceiling

This 2.1 TB number became institutionalized in both software and hardware over the last decade. The old-standby MBR partition method was never extended beyond this number, with manufacturers hoping for a wholesale transition to GUID partition table (GPT) format. GPT was bundled with Intel’s EFI as a replacement for legacy PC BIOS, but few PC manufacturers adopted it. Indeed, Apple is the only company to have standardized on EFI/GPT for all of their home computer systems.

Since so few PCs could ever surpass this capacity limit, many peripherals were designed with it in mind. Hard drive controllers and RAID cards may not be able to handle 3 TB drives, either. Although NTFS can expand very, very large (256 TB or more), most Windows PCs still use MBR and BIOS so they remain limited to 2 TB.

What happens if we put a 3 TB disk in a system that can only handle 32 bit LBA addresses, uses MBR, or has a limited piece of hardware? The disk might show up much smaller than expected (with addresses truncated), it might behave oddly (with truncated addresses “wrapping around”), or it might fail to work at all. Simply put, one should not use a hard disk drive larger than 2 TB in any system without checking with the various component manufacturers and operating system vendor first.

Making It Work

Western Digital attacked this problem by increasing the sector size to a more-modern 4 KB in their “Advanced Format” drives. These began appearing in December 2009 and are somewhat supported in modern computers and external disk units. Seagate does not appear to be adopting 4K and is instead pushing forward past the 32 bit address mark with their forthcoming 3 TB hard disk drives. It remains to be seen which course other drive manufacturers will choose.

Generally speaking, you should be in fairly good shape if your computer meets these tests:

1. Intel-powered Apple Macintosh computers (from 2006 onward) with OS X 10.4 “Tiger” or better should be fine
2. Non-Apple PCs require a fairly recent operating system – Windows Server 2003 or 2008, Windows Vista, Windows 7 or better
3. Non-Apple PCs must also have one of the following:
   1. A regular MBR boot disk and the big new drive mounted as a non-boot drive with GUID Partition Table (GPT)
   2. EFI or UEFI firmware rather than legacy BIOS (e.g. some HP, IBM, and MSI systems)
4. Most recent OSes will support these large disks in non-booting external drive units connected with USB 2.0, USB 3.0, FireWire, or eSATA

In other words, current PCs will never boot from these new giant drives, but most other uses should be fine as long as a modern OS is used.

We should also consider external storage systems, from the smaller Drobo, Iomega, or Synology units to the largest enterprise storage arrays from companies like EMC, HDS, HP, NetApp, and IBM. We expect that many of these systems already support the full 48-bit LBA address space, but manufacturers will certainly want to qualify these new drives before officially blessing them. We recommend holding off on trying these large drives in external storage systems until the manufacturers have officially added them to their compatibility lists.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Taming Monster Disk Drives: 3 TB and Beyond!
This post was categorized as Computer History, Enterprise storage, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Snow Leopard kernel refuses to boot in 64-bit mode on most Macs. Should you care?

There is clearly a ton of interest in Mac OS X 10.6, Snow Leopard: My Snow Leopard features hardware compatibility chart had over 20,000 visitors in just three days! But one chart element is generating an inordinate amount of interest: Whether or not Snow Leopard can boot in 64-bit mode.

Snow Leopard is a solid but visually unremarkable update to Apple’s OS X family. Although there are many new technical features, it’s not loaded with flash. In fact, even experienced Mac users would be hard-pressed to notice that their system had even been updated!

As I mentioned in that post, I was most-interested in the potential to use Grand Central Dispatch and OpenCL to accelerate my system’s performance. Both features are supported on most Macs built over the last year or two, and they create a great foundation for future development. QuickTime X is interesting as well, though only Nvidia 9400M-based Macs (like my 2009 Mini) support it’s hardware-accelerated H.264 capabilities. The fact that so much of the operating system has been re-written using these modern capabilities is probably the biggest Snow Leopard feature.

Snow Leopard Does Not Boot 64-Bit…

I wasn’t all that concerned about whether the OS X kernel booted in 32-bit or 64-bit mode, but was intrigued to find that Snow Leopard boots with a 32-bit kernel on every Mac other than the rack-mount Xserve.

Much pre-launch discussion revolved around whether or not a given system was even capable of booting in 64-bit mode. There are three critical elements that determine whether a given machine will boot a 64-bit Snow Leopard kernel:

1. All system hardware must have 64-bit drivers, including the CPU, chipset, and peripheral chips. The 64-bit kernel exclusively uses 64-bit kexts (drivers), and these are rarer than the 32-bit ones used by Leopard.
2. It must use 64-bit EFI (firmware). Even though EFI32 can technically boot a 64-bit operating system just fine, Apple will not allow this. Only Macs made over the last few years include EFI64.
3. Apple must allow the system to boot in 64-bit mode. This is the most confusing. Many Macs will not boot Snow Leopard in 64-bit mode even though they are brand new, have no unusual hardware, and include EFI64. This includes all non-Pro MacBooks and Mac Minis.

So there you have it. No matter what you do, Snow Leopard will not boot with a 64-bit kernel and kexts on the majority of Macs. One could understand if there was a lack of drivers or some other technical limitation, as is the case with older Macs, the but this is not the case for the latest MacBook and Mac Mini systems. They won’t boot with a 64-bit kernel because the kernel refuses to boot.

The only Mac model that boots with a 64-bit kernel by default is the 2008 and 2009 Xserve (Xserve2,1 and Xserve3,1) running Snow Leopard Server. The following Macs can boot in 64-bit mode in non-server Snow Leopard if you hold down the 6 and 4 keys on boot or if you modify your “/Library/Preferences/SystemConfiguration/com.apple.Boot.plist” file as described at Netkas:

If your Mac is not included in this list, you will not be able to boot Snow Leopard in 64-bit mode. MacBook and Mac Mini models definitely will not boot into 64-bit Snow Leopard. The later Unibody MacBook Pros (5,3 and 5,4) are not listed, and it is not known if they work or not.

…But Why Do You Care?

Why do enthusiasts care so much about the “bitness” of their hardware and software? Geeks have been arguing over the merits of “bigger-bits” for decades. I remember Mac users claiming PCs were inferior because Macs used true 16-bit Motorola CPUs while PCs were stuck with an 8-bit bus. The gaming world did much to foster 64-bit mania, with Atari claiming their Jaguar was the first 64-bit console, and Nintendo even naming theirs the “N64″. But does it really matter?

In theory, a 64-bit operating system should make most-effective use of 64-bit CPUs, increasing performance and maximum addressable RAM. In the case of Intel CPUs, there are many improvements when running the 64-bit (x64) instruction set instead of legacy 32-bit (IA-32) mode. So, from a hardware perspective, 64 bits is definitely superior.

But Mac OS X can run 64-bit applications regardless of the bitness of the kernel. I am writing this post in 64-bit Safari running on a 32-bit Snow Leopard kernel. Many of the applications I normally run are 64-bit in Snow Leopard, including Safari, Mail, and the Finder. This last is important: All Snow Leopard users get a speedy 64-bit ground-up rewrite of Finder, which is their main operating system experience.

32-bit Snow Leopard runs 64-bit versions of many applications, including Safari, QuickTime X, iCal, Mail, and the Finder itself. Notable 32-bit apps include Microsoft Office, iTunes, and iPhoto.

Therefore, regardless of which kernel mode you have booted into, many of your daily activities will use full 64-bit applications. 64-bit versions of iTunes, iPhoto, and Microsoft Office might be nice, but they’re the exception not the rule, and I expect to see these move up to 64-bit soon.

Why do so many Macs not use the 64-bit version of Snow Leopard? One reason is compatibility: The 64-bit kernel requires a full set of 64-bit drivers, and these may have been lacking at launch. Another reason might be that these systems would not see much benefit from running a 64-bit operating system since they are limited to 4 GB of RAM. I would not be at all surprised if later point-revisions of OS X 10.6 enable booting in 64-bit mode by default, at least for the machines listed above and newer models. Those of us with older machines (like this MacBookPro3,1) might never see it, though.

I should note that there are a few 64-bit features lacking in 32-bit Snow Leopard. But some applications, including VMware Fusion, will not run at all under a 64-bit Snow Leopard kernel. And there is the question of driver compatibility. I am not an Apple apologist or fanboy, but I do use Macs 100% of the time for business and pleasure. I think it stinks that Apple now lags behind Microsoft in the race to provide a 64-bit desktop operating system. But the decision is understandable and it has little impact on the daily lives of end users.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | No 64-Bit Snow Leopard Kernel For You!
This post was categorized as Apple, Computer History. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Mac OS X 10.6 "Snow Leopard" deliveries begin on August 28!

Although Apple’s web site clearly states that Snow Leopard “Delivers on August 28th,” it is unclear if other sources will be so punctual. We definitely expect Apple Stores to have plenty of stock on Friday morning, but what of Amazon.com pre-orders? Will they also deliver on Friday, or will those of us who pre-ordered there have to wait a few days for this undercover upgrade?

Who Gets What

Snow Leopard is really a major jump forward in terms of advanced hardware and software integration. It brings Grand Central, OpenCL, full 64-bit mode, and QuickTime X along with Exchange support and many tweaks. But many of these features are highly hardware-dependent, so not all Mac users will get everything.

Here’s a cheat sheet I put together based on Apple’s documentation:

This table is based on the following facts:

1. Snow Leopard only supports Intel-based Macs. PowerPC Macs need not apply.
2. Booting Snow Leopard in 64-bit mode requires 64-bit EFI, and many pre-2007 Macs have only a 32-bit EFI. You can check your Mac (and help me fix my table) by typing “ioreg -l -p IODeviceTree | grep firmware-abi” in a Terminal window. If it says “EFI64″, your system is capable of running 64-bit Snow Leopard (though it can still run 64-bit apps). But Snow Leopard defaults to 32-bit mode on ALL Macs other than the Xserve at this point, and no one is sure why!
3. Grand Central Dispatch requires a multi-core CPU. This eliminates the original base-model 2006 Mac Mini, since it used a single-core CPU.
4. OpenCL requires newer Nvidia or the ATI graphics chips. This leaves out many pre-2008 models and some iMac and Mac Pro configurations. Check Apple’s list if you’re not sure.
5. QuickTime H.264 Hardware Acceleration only works with the Nvidia 9400M chipset. This leaves out many pre-2008 machines and even the latest Mac Pros, though they probably have enough horsepower on their own.

Finally, if you’re excited about the new Microsoft Exchange support in Mail.app, iCal, and the Address Book be warned! In order for any of this to work, your server must be running Microsoft Exchange Server 2007 Service Pack 1 Update Rollup 4. And most companies still aren’t updated that far!

Update: Check my follow-up post on 64-bit Snow Leopard for more about this controversial aspect!

I’m Updating!

I held off on Amazon and placed my own order this morning with Apple. I’m with Mark Twomey:Upgrading to Snow Leopard is a safe day-1 activity. Most of the updates amount to new under-the-hood features and the OS has been through round after round of testing. Unlike the massive shift from Tiger to Leopard (which I missed, being a late switcher), Snow Leopard ought to be a lesser upgrade.

Why make the update at all? Here are a few of my reasons:

1. It’s cheap: Just $29 to update a single Mac, or $49 for up to five. Compared to Microsoft’s (expired) “limited-time-only” (and incredibly complicated) Windows 7 upgrade, Apple sets out a red velvet carpet.
2. It’s easy: There’s just one version of Mac OS X, and any system running OS X 10.5 “Leopard” can update to full-on Snow Leopard. There are no editions and no conflicts updating from 32-bit to 64-bit (like Windows 7).
3. It future-proofs your (Intel) Mac: Mac developers have a long history of quickly leveraging new OS X features since Mac users have a long history of quickly upgrading. Snow Leopard adds cool stuff like Grand Central and OpenCL that my Intel- and Nvidia-based Macs ought to be able to leverage. See below if your Mac can use these, too!

Sure, there aren’t any amazing features like Time Machine to set the world on fire. But the Snow Leopard update is still a slam-dunk for any Intel-based Mac user! My Late-2007 MacBook Pro will get everything but H.264 acceleration and my 2009 Mac Mini is all set to go!

Updated with Mac Mini and Xserve information. Keep the suggestions coming and test that EFI! Picked up, colorized, and reused (with permission) by the excellent Ed Bott in his ZDNet column!

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Mac OS X 10.6 “Snow Leopard”: In Our Hands August 28!
This post was categorized as Apple, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.