VAAI in 4.1: Three For Block (Plus One More)

vSphere 4.1 officially includes three VAAI primitives: Full Copy, Hardware Assisted Locking, and Block Zeroing. But it was intended to include a fourth (Thin Provisioning Stun) and variations in nomenclature have confused administrators.

Here’s the complete list of VAAI primitives in vSphere 4.1, including a little more about what they really do.

Atomic Test & Set (ATS) (AKA Hardware Assisted Locking or HardwareAcceleratedLocking)

This is one of the most valuable storage technologies to come out of VMware, but it’s usually overlooked. It allows compare and write of SCSI blocks in one operation using proprietary array-specific op-codes. This enables granular locking of block storage devices beyond the basic full-LUN “reservations” included in SCSI from days of yore. This is huge for performance when LUNs are shared, with HP reporting at VMworld allowing 6 times more VMs per LUN once VAAI is turned on. NFS doesn’t really need this since locking is a non-issue and VM files aren’t shared the same way LUNs are.

To verify from the CLI:

esxcfg-advcfg -g /VMFS3/HardwareAcceleratedLocking

Cloning Blocks (AKA Full Copy, Extended Copy, or HardwareAcceleratedMove)

This is the signature VAAI command. Instead of reading data from the array then writing it back, the hypervisor can command the array to make a mirror of a range of data on its behalf. If supported and enabled, operations like Clone and VMotion can be lightning quick, since many high-end arrays are very capable of this sort of action. This leverages the SCSI XCOPY command (or other proprietary alternatives enabled with a VMware plug-in) and defaults to a 4 MB block size.

To verify from the CLI:

esxcfg-advcfg -g /DataMover/HardwareAcceleratedMove

Zeroing File Blocks (AKA Block Zeroing or HardwareAcceleratedInit)

Thin provisioning is difficult to get right because storage arrays have no idea what’s going on in the hosts. VAAI includes a generic interface for communication of free space, allowing vast ranges of blocks to be zeroed out at once. Along with the generic T10 “WRITE_SAME” command, VAAI allows vendors to use “UNMAP” or their own mechanisms with a plug-in. VMware defaults to a 1 MB block size for zeroing operations. But zeroing only works for supported block arrays, and only for capacity inside a VMDK: Clones and vMotions leave the array guessing.

To verify from the CLI:

esxcfg-advcfg -g /DataMover/HardwareAcceleratedInit

Out of Space Condition (AKA Thin Provisioning Stun)

Running out of capacity is a catastrophe, but it’s easy to ignore the alerts in vCenter until it’s too late. This command allows the array to notify vCenter to “stun” (suspend) all virtual machines on a LUN that is running out of space due to thin provisioning over-commit. This is the “secret” fourth primitive that wasn’t officially acknowledged until vSphere 5 but apparently existed before. In vSphere 5, this works for both block and NFS storage. Signals are sent using SCSI “Check Condition” operations.

VAAI Commands in vSphere 4.1

esxcli corestorage device list

esxcli vaai device list

esxcli corestorage plugin list

VAAI in 5.0: Lots More

vSphere 5 includes a new generation of VAAI primitives, including support for NFS storage. Once again, VMware includes a “secret” primitive not yet implemented, and puzzlingly lumps some of the official entries together in company documentation. But VMworld changed that, with one presentation including insight into the real list of VAAI primitives.

vSphere 5 includes and enhances all the VAAI primitives from vSphere 4.1 (including “Out of Space Condition”) as well as the following new capabilities.

vSphere 5 includes the above primitives, plus these

UNMAP (AKA Space Reclamation or EnableBlockDelete)

Zeroing File Blocks is great, but VMFS operations like cloning and vMotion didn’t include any hints to the array. This means that some of the biggest storage operations aren’t accelerated or thinned out! UNMAP is a SCSI command, and it was added to VAAI to allow thin-capable arrays to clear unused VMFS space as well. But vCenter couldn’t handle waiting for the array to return “UNMAP” command status, so the use of this primitive is officially suspended by VMware until this issue is resolved.

To verify from the CLI:

esxcfg-advcfg -g /VMFS3/EnableBlockDelete

For now, disable it:

esxcfg-advcfg –s 0 /VMFS3/EnableBlockDelete

Quota Exceeded Behavior (AKA TP Soft Threshold)

It would be nice if vSphere could react before an out-of-space condition is reached. This primitive allows the array to notify the hypervisor preemptively, possibly triggering a Storage DRS re-balance, for example. It also allows the hypervisor to query the array for its threshold values using SCSI “mode sense”.

TP LUN Reporting (AKA Report Thin Capacity)

A very simple command, this allows the hypervisor to query the array to determine if a LUN is thinly provisioned. On block arrays, the SCSI “TPE” bit is used, but this also works with NFS.

NFS Full File Clone (AKA Hardware Snap)

NFS already did a nice job with storage provisioning and presentation, but VMware couldn’t make use of hardware snapshot capability. This changes in vSphere 5, with a “Full Copy”-like mechanism appearing for NAS. But the NFS version doesn’t support Storage vMotion operations like its block cousin.

NFS Space Reservation (AKA Zeroedthick Files)

NFS is always thin, which is a good and bad thing. This new primitive adds two new file type for NFS-based VMDK files: “Eager” and “Lazy” Zeroed Thick. These are similar to the old default “thick” volume in block storage, but thick provisioning and space reservation is brand new territory for NFS admins. This primitive uses the NFS “reserve_space” command.

NFS Extended Stat (AKA Extended Status)

One-upping the SCSI block world, vSphere 5 can query NAS servers for all sorts of information, including file size. This should make NFS in vCenter even more friendly!

NFS Space Reclaim (AKA Space Reclamation)

Although the block-only UNMAP feature doesn’t work, the NFS equivalent Space Reclaim seems fine. This enables the hypervisor to unlink old files once they’re no longer used. This primitive uses the NFS “unlink” command.

NFS Fast File Clone (AKA Linked Clone Offload)

This “secret” VAAI primitive allows a future release of VMware View (and only View) to offload creation of linked clones to a NAS array. Hopefully we’ll see more of this in the future!

VAAI Commands in vSphere 5

esxcli storage core device list

esxcli storage core device vaai status get

esxcli storage core plugin list

© sfoskett for Stephen Foskett, Pack Rat, 2011. | A Complete List of VMware VAAI Primitives
This post was categorized as Enterprise storage, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Why should my data be left unprotected for more than two days? Devices should allow us to "prime the pump" for upgrades and migrations.

Upgrades are an inevitability in our modern technological world. A new phone comes out every year or two, and the migration process begins. So why don’t devices have a special mode, priming them for upgrade and migration? This really hit home recently, as I upgraded the hard disks in my Drobo, but it applies equally to laptops, phones, and services like e-mail.

So It Goes

Most well-designed devices are remarkably adept at recovering from errors, and some use this capability as an upgrade mechanism. Pop one drive out of the Drobo and replace it with another and the system will happily rebuild. But this process takes hours or even days, and data is left unprotected all this time.

Computer operating systems are also fairly capable of adapting to change. When I swapped the mechanical hard drive out of my MacBook Pro in favor of an SSD, I copied the data over, switched the drives, and powered on with little obvious impact on my work environment. But Microsoft noticed the change, and I had to reenter my Office 2011 serial number. Things were a lot more challenging when I moved from my old MacBook Pro to the new 2011 model. Storing data in the cloud made things much easier, but migrating between laptops remains problematic.

Thankfully, my last three phone upgrades have all been in the iPhone family. This has allowed me to leverage iTunes to move my applications and data without interruption. Switching from iPhone to Android is much more difficult, even with e-mail and contacts stored in Google’s Gmail cloud.

In-Between Days

It seems to me that the key to a successful migration is minimizing the time in between the old state and the new. Consider the Drobo: thanks to the devices beyond raid data protection technology, I don’t really worry about the integrity and protection of my data. But swapping out hard disk drives leaves my data in an unprotected state for 48 hours or more. This critical time is nerve-racking.

The same often happens when one migrates from one computer or phone to another. It is critical to immediately give up the old and adopt the new, but moving data and applications can take hours or days. I recently used Microsoft’s Easy Transfer utility to upgrade a friend’s computer from Windows Vista to Windows 7. But they never really got used to Windows 7, leaving the new computer unused on the shelf.

What is one supposed to do during the time in between devices? We may not be able to actively use the new device, but we cannot continue making changes to the old. Should we simply set things aside and take a vacation? And what if the migration goes awry or the user decides they can’t make the transition?

Prime the Pump

Why don’t devices have the ability preemptively to “prime the pump” for migration? Why can’t they prepare data and applications, and perhaps even the user environment, for a smooth transition? The obvious answer is that an old system does not “know” when a new system requires. But there are certain cases where this could be done.

Two years ago, I suggested to Drobo (then called data robotics) that they should have a pre-eject button in their management application, allowing the user to notify the array that drive was about to be ejected. The Drobo could then move data around and prepare for this major event without leaving everything unprotected for days. This feature still has not been implemented, however. And you can only upgrade one drive at a time, meaning my data will be unprotected for over a week while switching to four larger drives.

What if Windows Easy Transfer began packing up user data while the old system was still in use, perhaps synchronizing it to the new computer while changes were being made. It could even keep both devices in sync for a while, allowing the user to adjust to the new interface. Certainly, this would improve the user experience, and Microsoft may be able to entice more users to upgrade.

And wouldn’t it be nice if our favorite phone application developers offer the ability to trade in iPhone apps for Android equivalents, or vice versa? Apple hasn’t even allowed this in the Mac App Store, and the incentives are questionable regardless. But this is what the average user wants, so why can’t it happen?

Stephen’s Stance

Although it’s not really practical for all devices, manufacturers should consider that migration is a critical time of user frustration and the loss of future revenues. If they spent some time reducing the impact of migration, they might build a repeat customer base. And the ability to prime the pump for migration preemptively is one concept that might help make this happen.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Why Can’t We Prime Our Devices For Upgrades?
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

If WRITE_SAME can be a semaphore for thin un-provisioning, what about TRIM? It sounds like a perfect fit, and has wider implementation to boot! Let’s take a deeper look.

First, I’d like to remind the reader that I’m not some specialist storage engineer. I am regular guy who wanted to know about this TRIM thing. I welcome corrections and feedback on this piece!

We should note that TRIM was invented for SSDs, not thin provisioning. TRIM lets an operating system tell a storage device that certain blocks of data are no longer used. Although this sounds like a thin provisioning tool, it’s not. TRIM was never intended to be used this way.

Even though Microsoft supports TRIM in the file system, and even though, hopefully, Apple will introduce TRIM support in OS X Lion, this doesn’t do what we need it to do for thin provisioning. In fact, TRIM is not anything for enterprise storage folks to be getting all that excited about.

One reason for is is that TRIM is an ATA command, and most enterprise systems are not connected to their storage using any kind of ATA-based protocol. Maybe there are SATA disks somewhere down the line, but, whether they use iSCSI, FC, SAS, or FCoE, the servers speak SCSI.

Even though there are SCSI analogs of of TRIM, these commands were really not designed for thin provisioning. They are designed for SSDs, which have a big performance problem: An SSD can delete data and write data just fine, but in order to update data, they actually have to read, delete, and write a big page of data. What happen is that, as an SSD gets full, it starts slowing down. And that’s a problem because people buy SSDs for speed. So TRIM is a way to tell the SSD, “Sometime in the future, you can go ahead and delete this if you feel like it,” and the SSD will take care of that later.

There’s great OS support for this, but unfortunately, this is not something that enterprise storage vendors are looking at. The only vendor (that I know of) that’s working on this, NetApp, did propose to T10 a mechanism to do thin provisioning based on TRIM. I don’t think it got anywhere. They say that it did, but I just don’t see it. HDS and EMC, seem to like UNMAP and SCSI, which are peripherally related, but I don’t know much about what they’re doing either. Perhaps they’ll let me know in comments on this post.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Is TRIM Useful For Thin Provisioning?
This post was categorized as Apple, Enterprise storage, Everything, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

2011 will look pretty much like 2010 apart from the differences...

Prognostication is a perilous business, but pundits are drawn to the topic in the month of December. The fact that most predictions fall on their faces demonstrates the intoxicating mix of hope, dreams, and irrationality that mark both geniuses and fools. I am neither, so I like to make predictions after the fact! But this year I’ve been asked to look to the future, so I’ll stick with the safe road and pick current trends rather than guessing what I hope will come.

Five Trends For Everyone

1 – Ubiquitous Connectivity

I named mobility as the mega-trend of the last decade, noting that it’s hard to spot a trend from the middle and harder still from the start. But I feel vindicated on that 2009 call, and will take it one further: 2011 will see ubiquitous connectivity become mainstream. With “MiFi” entering the vernacular, a proliferation of “4G” networks and integrated wireless data, and free WiFi having already become passe, I’d say “online everywhere” is here. Although incredibly challenging from both a technical and business perspective, I expect everyone and everything to be online-capable.

2 – The iPad and Foes

Steve Jobs’ bizarre claim that the overgrown iPhone tablet was the most important thing he ever did is starting to look prescient. A new generation is coming of age without windowing GUIs, mice, and keyboards thanks to gaming consoles, smartphones, iPods, and (finally) tablets. 2011 will finally see serious Android, WebOS, and Windows tablets, but the iPad (and forthcoming iPad 2) are the platform to beat, and Apple is firing on all cylinders. This war will be all-consuming next year.

3 – Wave “Hi” to Kinect!

Microsoft has a serious hit on their hands with the Kinect add-on to the Xbox 360. The gaming system is the best thing to come out of Redmond in a long time, and it continues the “no controller” concept of the iPad, finally kicking the Wii to the curb. The massive success of the Xbox will lead Microsoft shareholders to abandon their calls for Ballmer’s head, instead asking for a spin-out or IPO of the gaming division. Expect PCs to include Kinect-like features in the coming years as well.

4 – Facebook Is the Internet

Now boasting a quarter of all web pageviews. Facebook is looking increasingly like AOL for the rest of us. 2011 will see Facebook’s gravity pull in content from everywhere, and its satellites sprout all over the Internet. It will become the single sign-on, the central “like”, the address book, and the meeting place. But fear not, Facebook-phobes: Nothing is permanent, and this too shall pass.

5 – The Internet Changes and No One Notices

The IP address space is exhausted, BGP and DNS security lapses rise, and net neutrality falls by the wayside but Farmville still works so no one cares. The Internet is changing, and controversies over key components are coming coming to a head. I imagine the network engineers will be busy keeping ahead of catastrophe, but they’ll manage somehow. I’m not sure if IPv6 will finally take off or if the carrier gambit will succeed, but I’m confident we’ll still have an Internet at the heart of the technology world!

Five Trends For the Datacenter

6 – Clouds Gather Quietly

“Cloud” was the buzzword of the last two years, but now it’s getting down to work. Traditional IT staff still won’t see much of it outside of blogs and conferences, but non-traditional systems are all heading that way. Look for major uptake of cloud platforms and services from the home to enterprise applications and everywhere in between. Ironically, the “c-word” itself will soon be dropped from these successful services just as it gains acceptance in IT shops.

7 – Virtual Everything

The impact of server virtualization hasn’t been as great as supporters claim, but widespread acceptance of hypervisor-centric data centers is here. There’s really no reason not to deploy every datacenter server as a virtual machine and lots of resulting benefits. Expect to see mission-critical apps finally move to VMware ESX and Microsoft Hyper-V in 2011. And expect to see the resulting infrastructure called “cloud”!

8 – Farewell, Fast Hard Drives

Storage I/O performance is now the realm of solid state, not spinning disk. SSDs have reached the level of performance, capacity, availability, and sophistication that we no longer need 15k rpm enterprise hard disk drives. You’ll use SSD if you want IOPS, but you’ll still need spinning platters for capacity and maximum throughput for a long while. Don’t expect hard disk drives to disappear, but the fastest will exit at the end of the year.

9 – Not-So-Converged I/O (Yet)

DCB ain’t ready, folks. Neither is FCoE. Although Ethernet will eventually sideline InfiniBand and Fibre Channel, that’s not a 2011 topic. I expect to hear a lot of noise about converged network and storage I/O, including high-profile customer adoption stories, but we’re still a few years short of actual impact and serious market share movement. Practical application starts in 2011, though, and it’ll get major coverage and big-money action in the vendor space.

10 – RAID is (Finally) Dead!

There won’t be much ink spilled in memoriam outside storage blogs like this one, but conventional mirroring and parity has finally met its maker. Today’s hard disk drives are too big to rebuild singly, and alternatives like wide striping, erasure coding, and touch-me-not disk packs are taking over.

And One More For Me

11 – The Internet is Shiva

Every business will be permanently changed as Internet-enabled platforms destroy profitable monopolies and build new opportunities. Google conquered advertising and destroyed traditional publishing but enabled a flowering of democratic dialog. PayPal and Square will do the same to banking in 2011, but their own come-uppance might come sooner than they like. No matter your business, someone has their sights set on you and the Internet is their tool. The trick is to keep dancing, keep innovating, and love the paradox.
The text to appear after expiration date.
Image Credit: Eleven by Wetsun

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Eleven Tech Trends To Watch In 2011
This post was categorized as Apple, Computer History, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Introducing NTFS

A filesystem is the basic scheme for organizing data on a hard disk drive. The common FAT filesystem, supported by just about every operating system, is just too basic for modern operating systems. Microsoft introduced the advanced NTFS filesystem with their Windows NT operating system in 1993. Borrowing from OS/2′s HPFS and VMS’ Files-11, NTFS is a journalling filesystem with many features, including POSIX links, alternate data streams, sparse files, compression, and encryption.

Although just about everything Microsoft produces is maligned by UNIX and Mac geeks, this hatred is unwarranted in the case of the many storage advances coming out of Redmond. Microsoft may be fairly criticized for producing expensive, proprietary systems, but they have dome some amazing things with disks, and NTFS is arguably one of the most advanced filesystems in wide usage. Much of this credit goes to Windows NT architect Dave Cutler, who also led development of DEC VMS, the RISC concepts that became Alpha, and Microsoft’s Azure cloud platform.

Although every version of Windows supports evolutions of the old FAT filesystem, modern iterations increasingly require NTFS. Windows XP allowed either FAT32 or NTFS for booting; Windows Vista could be forced to boot from FAT32; Windows 7 requires NTFS. Therefore, most PC hard disk drives, including external USB drives, now come formatted with NTFS by default.

NTFS on Mac OS X

This leads to issues for Mac users, especially when they dual-boot with Boot Camp or buy external hard disk drives. As mentioned, all versions of Mac OS X since 10.3 “Panther” have included read-only NTFS support, but those wanting full access had two options:

1. Paragon Software’s NTFS for Mac OS X is a full and supported read/write solution. Although available for purchase separately, many hard disk drives like the Seagate GoFlex now include a free copy of Paragon NTFS.
2. The open source NTFS-3G driver has also been ported to Mac OS X, allowing read/write support for free.

Mac OS X 10.6 “Snow Leopard” adds a third alternative: Native read/write support for NTFS. But it’s not as simple as that. Although early betas enabled read/write support by default, it was limited once again to read-only in the final releases.

It is possible and even simple) to enable read/write support with the native Snow Leopard drivers, but this is definitely an “at your own risk” proposition. Users have reported kernel panics when using these methods, and although I have not heard of data corruption, it is possible as well.

The Manual Method: mount_ntfs

Making disks available is called “mounting” in UNIX geek-speak, and mount commands in modern operating systems are like ogres: They have many layers. Plug in a drive, and Mac OS X makes it available for you using a layer cake of commands, but the bottom-most command is “mount”. This simple command-line utility takes a few arguments and attaches a raw disk device to a point in the unified filesystem tree.

When presented an NTFS drive, Mac OS X calls a utility called mount_ntfs that resides in the /sbin directory, and we can call it, too. Although OS X always tells mount_ntfs to mount NTFS drives read-only, we can tell it to enable writing as well by using the “-o rw” flag.

There’s just one problem: Where should we connect the NTFS file system? Mac OS X automatically creates a new directory (called a “mount point” in UNIX-speak) in /Volumes with the name of the filesystem. For example, the GoFlex drive I bought yesterday mounts as “/Volumes/FreeAgent GoFlex Drive” by default. But mount_ntfs won’t create this mount point, so we have to do it.

Here are the steps to mount an NTFS drive read/write from the command line:

1. Plug in the drive and let Mac OS X mount it read-only.
2. Look in Finder to see the name of the drive. For example, my drive mounts with the name “FreeAgent GoFlex Drive” and this is what shows up on the desktop and in Frinder under “Devices”.
3. Open the Terminal application to get command-line access. We’ll use “sudo” which executes commands as the administrator instead of a regular user.
4. Type the following command in Terminal to get necessary information about the drive. Use the drive name you see in Finder in place of “drive name”, and make sure to use quotation marks around it since Terminal treats un-quoted spaces as separators rather than part of the name.

   diskutil info "/Volumes/drive name"

5. Record the “Device Node” entry, which will look something like “/dev/disk4s1″. This is where the raw partition is mapped by the disk subsystem, and where the filesystem driver will read its data.


The diskutil command gives us the "Node Name", which we'll need momentarily

6. Eject the drive, using option-click in Finder or with the hdiutil command in Terminal:

   hdiutil eject "/Volumes/drive name"

7. Now you must create the mount point again. Type the following:

   sudo mkdir "/Volumes/drive name"

8. Finally, use mount_ntfs to mount the drive read/write using the following command. Use the node name you spotted above and the mount point you just created:

   sudo mount_ntfs -o rw /dev/disk?s? "/Volumes/drive name"

9. When you’re done, you can use the Finder, the hdiutil command, or just about anything else to eject the drive. Make sure your mount point is deleted as well!

This method is great for occasional NTFS use, which is frankly all Mac users should need. If you intend to use a drive frequently, I highly recommend reformatting it with the native HFS+ filesystem since it will be fully-supported by the operating system.

The Per-Drive Method

Let’s say you have an NTFS drive you frequently need to mount read/write and find the above method too much of a hassle. Maybe it’s your Boot Camp drive, or perhaps it’s a USB hard disk drive you share with a Windows-using colleague. It is possible to tell Mac OS X to always mount a given NTFS drive read/write using its native driver using the following commands. However, I recommend investing in the Paragon driver for this use case, since it’s fully-supported!

Like all UNIX systems, Mac OS X uses a configuration file to determine what it should do with certain drives. This file, called “fstab” and located in “/etc”, is not present by default but will override the “read-only” flag on a per-drive basis if found. This allows a drive to always be mounted read/write without any hassle.

1. Get the drive name as in step 2 above.
2. Open Terminal and type the following command to edit the /etc/fstab file:

   sudo nano /etc/fstab



Adding a line in /etc/fstab will make Mac OS X always mount a single drive in read/write mode

3. “Nano” is a command-line editing program. Add the following line to the file (and don’t forget the quotes around the drive name if it contains spaces!)

   LABEL="drive name" none ntfs rw

4. Now type control-x to save and exit, type y for yes, and hit enter.
5. After a reboot, Mac OS X should automatically mount any drive with that label as read/write

Note that this can also be done using the NTFS UUID of the drive, but this isn’t present on every drive to LABEL works better. Also, some have reported needing multiple reboots to make it work. I have no idea why.

The Permanent Universal Method

It is also possible to permanently modify how Mac OS X calls the mount_ntfs command to always mount every NTFS drive in read/write mode. Again, I will point out that this isn’t necessarily a great idea since Apple doesn’t support using the driver in this way. But here it is!

1. Open Terminal. We won’t need the drive name or Node Name since this applies to every NTFS drive.
2. Rename the mount_ntfs command to something else by typing the following:

   sudo mv /sbin/mount_ntfs /sbin/mount_ntfs_orig

3. Create a shell script called mount_ntfs using the Nano editor:

   sudo nano /sbin/mount_ntfs

4. This script will simply call mount_ntfs with the “-o rw” flag every time it’s used. Type the following in Nano:

   #!/bin/sh
   /sbin/mount_ntfs_orig -o rw "$@"

5. Type control-x to save and exit, type y for yes, and hit enter.
6. Now we’ll fix the permissions on this new file so it’s owned by root and group wheel:

   sudo chown root:wheel /sbin/mount_ntfs

7. Finally, we make the shell script executable:

   sudo chmod 755 /sbin/mount_ntfs

8. Now every time OS X encounters an NTFS drive, it will mount it read/write by calling this script rather than the “mount_ntfs” executable it thinks it’s using.

Note that future OS updates might “break” this method by replacing the /sbin/mount_ntfs command, or it could cause other issues. But it works for now and doesn’t even require a reboot.

Stephen’s Stance

I’m hesitant to recommend that casual Mac users employ any of these methods to access NTFS drives. Each requires superuser access and careful typing. You can easily mess up your system this way, and unsupported uses of the NTFS driver might result in data loss.

If you aren’t comfortable with this kind of mucking-about or intend to frequently write to NTFS drives, I highly suggest installing NTFS-3G or Paragon’s NTFS driver. In fact, the best method is simply to go buy a Seagate GoFlex drive and use the Paragon driver that comes with it!

But hackers and geeks like me want to know how to do this. This is why I decided to document it here.

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | How To Write To Windows NTFS Drives In Mac OS X 10.6 “Snow Leopard”
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

How Big?

Hard disk drive capacity limits are familiar to PC enthusiasts. I remember worrying about the 504 MB barrier but pushing through with a BIOS update. The first greater than 2 GB disk I used required updates to both the BIOS and filesystem. Then there was the 32 GB limit imposed by Windows 95. Each of these limits was caused by two factors: Underestimation of the amount of storage capacity available to future PCs and technical “bitness” limitations.

The evolution of PC hard disk drive interfaces is well documented, and I commented on it in some depth when covering Western Digital’s 4K “Advanced Format” drives. In that piece, I mentioned that most PCs today use the “future-proof” 48-bit LBA addressing method, allowing a maximum of 128 PB of capacity. This wasn’t the whole story, however. Many operating systems still cannot use the full 48-bit address space, and we must also consider partitioning schemes and filesystems.

See also Drobo, XP Users: Beware of 4K “Advanced Format” Drives! and Drobo Adding 4K Drive Support – What About Everyone Else?

When 48 Bits is 32 Bits

LBA now allows 48 bits of block address space, and this should allow modern PCs to address a full 128 PB of capacity. This number is easy to calculate: 48 bits of address space allows us to count to 281,474,976,710,656 in (unsigned) integers. Multiply this by 512 bytes and we get 140,737,488,355,328 KB of storage capacity. Funny base-10 to base-2 disk math translates this to 128 PB of capacity.

But using 48 bits of address space requires a CPU and operating system that can work with 48 bit numbers, and many computer systems remain limited to 32 bits thanks to hardware, software, or both. These systems shrink the address space to 4,294,967,296 times 512 or 2,147,483,648 KB of capacity. This works out to 2048 GB or 2.15 TB in base-10 disk capacity speak.

The 2.1 TB Ceiling

This 2.1 TB number became institutionalized in both software and hardware over the last decade. The old-standby MBR partition method was never extended beyond this number, with manufacturers hoping for a wholesale transition to GUID partition table (GPT) format. GPT was bundled with Intel’s EFI as a replacement for legacy PC BIOS, but few PC manufacturers adopted it. Indeed, Apple is the only company to have standardized on EFI/GPT for all of their home computer systems.

Since so few PCs could ever surpass this capacity limit, many peripherals were designed with it in mind. Hard drive controllers and RAID cards may not be able to handle 3 TB drives, either. Although NTFS can expand very, very large (256 TB or more), most Windows PCs still use MBR and BIOS so they remain limited to 2 TB.

What happens if we put a 3 TB disk in a system that can only handle 32 bit LBA addresses, uses MBR, or has a limited piece of hardware? The disk might show up much smaller than expected (with addresses truncated), it might behave oddly (with truncated addresses “wrapping around”), or it might fail to work at all. Simply put, one should not use a hard disk drive larger than 2 TB in any system without checking with the various component manufacturers and operating system vendor first.

Making It Work

Western Digital attacked this problem by increasing the sector size to a more-modern 4 KB in their “Advanced Format” drives. These began appearing in December 2009 and are somewhat supported in modern computers and external disk units. Seagate does not appear to be adopting 4K and is instead pushing forward past the 32 bit address mark with their forthcoming 3 TB hard disk drives. It remains to be seen which course other drive manufacturers will choose.

Generally speaking, you should be in fairly good shape if your computer meets these tests:

1. Intel-powered Apple Macintosh computers (from 2006 onward) with OS X 10.4 “Tiger” or better should be fine
2. Non-Apple PCs require a fairly recent operating system – Windows Server 2003 or 2008, Windows Vista, Windows 7 or better
3. Non-Apple PCs must also have one of the following:
   1. A regular MBR boot disk and the big new drive mounted as a non-boot drive with GUID Partition Table (GPT)
   2. EFI or UEFI firmware rather than legacy BIOS (e.g. some HP, IBM, and MSI systems)
4. Most recent OSes will support these large disks in non-booting external drive units connected with USB 2.0, USB 3.0, FireWire, or eSATA

In other words, current PCs will never boot from these new giant drives, but most other uses should be fine as long as a modern OS is used.

We should also consider external storage systems, from the smaller Drobo, Iomega, or Synology units to the largest enterprise storage arrays from companies like EMC, HDS, HP, NetApp, and IBM. We expect that many of these systems already support the full 48-bit LBA address space, but manufacturers will certainly want to qualify these new drives before officially blessing them. We recommend holding off on trying these large drives in external storage systems until the manufacturers have officially added them to their compatibility lists.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Taming Monster Disk Drives: 3 TB and Beyond!
This post was categorized as Computer History, Enterprise storage, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Spot the Problem

Let’s play a little game, shall we? Let’s see if you can spot the problem!

Windows Vista reports robocopy.exe as version 5.1.10.1027, but the exe is just 85.5 KB

Windows 7 also reports robocopy.exe as version 5.1.10.1027, but the exe is 95.0 KB

Now let’s run “robocopy /mt” so see if multi-threading is supported!

The Windows 7 version of Robocopy supports the "/MT" parameter

The Windows Vista version of Robocopy does not support the "/MT" parameter

There you have it. Two executables with the same version number but substantial differences in functionality. Thanks, Microsoft!

It turns out that only the “6.1″ versions of Microsoft Windows (Windows 7 and Windows Server 2008 R2) include multi-threaded robocopy. And my buddies tell me you can’t just move the exe to older versions.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Versioning FAIL: Windows Vista/7 Robocopy
This post was categorized as Enterprise storage, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It's hard to take a product called "Robocopy" seriously!

Anyone doing much storage work on Microsoft Windows machines is familiar with Robocopy. It’s the best tool to move data between NTFS filesystems, since Robocopy maintains permissions and file attributes. It also tolerates dropped connections, resuming where it left off, and can throttle operations over slow networks.

But Robocopy was never very quick, especially when dealing with large data sets. It was single-threaded, hurting performance on high-latency networks, and startup was painfully slow on deep directory structures.

Robocopy XXVII

Windows Vista, Windows Server 2008, and later versions include a new version of Robocopy with performance tweaks designed to overcome these limitations. Right-click on the executable in Windows\System32 and make sure you are using version XP027, 5.1.10.1027.

Update: Oops! Only the “6.1″ versions of Microsoft Windows (Windows 7 and Windows Server 2008 R2) include multi-threaded robocopy!

The big deal here is multi-threading:

• The application can now run in multi-threaded mode using the /MT option. This defaults to 8 threads, but users can specify up to 128 if desired. For example, the following command would use 16 threads:

robocopy c:\ d:\ /MT:16

• Initial directory enumeration is also multi-threaded, so deep directory structures are examined much more quickly.
• XP027 also added the /EFSRAW parameter, allowing one to copy files from EFS using RAW mode. However neither this nor /IPG (inter-packet gap) can be combined with the multi-threaded option mentioned above.

Dead or alive, you’re coming with me!

I tried out the new /MT option on a Core 2 Duo laptop and was surprised by the dramatic improvement in copy performance. Copying my entire “Program Files” directory took well over a minute without multi-threading, but simply specifying “/MT” at the end of the command reduced a second copy to a different directory to about 20 seconds. Using “/MT:32″ was blazing fast – easily less than 15 seconds. I repeated the first single-threaded test again and watched it dawdle along, taking over a minute again to finish.

Microsoft suggests that multi-threaded Robocopy helps with network throughput as well. I tried a series of copies between two Windows 7 machines over Wi-Fi, to simulate a slow network. Although single-threaded Robocopy was able to saturate the network with large files, it really slowed down (thanks to latency) once it hits a patch of smaller files. Multi-threaded Robocopy was more capable of maintaining high throughput once smaller files were encountered, with a 32-thread test keeping the link at maximum pretty much the entire time. Again, a noticeable improvement.

Your Move, Creep

If you’re copying lots of Windows data on a machine running Windows 7 Vista or newer, I can confidently say that the /MT switch will speed things up dramatically. Whether you should stick with the default 8 threads or up it to 16 or 32 depends on the capabilities of your CPU, but it’s worth a try. One more tip: Use the /LOG switch or pipe the output to NULL to speed up copying even more. Displaying all that text delays the whole process!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Robocopy: Better, Faster, Stronger
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I'm a Microsoft MVP!

The news just came in from Redmond: I’m a Microsoft MVP again for 2009! I felt great last year, when I received the award for the first time, but this is even better since I now really understand what it’s all about and how I can use it to help the enterprise storage community!

Reading this blog, one might think that I’m some kind of Apple fanboy. Read some of my Windows storage posts and you might think I was a Microsoft apologist. Talk to me for ten minutes and you’ll see that I’m a long-time UNIX nerd. Or maybe I’m an EMC bigot. Or Sun. Or NetApp.

The truth is I’m not really an anything fanboy: I’m a fan of what works. Mac OS X has become my default desktop OS because it works for me, especially its UNIX underpinnings. Windows Server has awesome storage features. EMC and NetApp make some good storage kit. So do Sun and HDS. See the trend here?

I’m neither a Microsoft apologist or hater. They do some good stuff and they do some bad stuff. I’ll go out of my way to evalgelize the good (the iSCSI initiator, MPIO, VSS, PowerShell, FSRM, Windows Server and Storage Server 2008 R2) and I’ll let other people argue about the rest. And Microsoft folks are apparently wise enough to look past the MacBook and iPhone while I’m on campus!

So I sincerely thank Microsoft for the second MVP award. The access it has given me over the last year has been a huge help to me professionally and to those I work with. I look forward to another productive year!

PS: If you didn’t know, Dustin Pedroia is the best all around baseball player out there. He gives his all for my beloved Red Sox every day, and got his own MVP award last year as thanks!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Dustin Pedroia And I Have Two Things In Common!
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3

Expanded Support for Enhanced vmxnet Adapter

Not specifically a storage change, but the enhanced vmxnet adapter introduced back in the original release of ESX 3.5 now works with most versions of Windows Server 2003 and XP Pro. Look for improved performance when using guest-side SMB and NFS as well as the guest iSCSI initiator. Note that you cannot select this driver when configuring non-Enterprise Edition machines; you have to select Windows Server 2003 Enterprise Edition (64-bit) regardless of which version of Server 2003 you are using.

Expanded SAS and SATA Controller Support

If you’d like to install ESX on a server equipped with a PMC 8011, Intel ICH9 or ICH10, CERC 6/I SATA/SAS Integrated RAID Controller, or HP Smart Array P700m Controller, you’ll find happiness in Update 4.

The Intel controllers are especially important, as we’re seeing them used more and more and this driver is more full-featured than the earlier Broadcom HT 1000 and Intel ICH7 drivers. The Intel ICH9/ICH10 is a dual-mode (IDE/ATA and AHCI/SATA) driver, supports SATA hard drives, SSDs, and optical drives, and now enables VMFS support when in AHCI/SATA mode. It’s not clear whether VMware actually supports VMFS datastores on ICH9/10 SATA, but it says it works. Anyone want to try it out? One thing is certain: You can’t use SATA drives in a shared/clustered environment because SATA does not include reservations. See this tech note and especially this question:

Earlier, it was mentioned that we can create VMFS if we use AHCI/SATA mode. If so, why did VMware not claim VMFS support when using SATA controller running in AHCI/SATA mode?

VMware might decide to add support in the near future. There is no strong need to have VMFS support on a SATA drive, because native SATA protocol does not support reserve/release. Reserve/release is needed if VMFS is used as clustered file system in a shared disk environment.

PXE Boot Support

Rich at VM/ETC points out that Update 4 includes experimental PXE boot support for ESX and ESXi. As he notes, this has major implications for cloud computing platforms, since it means that ESX servers can boot guests without local storage at all. Very interesting! Let’s bet that Update 5 (expected in June or July) will include this as a supported option.

Updated QLogic, Emulex, and LSI Drivers

Like most ESX updates, this one included updated Fibre Channel drivers.

• The QLogic Fibre Channel Adapter driver and firmware (versions 7.08-vm66 and 4.04.06, respectively) include bug fixes and enhanced NPIV support.
• On the Emulex side, driver version 7.4.0.40 supports the company’s HBAnyware 4.0 management software.
• Users of SAS and SCSI LSI MegaRAIDs will find driver version 3.19vmw (megaraid_sas) and 2.6.48.18 vmw (mptscsi) which improves performance and enhances event handling capabilities.

Expanded Sun Storage Array Support

All you StorageTek loyalists out there will be happy to see support for Sun’s low-end StorageTek 2530 SAS array as well as the modular 6580 and 6780 Fibre Channel arrays. It looks like just about every model in Sun’s current storage lineup is now supported in ESX.

Expanded Network Card Support

Support for Gigabit cards is greatly expanded, including HP’s quad-port NC375i and dual-port NC362i and NC360m, Intel’s Gigabit CT and 82574L, and NetXtreme’s BCM5722, BCM5755, BCM5755M, and BCM5756. Intel’s widely-used 10-gig 82598EB cards are now supported as well.

Tweaks and Fixes

Looking through the release notes, a few storage-related tweaks and fixes stand out:

1. WMware can optionally automatically throttle back the queue depth when congestion is encountered. See Controlling LUN queue depth throttling in VMware ESX for 3PAR Storage Arrays for more information.
2. VMklinux module heap size can now be adjusted as LUN queue-depth values are increased. Since tuning LUN queue depths is one common trick of the storage trade to improve performance, especially in queue-stingy systems like ESX, this is welcome news. But call VMware support before you monkey with it!
3. An RDM-related issue where SCSI inquiry data over 36 bytes was truncated or corrupted (for example when using Microsoft VSS and NetApp SnapDrive) has been resolved.

Well, that’s all folks. I suggest you all read the release notes for yourself, and please leave a comment if you see an error in what I wrote here or have something to add!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Changes in VMware ESX 3.5 Update 4
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.