It's hard to take a product called "Robocopy" seriously!

Anyone doing much storage work on Microsoft Windows machines is familiar with Robocopy. It’s the best tool to move data between NTFS filesystems, since Robocopy maintains permissions and file attributes. It also tolerates dropped connections, resuming where it left off, and can throttle operations over slow networks.

But Robocopy was never very quick, especially when dealing with large data sets. It was single-threaded, hurting performance on high-latency networks, and startup was painfully slow on deep directory structures.

Robocopy XXVII

Windows Vista, Windows Server 2008, and later versions include a new version of Robocopy with performance tweaks designed to overcome these limitations. Right-click on the executable in Windows\System32 and make sure you are using version XP027, 5.1.10.1027.

Update: Oops! Only the “6.1″ versions of Microsoft Windows (Windows 7 and Windows Server 2008 R2) include multi-threaded robocopy!

The big deal here is multi-threading:

• The application can now run in multi-threaded mode using the /MT option. This defaults to 8 threads, but users can specify up to 128 if desired. For example, the following command would use 16 threads:

robocopy c:\ d:\ /MT:16

• Initial directory enumeration is also multi-threaded, so deep directory structures are examined much more quickly.
• XP027 also added the /EFSRAW parameter, allowing one to copy files from EFS using RAW mode. However neither this nor /IPG (inter-packet gap) can be combined with the multi-threaded option mentioned above.

Dead or alive, you’re coming with me!

I tried out the new /MT option on a Core 2 Duo laptop and was surprised by the dramatic improvement in copy performance. Copying my entire “Program Files” directory took well over a minute without multi-threading, but simply specifying “/MT” at the end of the command reduced a second copy to a different directory to about 20 seconds. Using “/MT:32″ was blazing fast – easily less than 15 seconds. I repeated the first single-threaded test again and watched it dawdle along, taking over a minute again to finish.

Microsoft suggests that multi-threaded Robocopy helps with network throughput as well. I tried a series of copies between two Windows 7 machines over Wi-Fi, to simulate a slow network. Although single-threaded Robocopy was able to saturate the network with large files, it really slowed down (thanks to latency) once it hits a patch of smaller files. Multi-threaded Robocopy was more capable of maintaining high throughput once smaller files were encountered, with a 32-thread test keeping the link at maximum pretty much the entire time. Again, a noticeable improvement.

Your Move, Creep

If you’re copying lots of Windows data on a machine running Windows 7 Vista or newer, I can confidently say that the /MT switch will speed things up dramatically. Whether you should stick with the default 8 threads or up it to 16 or 32 depends on the capabilities of your CPU, but it’s worth a try. One more tip: Use the /LOG switch or pipe the output to NULL to speed up copying even more. Displaying all that text delays the whole process!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Robocopy: Better, Faster, Stronger
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Western Digital is first to bring "Advanced Format" 4k-sector drives to market, but buyers should beware of installing them in old systems!

Personal computer hard disk drive access methods have repeatedly been forced to adapt to ever-expanding capacity. Physical (CHS) addressing and 28-bit LBA have been abandoned, but the disk drives themselves retain the tiny 512-byte sectors they have had since IBM introduced fixed-block addressing in the 1970′s. But time marches on, and Western Digital is leading the change to larger 4 kilobyte hard disk blocks. Although this new “Advanced Format” includes mechanisms for backwards compatibility, buyers should be wary of these new drives for the time being.

A Little History

A long, long time ago, before there were personal computers, IBM set the standard for data storage with their CKD (“count key device”) format. Variable-sized records were stored on a raw disk with gaps between them. Each included a sequence number (“count”), an optional key, and the data itself.

This flexible format proved impractical as multi-process systems evolved in the 1970′s, so IBM introduced a new format: Fixed-block architecture (FBA). FBA disk drives were divided up into equal-sized blocks and addressed by their physical location on a specific cylinder accessed by a specific head. Each cylinder/head pairing had its own series of sectors, starting at the outer edge of the physical disk platter and moving inward.

This system of addressing hard disk capacity with a triple number identifying the cylinder, head, and sector (“CHS”) continued in the PC space in the 1980′s. Since early hard disk drives were tiny by modern standards, the PC industry chose the smallest sector size specified by IBM, 512 bytes. The earliest MS-DOS/BIOS PCs were limited to just 504 MB, thanks to incompatible defaults for the CHS address space. This forced a transition, initially raising the addressable capacity to 7.8 GB.

But the writing was on the wall: CHS just didn’t make sense. Storage protocols, including ATA, were being pressed into service for devices that didn’t have cylinders or heads, and the 128 GB limit of ATA-1 wouldn’t cut it forever. So the industry transitioned again, abandoning physical references in favor of a simple logical block addressing (LBA) scheme. By the time ATA-6 was introduced in 2002, all systems used LBA, and the new 48-bit address system allowed a maximum of 128 PB of capacity per device.

Though hard disk capacity had ballooned from the megabyte to the terabyte range (a million-times increase in 30 years), the 512-byte disk sector remained, each with its own error-correcting code (ECC) and gap.

Up To Date

Processors and file systems have long since moved on from 512-byte blocks of data, with 4 kilobytes being the most common size used by the x86 CPU and NTFS, ext3/4, and HFS+ filesystems in Windows, Linux, and Mac OS X. With just about every storage access triggering the reading or writing of eight disk sectors, an increase in hard disk sector size to 4 KB seems obvious. Stepping up to 4K enables better ECC checksums, making disks more reliable (in theory) and freeing up capacity.

Modern operating systems, including Windows Vista, Windows 2008, Windows 7, Mac OS X 10.4-10.6, VMware ESX 3.x-4.x, and most recent versions of Linux, will have no problem with 4 KB sectors in most cases. Installing a 4K drive, like Western Digital’s new “R” line, and partitioning and formatting it fresh should present no issues for most users. In fact, these operating systems will not even “know” they are talking to any special kind of drive, since Western Digital presents its new 4 KB sectors as plain old 512 byte sectors for compatibility reasons.

What’s Your Alignment?

Other systems present a challenge, however. MS-DOS traditionally started filesystems at sector number 63, and most other PC operating systems (including old versions of Linux and VMware) followed this convention. This presented no issue at all as long as disks used 512-byte sectors: Requesting 8 sectors starting at number 63 was the same as requesting eight starting at sector 64. But things get mucked up when the disk drive uses 4 KB sectors.

See the misalignment here? A starting offset of 63 moves the filesystem cluster across the 4K sector boundary. This misalignment means that every filesystem cluster access results in two disk sector accesses. This is a Very Bad Thing, especially for to write performance. Layers upon layers of convention, standards, and compatibility have left us with a problem.

Western Digital solves this problem in one of two ways:

1. Jumper pins 7 and 8 on the drive and it will internally offset all of its logical 512-byte sectors by one
2. Correctly align all partitions written to these new drives

The jumper method, though simple, is not a great idea. If the drive was ever moved or reformatted to a system that aligns its volumes differently (as do all modern operating systems), it would be puzzlingly slow. Instead, it’s best to simply realign any filesystems written to the drives, and Western Digital is providing a utility do just that.

USB, Drobo, and TiVo, Oh My!

Most external drive enclosures will simply pass read and write requests to these new drives as they get them. This means that an external USB enclosure with an Advanced Format drive will have exactly the same issues as an internal drive if it is partitioned under Windows XP or another old operating system. Not only that, but portable drives often get moved from system to system, so an XP-formatted USB disk will have this misalignment issue even when attached to a Windows 7 machine! As these 4K drives begin hitting the market, users should be careful to note if they are correctly aligned.

But what about other disk-using devices like Drobos and TiVos? The jury is still out here. TiVo appears to use misaligned partitions, so I don’t recommend installing 4K drives in a TiVo at this point. I contacted a number of other manufacturers, most of whom are “looking into” the matter.

Data Robotics did respond, and even posted a tech note about 4K drives on their support site. Their short answer is no, that 4K drives should not be used in Drobo storage devices. However, it is unclear if they are affected or just erring on the side of caution until they can test these new hard disks. They also promise an update allowing the use of 4K drives in the future.

I would be cautious about using 4K drives in anything other than a newish PC or Mac at this point. I will continue following the situation and will post updates on my blog as news comes out.

More Info

The following articles also offer great insight into these new 4K drives:

• Western Digital: Advanced Format Technology White Paper
• StorageMojo: Why we need 4k drives
• AnandTech: Western Digital’s Advanced Format: The 4K Sector Transition Begins

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Drobo, XP Users: Beware of 4K “Advanced Format” Drives!
This post was categorized as Apple, Computer History, Enterprise storage, Everything, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The Iomega ix4-200d is a sleek 4-drive SOHO RAID system that does just about everything, from NAS to Time Machine to iSCSI for a list price right around $700

Iomega is well into its second coming as EMC’s entry-level storage division. First, they applied EMC’s compact and full-featured LifeLine home storage software to existing gear, giving birth to the Home Media Network Hard Drive, StorCenter ix2, and StorCenter Pro ix4-100. Then they wooed the small-business community with the rack-mount StorCenter ix4-200r, adding iSCSI target support and VMware compatibility.

Today, they are back with the new ix4-200d, probably Iomega’s best product yet. It includes every feature of the rack-mount ix4-200r, including NAS and iSCSI target mode plus great new stuff like one-touch synchronization. All of this is packaged in a Drobo-like desktop system with a starting list price of just $700, or less than half the cost of a comparable ix4-200r!

Desktop Storage

External desktop storage products, exemplified by Western Digital’s wildly successful My Book series, have been a huge retail hit. Priced just over $100, these drives pack a terabyte or more and offer plug and play simplicity. I recently visited a small business with a WD or Seagate USB enclosure on every single desk. I’ve purchased five USB- or FireWire-connected hard drives myself over the last two years!

But these single-drive desktop wonders are a disaster waiting to happen:

• They fail frequently (like my Maxtor 3200), instantly wiping out the data they contained
• They are targets for thieves, so data loss prevention (DLP) experts warn against their use
• They aren’t shareable natively, so most people resort to sneakernet swapping rather than fight with Windows to present them as a network share
• When they’re full, they’re full, forcing the purchase of a whole new drive

Many vendors sell grown-up versions that address some or all of these concerns with multiple drives, network connections, and encryption, but these have been slow to catch on. Since they contain redundant drives and extra hardware and software, they are much more expensive than their little cousins. Iomega has done battle in this arena with their original StorCenter ix2 and the ix4-100, but these have yet to catch on. Even Drobo, with their fanatical user-friendly focus, has failed to convince many buyers.

Then there is the world of business storage. Way down at the bottom of the enterprise storage pyramid lies the realm of small 4- and 8-drive storage arrays. These SMB storage arrays offer a lot of capacity and reliability for the money but very little in the way of features. Iomega’s StorCenter Pro ix4-200r, in contrast, wowed the techies with a full iSCSI target stack that was certified for VMware ESX. But the price, over $1500, definitely limited sales to the home hobbyist.

“Just Right” Storage

Although the name is similar to the StorCenter Pro ix4-200r launched this Spring, Iomega went back to the drawing board for the ix4-200d. They built an entirely new device that could offer the impressive features of their StorCenter Pro line at a price closer to consumer storage offerings. The result boasts everything the Pro has and more at less than half the price:

• You want connectivity? The ix4 sports dual gigabit Ethernet ports that can be teamed up for performance or split off for redundancy.
• You want NAS? The ix4 supports NFS, SMB, and even Apple’s AFP, plus it’s Active Directory compatible.
• How about iSCSI? It’s a full-featured iSCSI target, certified for Microsoft Server 2003 and 2008.
• Want to host virtual machines? The ix4 is certified with VMware ESX 4 vSphere using both NFS and iSCSI, on the Xen HCL, and that Microsoft logo means it will work with Hyper-V as well.
• Need backup? The ix4 comes with EMC’s Retrospect and supports OS X Time Machine over AFP just like an Apple Time Capsule.
• Looking for weird features? How about support for up to 5 Axis network cameras, BlueTooth Picture Transfer Protocol, and UPnP/DLNA media service!
• Iomega also added a new feature, QuickTransfer, to synchronize files between devices.

This is one seriously feature-rich storage system. In fact, this glut of features is the ix4′s Achilles heel: How do you effectively communicate the value of a device that does so much? Most of the buying public has never heard of most of these features, so the price remains hard for some to justify.

Every ix4 configuration comes loaded with four hard disk drives

Another hurdle for the ix4 is Iomega’s decision to fill it with hard drives. One cannot buy an empty ix4-200d, and both the 2 TB and 4 TB configurations come loaded with four hard disk drives. This raises the price of entry and scares off the very techies who might be interested in the device. The majority of Drobos are purchased with no drives at all, and storage geeks like me love the idea that disks can be added as-needed in the future. Although Iomega is open to users swapping out the drives in their own StorCenter device, this is not its intended use case. Iomega’s decision to sell the ix4-200d as a loaded appliance seems counter to the price sensitivity and flexibility needs of buyers.

QuickTransfer: Data Synchronization for Everything

One nifty new feature included with the ix4-200d is QuickTransfer, a one-touch data synchronization capability. Leveraging rsync technology, but hiding this complexity with a more-friendly wizard-based GUI, QuickTransfer allows users to set up synchronization jobs between the ix4 and a variety of targets:

• USB drives can be plugged into one of the three USB 2.0 ports and synchronized with a subset of the content of the ix4. For example, a portable USB drive could be “recharged” with the latest set of data before one heads out of the office.
• Two Iomega ix4′s, or other NAS systems for that matter, can be synchronized over the Ethernet/IP LAN. This would provide a robust and bandwidth-friendly remote office backup or data replication solution.
• A PC or Mac can also be synchronized over a network share, providing a simple alternative to the bundled backup software.

QuickTransfer is exclusive to the ix4-200d for now, but Iomega assured me that it would be included in future StorCenter products and added to the StorCenter Pro ix4-200r in the near future. It is unknown if or when other existing StorCenter devices will get QuickTransfer, however.

Iomega’s Next Move

What will Iomega do next? They must be ready to announce their vSphere 4 and Microsoft Windows Server certification soon, since both company’s web sites already list the device in their compatibility lists. Iomega ought to try to take advantage of the interest among VMware users with a big VMworld splash. They will be there, but it is awfully hard to get noticed at such a large event. I am looking forward to the event to get a hands-on test.

The ix4 should begin showing up for sale at online stores very quickly. Amazon listed the rackmount product within days of its release, and we expect the same this time. But will Iomega offer this cheaper device in retail stores? It would be great to have it available at Staples and Best Buy, but shelf space for a storage system this expensive would be hard to get. Instead, expect it at specialty outlets like Fry’s and perhaps Micro Center.

How much does the StorCenter ix4-200d cost?

• The 2 TB model (SKU# 34546 with four 500 GB drives) lists at $699.99
• The 4 TB model (SKU# 34549 with four 1 TB drives) lists at $899.99
• The 8 TB model (SKU# 34563 with four 2 TB drives) lists at $1,899.99

It also seems likely that the 2-bay product is up for a refresh in the near future. The ix2 can’t be said to be very attractive, so hopefully the company will do something about this with the next-generation product. It is unclear whether the inexpensive 2-bay device will get iSCSI support, but I suspect it will. Beyond this, might Iomega move further upmarket with an 8-drive unit? We shall see!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Iomega’s ix4-200d: A Killer Desktop Storage Array
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

One of the great things about being a Microsoft MVP is the access I get to Microsoft software and staff. As I mentioned in my post about the 10 cool storage features from the 2009 Microsoft MVP Summit, I was able to preview a lot of what Microsoft is doing with their Server software and storage features. And the best part is that the Microsoft product teams are keenly interested in our feedback and suggestions. I’m told, for example, that the awesome iSCSI Quick Connect feature in the new Windows iSCSI initiator software was developed based on my feedback!

As I note on my Windows Storage Server 2008 preview on Gestalt IT, Microsoft has always kept WSS close to the vest. It’s only available to OEMs, not retail customers, and has never even been shared with TechNet or MSDN subscribers in the past. So I was really pleased when Microsoft gave the File System Storage MVPs access to a beta version of WSS 2008 so we could get a feel for all of the new features. I’ve also had some great conversations this week with the Microsoft product managers responsible for it.

What’s exciting about Windows Storage Server 2008?

1. It includes all of the storage enhancements in Windows Server 2008, including SMB 2.0 for much much faster file servicing over higher-latency links, SMfS, FSRM, enhanced VDS, and failover clustering.
2. WSS is the only way to get access to Microsoft’s iSCSI target software. It’s been improved in many ways from the prior releases, but its support for what Microsoft calls dual-active clustering is probably its most notable feature: You can’t share the same active LUN between cluster members, but each can have its own active LUNs and the can all fail over in the event that one member goes down.
3. The included single-instance storage (SIS) file-based deduplication has been much improved, scaling to 128 volumes per server and millions of files. It’s still not as effective capacity-wise as block-level deduplication (which I’d love to see, hint hint), but the performance is solid enough to use it for primary storage with production applications.
4. Probably the coolest feature exclusive to Windows Storage Server 2008 is its new browser-based remote administration capability. Just point your browser to the Storage Server machine (for example, “http://wss/desktop”) and you’ll get a full ActiveX version of RDP. Don’t use Internet Exploder? Firefox and Linux users will get a Java-based RDP instead! I will cover this feature more in the future, but let’s just say that every operating system should offer this!

Want to try Windows Storage Server for yourself? Breaking from the past, Microsoft will soon (like next week!) allow TechNet subscribers to download the full install. OEMs have a sekrit back-door site to try it out, too.

One more thing… This will be the last release of Windows Storage Server. There won’t even be a special Storage Server version of Server 2008 Service Pack 2! Starting now, Storage Server is just an optional feature of Windows Server. Purchasing and production use will still be limited to storage OEMs, but Microsoft has finally reconciled Storage Server with the rest of the Windows Server world. I imagine that most OEMs will release Service Pack 2 updates for their Storage Server customers shortly, and that future versions of the product will come closer to the base Server versions than WSS 2008. Although I can’t share what I know, I will say that Microsoft is continuing active development on their iSCSI target, single-instance storage, and other Storage Server features. I imagine that Windows Server 2008 R2 will support storage systems in the very near future!

On a personal note, reading A Brief History of Windows Storage Server Releases from the Microsoft Storage Server blog reminded me of the original unveiling of Windows Storage Server at Storage Decisions Chicago in June, 2003. The company loaded us all on one of those lake cruise boats with some celebrity impersonators, chocolate “Oscar”-style statues, an open bar, and a band. Good times ensued!

For more details, check out my Gestalt IT piece, Windows Storage Server-Based Systems Step Into 2008

Microsoft is detailing the new version of Windows Storage Server 2008 in a webcast Thursday at 8 AM Pacific. You should also check out the Microsoft Storage Server blog.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | I Can Finally Talk About Windows Storage Server 2008!
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

So would there be a major Windows Server update just two or three years after the last major version? And what impact would this have on the masses that still haven’t moved from Windows Server 2003 (or even 2000)?

Thankfully, the Group Product Manager for Windows Server, Ward Ralston, has clarified the issue. There will be no Windows 7 Server. Instead, we will see an update to Windows Server 2008 called (unsurprisingly) Windows Server 2008 R2 which will incorporate any server-side enhancements needed for Windows 7 clients. Ralston’s comments about a 2/4 schedule of minor/major updates leads me to assume that the next major Windows Server update will come in 2012 at the earliest and will probably share whatever the post-Windows 7 client codebase becomes.

See the official Microsoft Windows Server Roadmap for more details, and note that Windows Storage Server 2008 is noted for release this year. I can’t wait to get my hands on that!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Windows 7 Server == Windows Server 2008 R2
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

1. Support for Volume Shadow Copy Service (VSS) for filesystems Windows Server 2003 and Windows Server 2008 guests.  This means VMware services like VCB and SRM can now signal Windows Server to quiesce filesystems before creating a clone or snapshot and is a major addition!
2. Windows Server 2003 guests also get application quiescing, where supported.
3. You can now extend a live, running VMFS volume as storage is added, just like Windows Vista and 2008 guests could already do with raw device mode (RDM).  Note that this only works for flat disks with no persistent snapshots open.

A few items are now officially supported, too:

1. Support for Storage VMotion between and among Fibre Channel and iSCSI datastores is here, and in Update 1.
2. iSCSI over 10 gigabit Ethernet (10 GbE) is now supported allegedly, though the I/O Compatibility Guide still says “Gigabit only”.  Of course, it already worked, but it’s nice to have official support just the same…  Bring on the datacenter Ethernet!  They’ll support this in Update 1 too.
3. And NFS over 10 GbE is supported, too!

1. They supposedly added support for 8 gigabit Fibre Channel (8 Gb FC) HBAs, but I don’t see it in the I/O Compatibility Guide. The Emulex LP12000/LP12002 are noted as 4 Gb only, the QLogic QLE2560/QLE2562 aren’t listed, the HP 81/82 series isn’t there either.  Update:  QLogic and Emulex HBAs are supported after all!
2. Support for more Serial-Attached SCSI (SAS), including Dell SAS 5/E, LSI 1078, Sun SG‐XPCIESAS‐R‐EXT‐Z, and HP SC08GE is listed in the Stotage Compatibility Guide.
3. CIM system health monitoring of some QLogic and Emulex HBAs is new too.

[ad#banner]

See also:

• Scott Lowe’s (as always) excellent overview
• VM /ETC’s colorful post
• Excellent coverage at Yellow Bricks

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Storage Fixes in VMware ESX Server 3.5 Update 2
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.