Anyone doing much storage work on Microsoft Windows machines is familiar with Robocopy. It’s the best tool to move data between NTFS filesystems, since Robocopy maintains permissions and file attributes. It also tolerates dropped connections, resuming where it left off, and can throttle operations over slow networks.
But Robocopy was never very quick, especially when dealing with large data sets. It was single-threaded, hurting performance on high-latency networks, and startup was painfully slow on deep directory structures.
Robocopy XXVII
Windows Vista, Windows Server 2008, and later versions include a new version of Robocopy with performance tweaks designed to overcome these limitations. Right-click on the executable in Windows\System32 and make sure you are using version XP027, 5.1.10.1027.
Update: Oops! Only the “6.1” versions of Microsoft Windows (Windows 7 and Windows Server 2008 R2) include multi-threaded robocopy!
The big deal here is multi-threading:
- The application can now run in multi-threaded mode using the /MT option. This defaults to 8 threads, but users can specify up to 128 if desired. For example, the following command would use 16 threads:
robocopy c:\ d:\ /MT:16
- Initial directory enumeration is also multi-threaded, so deep directory structures are examined much more quickly.
- XP027 also added the /EFSRAW parameter, allowing one to copy files from EFS using RAW mode. However neither this nor /IPG (inter-packet gap) can be combined with the multi-threaded option mentioned above.
Dead or alive, you’re coming with me!
I tried out the new /MT option on a Core 2 Duo laptop and was surprised by the dramatic improvement in copy performance. Copying my entire “Program Files” directory took well over a minute without multi-threading, but simply specifying “/MT” at the end of the command reduced a second copy to a different directory to about 20 seconds. Using “/MT:32” was blazing fast – easily less than 15 seconds. I repeated the first single-threaded test again and watched it dawdle along, taking over a minute again to finish.
Microsoft suggests that multi-threaded Robocopy helps with network throughput as well. I tried a series of copies between two Windows 7 machines over Wi-Fi, to simulate a slow network. Although single-threaded Robocopy was able to saturate the network with large files, it really slowed down (thanks to latency) once it hits a patch of smaller files. Multi-threaded Robocopy was more capable of maintaining high throughput once smaller files were encountered, with a 32-thread test keeping the link at maximum pretty much the entire time. Again, a noticeable improvement.
Your Move, Creep
If you’re copying lots of Windows data on a machine running Windows 7 Vista or newer, I can confidently say that the /MT switch will speed things up dramatically. Whether you should stick with the default 8 threads or up it to 16 or 32 depends on the capabilities of your CPU, but it’s worth a try. One more tip: Use the /LOG switch or pipe the output to NULL to speed up copying even more. Displaying all that text delays the whole process!
I’m curious whether they’ve fixed the fragmentation problems that occurred with multithreaded RichCopy. With multithreaded copy tools, the tool should completely preallocate the target files prior to starting the copy, so that a contiguous block of space is reserved within the target filesystem for each file. Without preallocation, the filesystem will allocate space on demand, as the copy tool writes data.. and with multiple concurrent threads handling different files, this results in the target filesystem being heavily fragmented.
I’ve typically used EMCopy for file-based migrations; it’s included with the EMC Celerra, and the latest version is multithreaded and preallocates target files to avoid target-side fragmentation.
Just tried /MT option on Windows 2008. Doesn’t work. May be on R2?
We ran some tests comparing Robocopy with and without the /MT (multithread) option with local disk, 1GbE network and 10GbE network. Our results are available at http://www.demartek.com/Demartek_Presenting_RMWTUG_March_2011-03.html. The /MT option provides improved performance.
Does anyone else notice that the log looks significantly different?! It’s very hard to read. I do
Nevermind. It seems to behave differently when executed from a network share.
Hello,
just another information : /MT only work with 64 bits versions
Best way to Move or Copy date with same Date is using this software
http://www.lantechsoft.com/data-copy.html
It has GUI interface and works beautifully.
Lovely & Easy way to retain Original Date Time Stamp while copying files.
GSCopy
Pro v6.0 (RoboCopy Alternative) with Open File Agent
GSCopyPro is
a single command-line tool (CLI) that can copy, replicate and move files from
one folder to another. This folder can be on the same machine/ server or
another server elsewhere. What makes GSCopyPro stand out from other competitors is the fact it works on 32-bit as well as 64-bit
systems and has no restrictions. It can easily be scheduled to run as
a scheduled task and fully automated. GSCopyPro also comes with an open file
agent which can copy files that are locked/ opened by other processes. This
feature is supported in all windows versions from widows XP/ 2003 and later.
Go
To:>> http://www.gurusquad.com/GSCOPYPRO
Can anyone plz tell that Robocopy running on 32 Threads in 64 GB RAM and 128 GB RAM would save time?
In other words do robocopy Multithreading depedns on RAM Memory??
Plz Help
Just share the best alternative is try to use “Long Path Tool” program for the real robocopy.
Haha, yes, I love the Robcop references. I sent some tips on using Robcopy to a coworker and I had said the same things in my email to him.
You should probably clarify that by NULL you mean > NUL, otherwise people may end up with an unwanted file 😉
Robocopy looks like a very complex task and feels like it requires very much technical knowledge so I was looking for a simple yet effective software. My friend suggested me a software called GS Richcopy, its a paid software but it it affordable and provides overwhelming features like pre scheduled file transfer, email notification when task is done. This software is much better than any other software that I have used till date.
The extent of multi threaded file transfer is still unknown. Many software are multi threaded these days, but they can only handle 3 threads at max. A software which claims to provide 100% multi threaded file transfer is GS Richcopy 360. And I have seen myself that the transfer speed has significantly increased. Give it a try!
NICE ARTICLE
I ran test comparing this software with /MT attribute and GS Richcopy 360, my software was way ahead than this one. My software doesn’t mean I own it, just a customer, a happy and satisfied one! Haha!
use robocopy
We didn’t like the log of RoboCopy as it is in DOS format which isn’t really good.. Try GS RichCopy – has got all the features of RoboCopy, but it has some more features like copying open/locked files, can copy the timestamps, and has a GUI interface with a nice looking log which you can easily copy and paste in a notepad.
I ran test comparing this software with /MT attribute and GS Richcopy 360, my software was way ahead than this one. My software doesn’t mean I own it, just a customer, a happy and satisfied one! Haha!