Introducing NTFS

A filesystem is the basic scheme for organizing data on a hard disk drive. The common FAT filesystem, supported by just about every operating system, is just too basic for modern operating systems. Microsoft introduced the advanced NTFS filesystem with their Windows NT operating system in 1993. Borrowing from OS/2′s HPFS and VMS’ Files-11, NTFS is a journalling filesystem with many features, including POSIX links, alternate data streams, sparse files, compression, and encryption.

Although just about everything Microsoft produces is maligned by UNIX and Mac geeks, this hatred is unwarranted in the case of the many storage advances coming out of Redmond. Microsoft may be fairly criticized for producing expensive, proprietary systems, but they have dome some amazing things with disks, and NTFS is arguably one of the most advanced filesystems in wide usage. Much of this credit goes to Windows NT architect Dave Cutler, who also led development of DEC VMS, the RISC concepts that became Alpha, and Microsoft’s Azure cloud platform.

Although every version of Windows supports evolutions of the old FAT filesystem, modern iterations increasingly require NTFS. Windows XP allowed either FAT32 or NTFS for booting; Windows Vista could be forced to boot from FAT32; Windows 7 requires NTFS. Therefore, most PC hard disk drives, including external USB drives, now come formatted with NTFS by default.

NTFS on Mac OS X

This leads to issues for Mac users, especially when they dual-boot with Boot Camp or buy external hard disk drives. As mentioned, all versions of Mac OS X since 10.3 “Panther” have included read-only NTFS support, but those wanting full access had two options:

1. Paragon Software’s NTFS for Mac OS X is a full and supported read/write solution. Although available for purchase separately, many hard disk drives like the Seagate GoFlex now include a free copy of Paragon NTFS.
2. The open source NTFS-3G driver has also been ported to Mac OS X, allowing read/write support for free.

Mac OS X 10.6 “Snow Leopard” adds a third alternative: Native read/write support for NTFS. But it’s not as simple as that. Although early betas enabled read/write support by default, it was limited once again to read-only in the final releases.

It is possible and even simple) to enable read/write support with the native Snow Leopard drivers, but this is definitely an “at your own risk” proposition. Users have reported kernel panics when using these methods, and although I have not heard of data corruption, it is possible as well.

The Manual Method: mount_ntfs

Making disks available is called “mounting” in UNIX geek-speak, and mount commands in modern operating systems are like ogres: They have many layers. Plug in a drive, and Mac OS X makes it available for you using a layer cake of commands, but the bottom-most command is “mount”. This simple command-line utility takes a few arguments and attaches a raw disk device to a point in the unified filesystem tree.

When presented an NTFS drive, Mac OS X calls a utility called mount_ntfs that resides in the /sbin directory, and we can call it, too. Although OS X always tells mount_ntfs to mount NTFS drives read-only, we can tell it to enable writing as well by using the “-o rw” flag.

There’s just one problem: Where should we connect the NTFS file system? Mac OS X automatically creates a new directory (called a “mount point” in UNIX-speak) in /Volumes with the name of the filesystem. For example, the GoFlex drive I bought yesterday mounts as “/Volumes/FreeAgent GoFlex Drive” by default. But mount_ntfs won’t create this mount point, so we have to do it.

Here are the steps to mount an NTFS drive read/write from the command line:

1. Plug in the drive and let Mac OS X mount it read-only.
2. Look in Finder to see the name of the drive. For example, my drive mounts with the name “FreeAgent GoFlex Drive” and this is what shows up on the desktop and in Frinder under “Devices”.
3. Open the Terminal application to get command-line access. We’ll use “sudo” which executes commands as the administrator instead of a regular user.
4. Type the following command in Terminal to get necessary information about the drive. Use the drive name you see in Finder in place of “drive name”, and make sure to use quotation marks around it since Terminal treats un-quoted spaces as separators rather than part of the name.

   diskutil info "/Volumes/drive name"

5. Record the “Device Node” entry, which will look something like “/dev/disk4s1″. This is where the raw partition is mapped by the disk subsystem, and where the filesystem driver will read its data.


The diskutil command gives us the "Node Name", which we'll need momentarily

6. Eject the drive, using option-click in Finder or with the hdiutil command in Terminal:

   hdiutil eject "/Volumes/drive name"

7. Now you must create the mount point again. Type the following:

   sudo mkdir "/Volumes/drive name"

8. Finally, use mount_ntfs to mount the drive read/write using the following command. Use the node name you spotted above and the mount point you just created:

   sudo mount_ntfs -o rw /dev/disk?s? "/Volumes/drive name"

9. When you’re done, you can use the Finder, the hdiutil command, or just about anything else to eject the drive. Make sure your mount point is deleted as well!

This method is great for occasional NTFS use, which is frankly all Mac users should need. If you intend to use a drive frequently, I highly recommend reformatting it with the native HFS+ filesystem since it will be fully-supported by the operating system.

The Per-Drive Method

Let’s say you have an NTFS drive you frequently need to mount read/write and find the above method too much of a hassle. Maybe it’s your Boot Camp drive, or perhaps it’s a USB hard disk drive you share with a Windows-using colleague. It is possible to tell Mac OS X to always mount a given NTFS drive read/write using its native driver using the following commands. However, I recommend investing in the Paragon driver for this use case, since it’s fully-supported!

Like all UNIX systems, Mac OS X uses a configuration file to determine what it should do with certain drives. This file, called “fstab” and located in “/etc”, is not present by default but will override the “read-only” flag on a per-drive basis if found. This allows a drive to always be mounted read/write without any hassle.

1. Get the drive name as in step 2 above.
2. Open Terminal and type the following command to edit the /etc/fstab file:

   sudo nano /etc/fstab



Adding a line in /etc/fstab will make Mac OS X always mount a single drive in read/write mode

3. “Nano” is a command-line editing program. Add the following line to the file (and don’t forget the quotes around the drive name if it contains spaces!)

   LABEL="drive name" none ntfs rw

4. Now type control-x to save and exit, type y for yes, and hit enter.
5. After a reboot, Mac OS X should automatically mount any drive with that label as read/write

Note that this can also be done using the NTFS UUID of the drive, but this isn’t present on every drive to LABEL works better. Also, some have reported needing multiple reboots to make it work. I have no idea why.

The Permanent Universal Method

It is also possible to permanently modify how Mac OS X calls the mount_ntfs command to always mount every NTFS drive in read/write mode. Again, I will point out that this isn’t necessarily a great idea since Apple doesn’t support using the driver in this way. But here it is!

1. Open Terminal. We won’t need the drive name or Node Name since this applies to every NTFS drive.
2. Rename the mount_ntfs command to something else by typing the following:

   sudo mv /sbin/mount_ntfs /sbin/mount_ntfs_orig

3. Create a shell script called mount_ntfs using the Nano editor:

   sudo nano /sbin/mount_ntfs

4. This script will simply call mount_ntfs with the “-o rw” flag every time it’s used. Type the following in Nano:

   #!/bin/sh
   /sbin/mount_ntfs_orig -o rw "$@"

5. Type control-x to save and exit, type y for yes, and hit enter.
6. Now we’ll fix the permissions on this new file so it’s owned by root and group wheel:

   sudo chown root:wheel /sbin/mount_ntfs

7. Finally, we make the shell script executable:

   sudo chmod 755 /sbin/mount_ntfs

8. Now every time OS X encounters an NTFS drive, it will mount it read/write by calling this script rather than the “mount_ntfs” executable it thinks it’s using.

Note that future OS updates might “break” this method by replacing the /sbin/mount_ntfs command, or it could cause other issues. But it works for now and doesn’t even require a reboot.

Stephen’s Stance

I’m hesitant to recommend that casual Mac users employ any of these methods to access NTFS drives. Each requires superuser access and careful typing. You can easily mess up your system this way, and unsupported uses of the NTFS driver might result in data loss.

If you aren’t comfortable with this kind of mucking-about or intend to frequently write to NTFS drives, I highly suggest installing NTFS-3G or Paragon’s NTFS driver. In fact, the best method is simply to go buy a Seagate GoFlex drive and use the Paragon driver that comes with it!

But hackers and geeks like me want to know how to do this. This is why I decided to document it here.

Note: Some of these links include affiliate codes that help pay for this blog. For example, buying an Amazon Kindle with this link sends a few bucks my way! But I don't write this blog to make money, and am happy to link to sites and stores that don't pay anything. I like Amazon and buy tons from them, but you're free to buy whatever and wherever you want.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | How To Write To Windows NTFS Drives In Mac OS X 10.6 “Snow Leopard”
This post was categorized as Apple, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

File Descriptors

Anyone familiar with its internals will tell you that everything is a file to a UNIX operating system. It’s the philosophy behind the system: Network connections, storage systems, system parameters, and processes all have file interfaces, and each of these pseudo-files needs a unique file descriptor.

What does this mean for lighttpd? Well, every time a visitor accesses a page, lighttpd uses three file descriptors: An IP socket to the client, a fastCGI process socket, and a filehandle for the document accessed. Lighttpd stops accepting new connections when 90% of the available sockets are in use, restarting again when usage has fallen to 80%. With the default setting of 1024 file descriptors, lighttpd can handle a maximum of 307 connections. This is a lot. But it is possible to exceed this number under times of high load.

To prevent this from happening, we can double the limit without any trouble. Simply set “server.max-fds” to 2048 in /etc/lighttpd/lighttpd.conf.

Contrary to much of the advice I found on the Internet, lighttpd spawned by root does not appear to use the “nofile” limits set in /etc/security/limits.conf, since these are for PAM and only apply to full interactive logins. There is a system-wide limit that can be set in /etc/sysctl.conf, however. Check your default with “cat /proc/sys/fs/file-max” and make sure it’s over 10,000. Mine was set to 12640 so I left that alone.

HTTP Keep-Alive

One reason that file descriptors get used up so quickly is HTTP keep-alive. To improve performance, modern web servers keep client connections alive to handle multiple requests instead of building up and tearing down connections for each item in a page. Keep-alive is tremendously beneficial to performance, but tends to keep unnecessary connections alive, too. By default, lighttpd allows 16 keep-alive requests per connection, allows idle sessions to remain alive for 5 seconds, and gives reads and writes 1 minute and 6 minutes to complete, respectively.

Although lighttpd has pretty aggressive defaults (especially compared to Apache), a period of heavy traffic and a few slow clients could see many unused connections sticking around. The server.max-keep-alive-idle setting default of 5 seconds can be reduced to as low as 2, if you assume your clients are reasonably quick about requesting data, but a value of 3 or 4 is probably realistic. You may want to increase the server.max-keep-alive-requests value from the default of 16, but you probably don’t need to. The server.max-read-idle and server.max-write-idle settings are tempting targets, but these situations are usually fairly rare so let’s not monkey with them.

Mechanics: Polling and Sending

The best bang for your lighttpd buck is to tune the server to use better kernel resources to check for file changes and write data to the network. There are three critical items here, each of which is set to a conservative universal setting by default.

One of the major areas of UNIX development over the last decade was how to handle the tens of thousands of connections experienced by Internet servers. This “C10K Problem” is documented in excruciating detail if you’re interested, but the net of it is that each version of UNIX has an advanced mechanism to handle I/O events. Since kernel version 2.6, Linux has sys_epoll, a so-called edge-triggered polling mechanism which scales linearly with the number of connections. But lighttpd runs on many different flavors of UNIX, so it has to default to the older and less-scalable “level-triggered” poll system. To remedy this, set “server.event-handler” to “linux-sysepoll”.

Another mechanism that varies widely across UNIX systems is how to actually read and write data from the disk to the network. All systems include basic read() and write() calls, which transfer data into and out of system memory. Lighttpd defaults to using these to move data around. But Linux includes a more advanced call, sendfile, which can move data around without copying it into memory. We can enable this by setting “server.network-backend” to “linux-sendfile”, which ought to improve performance for larger (multi-megabyte) files without impacting smaller ones.

Lighttpd attempts to improve performance further by caching the output of the UNIX stat() command. It includes a basic (“simple”) cache which keeps the result of file system calls in memory for one second. But many Linux distributions include more advanced accelerators: FAM was the original, and a lighter-weight workalike called Gamin is now included by default in Ubuntu’s lighttpd install. Therefore, we can improve stat calls simply by allowing lighttpd to use Gamin: Set “server.stat-cache-engine” to “fam” and you’re rolling!

One more useful tweak to consider, although it’s not included in the official lighttpd performance document, is not updating the “atime” parameter on served pages. This is a bit of a religious issue among some UNIX administrators, but I feel safe in saying that since my web server logs all accesses and I’m not using any kind of hierarchical storage system to store them, I don’t care when each php, html, and png file was last accessed. We can stop writing atime values by mounting the entire filesystem with “noatime”, but I like the more granular approach offered by lighttpd: Simply set “server.use-noatime” to “enable” and it won’t bother keeping this updated for the files it accesses. Everything else will continue as it always has but with reduced disk I/O.

Summary

Lighttpd has pretty good default settings, but a few might be tweaked if we need to respond to higher server loads. The more important area of tuning is simply enabling the advanced features of the 2.6.x Linux kernel and Ubuntu system we are using: Enable sys_epoll, sendfile, and Gamin and disable atime updates.

I’ll post more information as I stumble across it. I’m still learning, but my server performance as improved dramatically: Pingdom tools reports that it used to take upwards of half a minute to load my blog’s home page and it now loads in under seven seconds! That’s progress!

# Maximum number of file descriptors, default = 1024
server.max-fds = 2048
# Maximum number of request within a keep-alive session before the server terminates the connection, default = 16
server.max-keep-alive-requests = 16
# Maximum number of seconds until an idling keep-alive connection is dropped, default = 5
server.max-keep-alive-idle = 4
# Maximum number of seconds until a waiting, non keep-alive read times out and closes the connection, default = 60
server.max-read-idle = 60
# Maximum number of seconds until a waiting write call times out and closes the connection, default = 360
server.max-write-idle = 360
# Which event handler to use, default = poll
server.event-handler = "linux-sysepoll"
# How to handle network writes, default = writev
server.network-backend = "linux-sendfile"
# Requires FAM or Gamin to be installed, default = simple
server.stat-cache-engine = "fam"
# Whether to update the atime setting on file access, default = disable
server.use-noatime = "enable"

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Tuning Lighttpd For Linux
This post was categorized as Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This ought to be useful for those of us who have standardized our desktops on OS X yet still maintain a variety of old data storage systems and archives.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | AncientFS Opens Archaic Filesystems To Mac OS X
This post was categorized as Apple, Computer History. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.