Setting Up a Multi-Server Web Hosting Environment

The last few weeks have been tough on my web servers. The release of iPhone OS 3.0 tripled my site traffic overnight as folks investigate the new Exchange integration features, and traffic to IT commentary site, Gestalt IT, which I also host, has been growing rapidly. Plus, Google just refreshed PageRank again, sending even more visitors my way.

I had switched from Dreamhost to Slicehost back in February to improve reliability and performance, but the meagre 256 MB of RAM in my virtual private server (VPS) “slice” proved insufficient. The time had come to completely redo my core hosting infrastructure. After some experimentation, I have settled on a simple two-server configuration based on Ubuntu Linux, MySQL, and lighttpd. I thought it would be a good idea to document this new configuration, as well as my previous experiments, for posterity.

If you’re interested in setting up a web site as outlined here, I recommend Slicehost. They’re not the cheapest, but their VPS servers are fast and reliable.

History

My web hosting environment has transitioned over the past six months. I had relied on shared hosting from Dreamhost for almost a decade, with my servers sharing infrastructure and management with thousands of others. This worked fine until I began to see significant traffic increases since creating this blog. Shared hosting just didn’t cut it once I had more than a few thousand pageviews per day.

I tried Dreamhost’s interesting and very flexible virtual private server capabilities, but could never get them working reliably. Plus, the core networking and storage performance of the Dreamhost infrastructure left something to be desired. After much research I switched to Slicehost, an all-VPS provider that has recently been acquired by Rackspace. Although they are not the cheapest or most flexible, Slicehost is a very professional service with good support and excellent infrastructure and connectivity.

I had been using a single 256 MB slice to host my entire site, and had managed to get everything well with lighttpd and MySQL, but this configuration ran into serious performance issues once traffic built again. Once I passed 10,000 pageviews per day, which happened quicker than I hoped, it was again time to upgrade.

Server Configuration

My core question was whether to go with a single 512MB or two 256 MB slices. Would resource contention in a single server be outweighed by the extra available RAM? After consulting with the experts, I decided that it was time to separate the database and web servers.

As illustrated above, the database and web servers communicate via a high-speed private network, a standard Slicehost component. I created extremely restrictive iptables firewall policies to control access to both servers, disabling almost all communication. The database server in particular is inaccessible except from the web server, and even then only allows MySQL and public key ssh. Both servers are running bare-bones versions of Ubuntu 9.04.

MySQL Configuration

MySQL is much happier on its own dedicated server. It makes excellent use of its own query cache and operating system buffers and has very little disk access at all. Query performance immediately jumped, and site performance noticeably improved.

My only change to my.cnf was to enable the query cache. This works great on my 256 MB slice:

# Enable query cache
query_cache_limit       = 2M
query_cache_size        = 32M

I’m not a believer in security through obscurity. Sure, you can use a special high MySQL port, but by then you’ve probably lost anyway. Regardless, set up iptables to only allow access from your web server’s private interface using the -s parameter:

# Allow connections on our MySQL port
-A INPUT -p tcp --dport mysql -s 10.176.x.x -j ACCEPT

It’s a pretty simple configuration, but it works well. MySQL is humming along without much paging at all, using the entirety of the 256 MB available for caching. I haven’t had to resort to any tricks to keep the performance up.

One more configuration suggestion: Use a unique username and password for each database application and grant access only to that user on the private network interface. That way, your risk is segmented to a single database should an intruder use a SQL injection or something similar.

For example, say you were configuring the database “myblog_wp” for the user “me_myblog” with the password “123456abcdef”. You use the web server’s private interface address, 10.176.x.x as well so it is harder to get in even if your firewall is breached.

mysql -u root -p
Enter password:
mysql> grant all on myblog_wp.* to 'me_myblog'@'10.176.x.x' identified by '123456abcdef';

All set!

Basic Lighttpd Configuration

Back on the web server, we configure iptables to only allow connections to lighttpd on port 80 and ssh on whatever port you decide. Again, some suggest using a high port for ssh, but if you’ve configured sshd correctly then using an obscure port number is more likely to be a hassle for you than keep anyone out. Here are some good settings for /etc/ssh/sshd_config:

PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
X11Forwarding no
UsePAM no
UseDNS no

This forces all logins to use established public/private key pairs rather than passwords. Protect your keys and you’re in good shape. Key passphrases are a very good idea here! You’re also disallowing root in any case, since you’re using sudo for all administrative tasks, right?

Next we need to set up lighttpd. I’m using the basic packaged versions for ease of maintenance rather than building my own. I also loaded the stock versions of php5 and mysql-client which integrate nicely. I used to use eaccelerator but didn’t love having to recompile it myself and had some mysterious lockups with it running. So I went with XCache, which is developed by the same fine folks who created lighttpd.

sudo aptitude install mysql-client lighttpd php5-xcache

Why not use Apache? I actually tested Apache with my site before deciding to use lighttpd. I was able to tweak it to run in 256 MB of RAM by limiting the number of worker processes created to 4, but Apache just couldn’t handle the site load. Connections were stacking up and pageviews dropped as people gave up. Lighttpd with XCache is blindingly fast and fits in my 256 MB RAM envelope nicely.

The only real issue I have with lighttpd is that configuration is entirely different from Apache and remains less well-supported. It doesn’t use with .htaccess files, for example, and rewrites use a unique syntax.

Lighttpd for WordPress

Here’s a basic virtual server for /etc/lighttpd/lighttpd.conf for a WordPress domain:

$HTTP["host"] =~ "^(blog\.)?yourdomain\.com$" {
  server.name = "yourdomain.com"
  server.document-root = basedir + server.name + "/blog"
  url.access-deny = ( "wp-config.php" )
  dir-listing.activate = "disable"
  url.rewrite-final = (
    # Exclude some directories from rewriting
    "^/(wp-admin|wp-includes|wp-content)/(.*)" => "$0",
    # Uncomment to exclude Google FriendConnect files
    # "^/(canvas.html|rpc_relay.html)" => "$0",
    # Exclude .php, robots.txt, favicon.*, and sitemap.xml files at root from rewriting
    "^/(.*.php|sitemap.xml|robots.txt|favicon.*)" => "$0",
    # Handle WordPress permalinks and feeds
    "^/(.*)$" => "/index.php/$1"
  )
  accesslog.filename = basedir + "/" + server.name + "/log/blog.access.log"
}

This configuration accomplishes many of the important tasks of WordPress configuration. Most importantly, it works!

1. Permalinks and feeds are correctly redirected so no ugly index.php is used
2. Critical directories like wp-admin and files like robots.txt still work
3. Your wp-config.php file is explicitly protected
4. Directory listings are disabled
5. A site-specific access log is used (like Apache)

I’d love feedback on this configuration! One weird thing I’ve not yet figured out is how to use a site-specific error log. I haven’t configured pretty MediaWiki links in lighttpd yet, but imagine a similar configuration would work there.