The Easy Way

Apache is one amazingly flexible web server. It handles multiple domains with ease, using the VirtualHost method, and the ServerAlias directive allows you to host permutations easily. Consider the following totally made-up example:

<VirtualHost *:80>
  ServerName blog.fosketts.net
  ServerAlias www.blog.fosketts.net
  ServerAlias fosketts.net
  ServerAlias www.fosketts.net
  DocumentRoot /var/www/
</VirtualHost>

This looks great, right? It tells Apache to watch any IP on port 80 for an HTTP request for a server called blog.fosketts.net and to serve it the content from /var/www/. It also tells Apache to accept plain old “fosketts.net”, “www.fosketts.net”, and even ”www.blog.fosketts.net”.

What’s Wrong With Easy?

Although accepting all these hostnames seems like the friendly and correct thing to do, it’s not in your best interest. It tells web clients that the exact same content lives on four different servers, and they’ll start linking to your content every which way. Pretty soon you’ll have incoming links for all four hostnames. So what’s wrong with this?

1. It’s confusing for users – They’ll start asking, “is your site www.fosketts.net or blog.fosketts.net?” It’s fine to segment things, but confusing to do it unnecessarily.
2. It’s hard to configure and maintain – Once your site starts getting linked to and shared around, you’re stuck supporting all possible combinations. When you switch hosts or server platforms (ahem) you have to make sure everything still works.
3. It hurts your search ranking – You might not be all that concerned with search engine placement, and it’s not as bad as some say, but splitting your traffic between multiple sites also splits your “SEO juice”.
4. Web crawls overload your servers – Search engines treat each host name as a different server. If you allow links to multiple names without a proper redirect, you’ll get multiple crawls, often at the same time.

In summary, the easy was isn’t good. ServerAlias looks friendly, but it’s not a friend when used this way.

Let’s say your name was Stephen, but some people call you Steve. Rather than insist on one or the other, you could just go through life accepting either. But imprecision can lead to issues, even in the real world. Will people know to look up Stephen in the company directory when they know you as Steve? You might start getting duplicate junk mail for both names as they find their way onto mailing lists. Then there’s the embarrassing “I always called him Steve” moment at the company party, when someone feels like they’re not part of the “in crowd” that knows your real name. It’s best to be friendly and accept anything but politely suggest that everyone uses just one name in the interest of sanity.

Redirection is Right

The best approach in life is also the correct method on the web. Your server should be set to accept any number of possible names in case someone comes in with the wrong one. But rather than blithely accepting the name, your server should issue a proper “redirect” call, instructing the browser or crawler to reload the page using the correct name from that point on.

This is simple when using Lighttpd. I just added the following lines to my lighttpd.conf file and it magically issued a proper redirect whenever someone came in using the “www” name:

$HTTP["host"] =~ "^www\.(.*)$" {
  url.redirect  = (
    "^/(.*)" => "http://%1/$1",
  )
}

I was amazed that I could locate no such universal redirect option in Apache. You can do all the RedirectMatch calls you want, but their regular expressions only operate on the path part of the URL, not the hostname. This is great for adding a “www” but makes it impossible to create a generic rule to eliminate them!

Instead, we have to use RedirectMatch on each VirtualHost domain individually. This also opens the possibility to deal with other conditions we might come across, but it’s not as simple and clean as the Lighttpd method.

Here’s where the magic is. Each VirtualHost configuration you add (in /etc/apache2/sites-available on Ubuntu) should include rules to deal with the incorrect names as well as the single correct one. Here’s the correct redirect rule for the example above:

<VirtualHost *:80>
  ServerName fosketts.net
  ServerAlias www.fosketts.net
  ServerAlias www.blog.fosketts.net
  RedirectMatch 301 (.*) http://blog.fosketts.net$1
</VirtualHost>

<VirtualHost *:80>
  DocumentRoot /var/www/
  ServerName blog.fosketts.net
</VirtualHost>

The first VirtualHost block matches all the incorrect hostnames and redirects them (with a code of 301 for “Permanent”) to the correct hostname. The “(.*)” part matches any and all paths and arguments and the “$1″ part appends them to the new hostname. Then we set up another VirtualHost block for only the correct hostname and put any and all rules in there.

This way, any clients or crawlers that hit “www.fosketts.net” or any of the other alternatives will get a proper 301 redirect to “blog.fosketts.net” and go about its business. It tells Google that there is only one proper server name for this content and encourages users (who will likely copy and paste from the address bar) to use it, too. Neat and tidy, and very friendly.

I’d love to hear alternative methods of doing this. Please leave a comment if you have a suggestion that uses a 301 redirect and works across multiple domains!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | How To Force Apache To Redirect To Canonical Hostnames, or ServerAlias Is Not Your Friend
This post was categorized as Enterprise storage, Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

A Brief History of the Web

I’ve been pretty happy with lighttpd as my web server. It’s light and quick and has a workable fastcgi/PHP implementation. Right out of the box, lighttpd kicks Apache up and down the block on low-memory servers – the kind you’ll find in virtual private server hosting companies.

To understand why lighttpd is so good, we have to understand why Apache is so bad. And this means going way back in time to the dawn of the web. In the beginning, web servers were single-threaded processes that slapped pages over network connections on demand. This is like a single lemonade stand, where every customer waits in line but there’s only one item on offer and it’s a quick serve. So the line moves pretty quickly but it’s not all that interesting.

The Netscape Navigator browser revolutionized end-user experience by downloading web page elements in parallel and displaying them as they came in. Web servers responded by “forking” multiple copies of themselves to handle these requests (and those from other users) in parallel. Our lemonade stand is still limited in selection, but it now has more workers to hand over the juice.

At the same time, users began demanding more-interesting content. Web 2.0 replaced the old static HTML pages and graphics with interactive web applications, and the PHP language became dominant. Today, most popular web software (including WordPress and Mediawiki) require a PHP interpreter as well as an HTTP server to run. In the old days, server-side compute was handled by a “back room” CGI task. Our lemonade counter is now a restaurant, with a kitchen and added a variety of hot foods. But customers (and servers) have to wait while these are cooked up.

The Apache answer to this challenge was to add a kitchen to every checkout position. Out of the box, installing Apache and PHP adds a complete PHP runtime to each Apache process. And since the Apache PHP module is reputed to be unstable in multi-threaded environments, these default setups use the “prefork” single-thread-per-process method of handling multiple connections. Now our imaginary fast food restaurant can handle an unlimited number of customers in parallel, right?

In practice, this doesn’t work well. A 50-position counter could handle tons of customers, but that would be one wide restaurant! The same issue crops up with Apache. It looks super-speedy until loaded up (say, with a midnight Google, Yahoo, or MSN crawl) when it promptly uses up all of your memory forcing the OOM killer to murder your system. The only way to get this default (mod_php/mpm-prefork) version of Apache to be stable, other than adding RAM, is to limit the number of processes spawned. Customers are now back to waiting in line for just a few servers and their experience suffers.

Lighty Did It, Apache Didn’t?

Lighttpd (“lighty”) is totally different. It is a small, lightweight HTTP server that farms out PHP tasks to a constantly-running PHP process. This sounds pretty much exactly like the single-kitchen CGI approach of the mid-1990′s, but there’s a trick here (as there was then) that makes it work. In traditional CGI, the command processor was started on demand and closed when its work was done. Like a short-order cook, PHP-CGI makes everything to order.

But Netscape had a better idea. They built an always-running server (NSAPI) that could save work in a cache and reuse it for the next visitor. Suddenly we have a full kitchen up and running, complete with prep and sous chefs, able to turn orders around almost as quickly as the servers placed them. FastCGI was an open implementation of this concept, and it allows little lighttpd processes to serve up complex PHP web sites with ease.

Apache has a similar capability, using the modern FCGI implementation, and has also added multi-threaded server capability called mpm-worker. But the multi-threaded server is reputedly unstable when running PHP as a module, and configuring both mpm-worker and mod_fcgi isn’t yet standard-issue for Apache installs. So most Apache users still use the old prefork/mod_php configuration while most lighttpd users now rely on fastcgi.

Now, these FCGI PHP processes aren’t exactly light. Each can use 100 MB or more, especially when xcache is used for faster processing. So a lightweight server still needs to limit the number of “kitchens” serving the customers. And using a separate process to handle PHP isn’t as speedy as having an integrated PHP engine. But having more light HTTP servers to move files around reduces the overall load on the system, even if complicated tasks might have to wait for PHP processing.

Then there’s WP-SuperCache. This WordPress plugin converts complicated PHP pages into simple HTML for the HTTP server to hand off to visitors. In combination with a lightweight server like Apache/mpm-worker or Lighttpd, WP-SuperCache speeds up WordPress sites dramatically.

Although it’s possible to convince Lighttpd to play nicely with WP-SuperCache (using mod_magnet), it’s much easier and better-supported in Apache. I was also having an odd issue with Lighttpd “pausing” for a few seconds when clients connected it. So I decided to make the switch back to Apache.

Configuring Apache With MPM-Worker and Mod_FCGI/PHP-CGI

Now for the technical details. WP-SuperCache and Lighttpd might be poorly-documented, but Mod_FCGI in Apache isn’t much better. There are a dozen ways to do everything, and everyone’s recipe differs. Here’s how I got it to work.

First, I spun up a fresh 512 MB VPS on Slicehost running Ubuntu 10.04 Lucid. I configured it my normal way (locking down access, setting up a restrictive firewall, and installing only basic packages) and rsync-ed over my web site content.

One goal when tuning a server is to use all of the RAM but not much of the swap. Unused RAM is wasted potential, while swapping processes can quickly kill performance. This is one reason to run Apache with FCGI: You can run just a few heavy (large RAM footprint) PHP engines and many more light (3-5 MB) HTTP servers. Using the multi-threaded mpm-worker mode allows each of those Apache processes to serve more requests with less process creation and destruction.

My next step was to install Apache and its required modules along with mpm-worker, mod_fcgi, php5-cgi, and the rest needed to support wordpress. I think the following command should do the trick:

sudo apt-get install apache2-mpm-worker libapache2-mod_fcgi \
php5-cgi php5-xcache php5-mysql wordpress

This won’t work out of the box. We have to set up some wrapper scripts and configuration files for fcgi and php-cgi, but more important is the tuning, which we’ll get to next.

First, create a php-cgi wrapper for FCGI to use. I called mine /usr/local/bin/php-wrapper and set up a very basic environment. Be sure to make this script executable, too.

#!/bin/sh
# Set desired PHP_FCGI_* environment variables.
# Example:
# PHP FastCGI processes exit after 500 requests by default.
PHP_FCGI_MAX_REQUESTS=1000
export PHP_FCGI_MAX_REQUESTS
# DO NOT SET PHP_FCGI_CHILDREN!
# Replace with the path to your FastCGI-enabled PHP executable
exec /usr/bin/php-cgi

As the script comment says, whatever you do, do not set PHP_FCGI_CHILDREN in some misguided attempt to conserve RAM. It will conflict with mod_fcgi and you’ll end up spawning RAM-hungry but useless child processes and killing your system!

Next, create an fcgi configuration file called /etc/apache2/conf.d/php-fcgid.conf and containing something like the following lines:

FcgidInitialEnv PHPRC=/etc/php5/cgi
FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 1000
# FcgidMaxRequestsPerProcess should be <= PHP_FCGI_MAX_REQUESTS
# The example PHP wrapper script overrides the default PHP setting.
FcgidMaxRequestsPerProcess 1000
FcgidMaxProcesses 3
FcgidMaxProcessesPerClass 3
FcgidMinProcessesPerClass 1
# Uncomment the following line if cgi.fix_pathinfo is set to 1 in php.ini:
# FcgidFixPathinfo 1
# This makes php scripts work everywhere Apache serves
<Location />
AddHandler fcgid-script .php
Options +ExecCGI
FcgidWrapper /usr/local/bin/php-wrapper .php
# Customize the next two directives for your requirements.
Order allow,deny
Allow from all
</Location>

This is a very permissive configuration, and you might want to tweak things somewhat. But you get the general idea. The important things going on here are the configuration of FCGI to launch just 3 PHP-CGI engines and the assignment of the php-wrapper script to files ending in “php” anywhere Apache finds them. This means no special configuration is needed in VirtualHost directives, or anywhere else really.

One more tweak I like is to limit Apache to just 6 worker processes. Since each is multi-threaded and under 5 MB in this configuration, this works well. Add a line to /etc/apache2/apache2.conf in the “<IfModule mpm_worker_module>” section saying the following. This is a good complement to the default setting of 25 ThreadsPerChild and  150 MaxClients.

ServerLimit             6

Limiting the number of php-cgi processes launched is critical for low-memory systems. A good-sized opcode cache (thanks to xcache) helps them perform well but grows the memory usage like crazy. Although I had just three php-cgi processes running, with one topping 225 MB, overall system performance remained good thanks to WP-SuperCache and six multi-threaded lightweight Apache HTTP servers.

So far, I’m able to serve multiple WordPress and Mediawiki sites with a few thousand pageviews per day on a 512 MB slice. Running with six Apache workers and 3 or 4 php-cgi processes just gets me under the RAM limit. Of course, I’m still using a separate 256 MB server for MySQL in addition.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | A High-Performance, Low-Memory Apache/PHP Virtual Private Server
This post was categorized as Computer History, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I had switched from Dreamhost to Slicehost back in February to improve reliability and performance, but the meagre 256 MB of RAM in my virtual private server (VPS) “slice” proved insufficient. The time had come to completely redo my core hosting infrastructure. After some experimentation, I have settled on a simple two-server configuration based on Ubuntu Linux, MySQL, and lighttpd. I thought it would be a good idea to document this new configuration, as well as my previous experiments, for posterity.

If you’re interested in setting up a web site as outlined here, I recommend Slicehost. They’re not the cheapest, but their VPS servers are fast and reliable.

History

My web hosting environment has transitioned over the past six months. I had relied on shared hosting from Dreamhost for almost a decade, with my servers sharing infrastructure and management with thousands of others. This worked fine until I began to see significant traffic increases since creating this blog. Shared hosting just didn’t cut it once I had more than a few thousand pageviews per day.

I tried Dreamhost’s interesting and very flexible virtual private server capabilities, but could never get them working reliably. Plus, the core networking and storage performance of the Dreamhost infrastructure left something to be desired. After much research I switched to Slicehost, an all-VPS provider that has recently been acquired by Rackspace. Although they are not the cheapest or most flexible, Slicehost is a very professional service with good support and excellent infrastructure and connectivity.

I had been using a single 256 MB slice to host my entire site, and had managed to get everything well with lighttpd and MySQL, but this configuration ran into serious performance issues once traffic built again. Once I passed 10,000 pageviews per day, which happened quicker than I hoped, it was again time to upgrade.

Server Configuration

My core question was whether to go with a single 512MB or two 256 MB slices. Would resource contention in a single server be outweighed by the extra available RAM? After consulting with the experts, I decided that it was time to separate the database and web servers.

A multi-server setup delivers performance, reliability, and future capability.

As illustrated above, the database and web servers communicate via a high-speed private network, a standard Slicehost component. I created extremely restrictive iptables firewall policies to control access to both servers, disabling almost all communication. The database server in particular is inaccessible except from the web server, and even then only allows MySQL and public key ssh. Both servers are running bare-bones versions of Ubuntu 9.04.

MySQL Configuration

MySQL is much happier on its own dedicated server. It makes excellent use of its own query cache and operating system buffers and has very little disk access at all. Query performance immediately jumped, and site performance noticeably improved.

My only change to my.cnf was to enable the query cache. This works great on my 256 MB slice:

# Enable query cache
query_cache_limit       = 2M
query_cache_size        = 32M

I’m not a believer in security through obscurity. Sure, you can use a special high MySQL port, but by then you’ve probably lost anyway. Regardless, set up iptables to only allow access from your web server’s private interface using the -s parameter:

# Allow connections on our MySQL port
-A INPUT -p tcp --dport mysql -s 10.176.x.x -j ACCEPT

It’s a pretty simple configuration, but it works well. MySQL is humming along without much paging at all, using the entirety of the 256 MB available for caching. I haven’t had to resort to any tricks to keep the performance up.

One more configuration suggestion: Use a unique username and password for each database application and grant access only to that user on the private network interface. That way, your risk is segmented to a single database should an intruder use a SQL injection or something similar.

For example, say you were configuring the database “myblog_wp” for the user “me_myblog” with the password “123456abcdef”. You use the web server’s private interface address, 10.176.x.x as well so it is harder to get in even if your firewall is breached.

mysql -u root -p
Enter password:
mysql> grant all on myblog_wp.* to 'me_myblog'@'10.176.x.x' identified by '123456abcdef';

All set!

Basic Lighttpd Configuration

Back on the web server, we configure iptables to only allow connections to lighttpd on port 80 and ssh on whatever port you decide. Again, some suggest using a high port for ssh, but if you’ve configured sshd correctly then using an obscure port number is more likely to be a hassle for you than keep anyone out. Here are some good settings for /etc/ssh/sshd_config:

PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
X11Forwarding no
UsePAM no
UseDNS no

This forces all logins to use established public/private key pairs rather than passwords. Protect your keys and you’re in good shape. Key passphrases are a very good idea here! You’re also disallowing root in any case, since you’re using sudo for all administrative tasks, right?

Next we need to set up lighttpd. I’m using the basic packaged versions for ease of maintenance rather than building my own. I also loaded the stock versions of php5 and mysql-client which integrate nicely. I used to use eaccelerator but didn’t love having to recompile it myself and had some mysterious lockups with it running. So I went with XCache, which is developed by the same fine folks who created lighttpd.

sudo aptitude install mysql-client lighttpd php5-xcache

Why not use Apache? I actually tested Apache with my site before deciding to use lighttpd. I was able to tweak it to run in 256 MB of RAM by limiting the number of worker processes created to 4, but Apache just couldn’t handle the site load. Connections were stacking up and pageviews dropped as people gave up. Lighttpd with XCache is blindingly fast and fits in my 256 MB RAM envelope nicely.

The only real issue I have with lighttpd is that configuration is entirely different from Apache and remains less well-supported. It doesn’t use with .htaccess files, for example, and rewrites use a unique syntax.

Lighttpd for WordPress

Here’s a basic virtual server for /etc/lighttpd/lighttpd.conf for a WordPress domain:

$HTTP["host"] =~ "^(blog\.)?yourdomain\.com$" {
  server.name = "yourdomain.com"
  server.document-root = basedir + server.name + "/blog"
  url.access-deny = ( "wp-config.php" )
  dir-listing.activate = "disable"
  url.rewrite-final = (
    # Exclude some directories from rewriting
    "^/(wp-admin|wp-includes|wp-content)/(.*)" => "$0",
    # Uncomment to exclude Google FriendConnect files
    # "^/(canvas.html|rpc_relay.html)" => "$0",
    # Exclude .php, robots.txt, favicon.*, and sitemap.xml files at root from rewriting
    "^/(.*.php|sitemap.xml|robots.txt|favicon.*)" => "$0",
    # Handle WordPress permalinks and feeds
    "^/(.*)$" => "/index.php/$1"
  )
  accesslog.filename = basedir + "/" + server.name + "/log/blog.access.log"
}

This configuration accomplishes many of the important tasks of WordPress configuration. Most importantly, it works!

1. Permalinks and feeds are correctly redirected so no ugly index.php is used
2. Critical directories like wp-admin and files like robots.txt still work
3. Your wp-config.php file is explicitly protected
4. Directory listings are disabled
5. A site-specific access log is used (like Apache)

I’d love feedback on this configuration! One weird thing I’ve not yet figured out is how to use a site-specific error log. I haven’t configured pretty MediaWiki links in lighttpd yet, but imagine a similar configuration would work there.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Setting Up a Multi-Server Web Hosting Environment
This post was categorized as Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.