A High-Performance, Low-Memory Apache/PHP Virtual Private Server

I’ve previously written about my Multi-Server Web Hosting Environment and the how I Tuned Lighttpd For Linux to run as well as possible. But I’ve been having weird issues with that setup lately and was forced to rebuild the server entirely. While I was at it, I decided to give Apache another try, since the lighttpd hackery I needed to perform to get things like WP-SuperCache running were starting to get me down. The configuration documented here uses Ubuntu Lucid (10.04), Apache 2.2, FCGI, and PHP-CGI tuned for a 512 MB virtual private server (VPS) running on Rackspace Slicehost.

A Brief History of the Web

I’ve been pretty happy with lighttpd as my web server. It’s light and quick and has a workable fastcgi/PHP implementation. Right out of the box, lighttpd kicks Apache up and down the block on low-memory servers – the kind you’ll find in virtual private server hosting companies.

To understand why lighttpd is so good, we have to understand why Apache is so bad. And this means going way back in time to the dawn of the web. In the beginning, web servers were single-threaded processes that slapped pages over network connections on demand. This is like a single lemonade stand, where every customer waits in line but there’s only one item on offer and it’s a quick serve. So the line moves pretty quickly but it’s not all that interesting.

The Netscape Navigator browser revolutionized end-user experience by downloading web page elements in parallel and displaying them as they came in. Web servers responded by “forking” multiple copies of themselves to handle these requests (and those from other users) in parallel. Our lemonade stand is still limited in selection, but it now has more workers to hand over the juice.

At the same time, users began demanding more-interesting content. Web 2.0 replaced the old static HTML pages and graphics with interactive web applications, and the PHP language became dominant. Today, most popular web software (including WordPress and Mediawiki) require a PHP interpreter as well as an HTTP server to run. In the old days, server-side compute was handled by a “back room” CGI task. Our lemonade counter is now a restaurant, with a kitchen and added a variety of hot foods. But customers (and servers) have to wait while these are cooked up.

The Apache answer to this challenge was to add a kitchen to every checkout position. Out of the box, installing Apache and PHP adds a complete PHP runtime to each Apache process. And since the Apache PHP module is reputed to be unstable in multi-threaded environments, these default setups use the “prefork” single-thread-per-process method of handling multiple connections. Now our imaginary fast food restaurant can handle an unlimited number of customers in parallel, right?

In practice, this doesn’t work well. A 50-position counter could handle tons of customers, but that would be one wide restaurant! The same issue crops up with Apache. It looks super-speedy until loaded up (say, with a midnight Google, Yahoo, or MSN crawl) when it promptly uses up all of your memory forcing the OOM killer to murder your system. The only way to get this default (mod_php/mpm-prefork) version of Apache to be stable, other than adding RAM, is to limit the number of processes spawned. Customers are now back to waiting in line for just a few servers and their experience suffers.

Lighty Did It, Apache Didn’t?

Lighttpd (“lighty”) is totally different. It is a small, lightweight HTTP server that farms out PHP tasks to a constantly-running PHP process. This sounds pretty much exactly like the single-kitchen CGI approach of the mid-1990’s, but there’s a trick here (as there was then) that makes it work. In traditional CGI, the command processor was started on demand and closed when its work was done. Like a short-order cook, PHP-CGI makes everything to order.

But Netscape had a better idea. They built an always-running server (NSAPI) that could save work in a cache and reuse it for the next visitor. Suddenly we have a full kitchen up and running, complete with prep and sous chefs, able to turn orders around almost as quickly as the servers placed them. FastCGI was an open implementation of this concept, and it allows little lighttpd processes to serve up complex PHP web sites with ease.

Apache has a similar capability, using the modern FCGI implementation, and has also added multi-threaded server capability called mpm-worker. But the multi-threaded server is reputedly unstable when running PHP as a module, and configuring both mpm-worker and mod_fcgi isn’t yet standard-issue for Apache installs. So most Apache users still use the old prefork/mod_php configuration while most lighttpd users now rely on fastcgi.

Now, these FCGI PHP processes aren’t exactly light. Each can use 100 MB or more, especially when xcache is used for faster processing. So a lightweight server still needs to limit the number of “kitchens” serving the customers. And using a separate process to handle PHP isn’t as speedy as having an integrated PHP engine. But having more light HTTP servers to move files around reduces the overall load on the system, even if complicated tasks might have to wait for PHP processing.

Then there’s WP-SuperCache. This WordPress plugin converts complicated PHP pages into simple HTML for the HTTP server to hand off to visitors. In combination with a lightweight server like Apache/mpm-worker or Lighttpd, WP-SuperCache speeds up WordPress sites dramatically.

Although it’s possible to convince Lighttpd to play nicely with WP-SuperCache (using mod_magnet), it’s much easier and better-supported in Apache. I was also having an odd issue with Lighttpd “pausing” for a few seconds when clients connected it. So I decided to make the switch back to Apache.

Configuring Apache With MPM-Worker and Mod_FCGI/PHP-CGI

Now for the technical details. WP-SuperCache and Lighttpd might be poorly-documented, but Mod_FCGI in Apache isn’t much better. There are a dozen ways to do everything, and everyone’s recipe differs. Here’s how I got it to work.

First, I spun up a fresh 512 MB VPS on Slicehost running Ubuntu 10.04 Lucid. I configured it my normal way (locking down access, setting up a restrictive firewall, and installing only basic packages) and rsync-ed over my web site content.

One goal when tuning a server is to use all of the RAM but not much of the swap. Unused RAM is wasted potential, while swapping processes can quickly kill performance. This is one reason to run Apache with FCGI: You can run just a few heavy (large RAM footprint) PHP engines and many more light (3-5 MB) HTTP servers. Using the multi-threaded mpm-worker mode allows each of those Apache processes to serve more requests with less process creation and destruction.

My next step was to install Apache and its required modules along with mpm-worker, mod_fcgi, php5-cgi, and the rest needed to support wordpress. I think the following command should do the trick:

sudo apt-get install apache2-mpm-worker libapache2-mod_fcgi \
php5-cgi php5-xcache php5-mysql wordpress

This won’t work out of the box. We have to set up some wrapper scripts and configuration files for fcgi and php-cgi, but more important is the tuning, which we’ll get to next.

First, create a php-cgi wrapper for FCGI to use. I called mine /usr/local/bin/php-wrapper and set up a very basic environment. Be sure to make this script executable, too.

#!/bin/sh
# Set desired PHP_FCGI_* environment variables.
# Example:
# PHP FastCGI processes exit after 500 requests by default.
PHP_FCGI_MAX_REQUESTS=1000
export PHP_FCGI_MAX_REQUESTS
# DO NOT SET PHP_FCGI_CHILDREN!
# Replace with the path to your FastCGI-enabled PHP executable
exec /usr/bin/php-cgi

As the script comment says, whatever you do, do not set PHP_FCGI_CHILDREN in some misguided attempt to conserve RAM. It will conflict with mod_fcgi and you’ll end up spawning RAM-hungry but useless child processes and killing your system!

Next, create an fcgi configuration file called /etc/apache2/conf.d/php-fcgid.conf and containing something like the following lines:

FcgidInitialEnv PHPRC=/etc/php5/cgi
FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 1000
# FcgidMaxRequestsPerProcess should be <= PHP_FCGI_MAX_REQUESTS
# The example PHP wrapper script overrides the default PHP setting.
FcgidMaxRequestsPerProcess 1000
FcgidMaxProcesses 3
FcgidMaxProcessesPerClass 3
FcgidMinProcessesPerClass 1
# Uncomment the following line if cgi.fix_pathinfo is set to 1 in php.ini:
# FcgidFixPathinfo 1
# This makes php scripts work everywhere Apache serves
<Location />
AddHandler fcgid-script .php
Options +ExecCGI
FcgidWrapper /usr/local/bin/php-wrapper .php
# Customize the next two directives for your requirements.
Order allow,deny
Allow from all
</Location>

This is a very permissive configuration, and you might want to tweak things somewhat. But you get the general idea. The important things going on here are the configuration of FCGI to launch just 3 PHP-CGI engines and the assignment of the php-wrapper script to files ending in “php” anywhere Apache finds them. This means no special configuration is needed in VirtualHost directives, or anywhere else really.

One more tweak I like is to limit Apache to just 6 worker processes. Since each is multi-threaded and under 5 MB in this configuration, this works well. Add a line to /etc/apache2/apache2.conf in the “<IfModule mpm_worker_module>” section saying the following. This is a good complement to the default setting of 25 ThreadsPerChild and  150 MaxClients.

ServerLimit             6

Limiting the number of php-cgi processes launched is critical for low-memory systems. A good-sized opcode cache (thanks to xcache) helps them perform well but grows the memory usage like crazy. Although I had just three php-cgi processes running, with one topping 225 MB, overall system performance remained good thanks to WP-SuperCache and six multi-threaded lightweight Apache HTTP servers.

So far, I’m able to serve multiple WordPress and Mediawiki sites with a few thousand pageviews per day on a 512 MB slice. Running with six Apache workers and 3 or 4 php-cgi processes just gets me under the RAM limit. Of course, I’m still using a separate 256 MB server for MySQL in addition.