A Brief History of the Web

I’ve been pretty happy with lighttpd as my web server. It’s light and quick and has a workable fastcgi/PHP implementation. Right out of the box, lighttpd kicks Apache up and down the block on low-memory servers – the kind you’ll find in virtual private server hosting companies.

To understand why lighttpd is so good, we have to understand why Apache is so bad. And this means going way back in time to the dawn of the web. In the beginning, web servers were single-threaded processes that slapped pages over network connections on demand. This is like a single lemonade stand, where every customer waits in line but there’s only one item on offer and it’s a quick serve. So the line moves pretty quickly but it’s not all that interesting.

The Netscape Navigator browser revolutionized end-user experience by downloading web page elements in parallel and displaying them as they came in. Web servers responded by “forking” multiple copies of themselves to handle these requests (and those from other users) in parallel. Our lemonade stand is still limited in selection, but it now has more workers to hand over the juice.

At the same time, users began demanding more-interesting content. Web 2.0 replaced the old static HTML pages and graphics with interactive web applications, and the PHP language became dominant. Today, most popular web software (including WordPress and Mediawiki) require a PHP interpreter as well as an HTTP server to run. In the old days, server-side compute was handled by a “back room” CGI task. Our lemonade counter is now a restaurant, with a kitchen and added a variety of hot foods. But customers (and servers) have to wait while these are cooked up.

The Apache answer to this challenge was to add a kitchen to every checkout position. Out of the box, installing Apache and PHP adds a complete PHP runtime to each Apache process. And since the Apache PHP module is reputed to be unstable in multi-threaded environments, these default setups use the “prefork” single-thread-per-process method of handling multiple connections. Now our imaginary fast food restaurant can handle an unlimited number of customers in parallel, right?

In practice, this doesn’t work well. A 50-position counter could handle tons of customers, but that would be one wide restaurant! The same issue crops up with Apache. It looks super-speedy until loaded up (say, with a midnight Google, Yahoo, or MSN crawl) when it promptly uses up all of your memory forcing the OOM killer to murder your system. The only way to get this default (mod_php/mpm-prefork) version of Apache to be stable, other than adding RAM, is to limit the number of processes spawned. Customers are now back to waiting in line for just a few servers and their experience suffers.

Lighty Did It, Apache Didn’t?

Lighttpd (“lighty”) is totally different. It is a small, lightweight HTTP server that farms out PHP tasks to a constantly-running PHP process. This sounds pretty much exactly like the single-kitchen CGI approach of the mid-1990′s, but there’s a trick here (as there was then) that makes it work. In traditional CGI, the command processor was started on demand and closed when its work was done. Like a short-order cook, PHP-CGI makes everything to order.

But Netscape had a better idea. They built an always-running server (NSAPI) that could save work in a cache and reuse it for the next visitor. Suddenly we have a full kitchen up and running, complete with prep and sous chefs, able to turn orders around almost as quickly as the servers placed them. FastCGI was an open implementation of this concept, and it allows little lighttpd processes to serve up complex PHP web sites with ease.

Apache has a similar capability, using the modern FCGI implementation, and has also added multi-threaded server capability called mpm-worker. But the multi-threaded server is reputedly unstable when running PHP as a module, and configuring both mpm-worker and mod_fcgi isn’t yet standard-issue for Apache installs. So most Apache users still use the old prefork/mod_php configuration while most lighttpd users now rely on fastcgi.

Now, these FCGI PHP processes aren’t exactly light. Each can use 100 MB or more, especially when xcache is used for faster processing. So a lightweight server still needs to limit the number of “kitchens” serving the customers. And using a separate process to handle PHP isn’t as speedy as having an integrated PHP engine. But having more light HTTP servers to move files around reduces the overall load on the system, even if complicated tasks might have to wait for PHP processing.

Then there’s WP-SuperCache. This WordPress plugin converts complicated PHP pages into simple HTML for the HTTP server to hand off to visitors. In combination with a lightweight server like Apache/mpm-worker or Lighttpd, WP-SuperCache speeds up WordPress sites dramatically.

Although it’s possible to convince Lighttpd to play nicely with WP-SuperCache (using mod_magnet), it’s much easier and better-supported in Apache. I was also having an odd issue with Lighttpd “pausing” for a few seconds when clients connected it. So I decided to make the switch back to Apache.

Configuring Apache With MPM-Worker and Mod_FCGI/PHP-CGI

Now for the technical details. WP-SuperCache and Lighttpd might be poorly-documented, but Mod_FCGI in Apache isn’t much better. There are a dozen ways to do everything, and everyone’s recipe differs. Here’s how I got it to work.

First, I spun up a fresh 512 MB VPS on Slicehost running Ubuntu 10.04 Lucid. I configured it my normal way (locking down access, setting up a restrictive firewall, and installing only basic packages) and rsync-ed over my web site content.

One goal when tuning a server is to use all of the RAM but not much of the swap. Unused RAM is wasted potential, while swapping processes can quickly kill performance. This is one reason to run Apache with FCGI: You can run just a few heavy (large RAM footprint) PHP engines and many more light (3-5 MB) HTTP servers. Using the multi-threaded mpm-worker mode allows each of those Apache processes to serve more requests with less process creation and destruction.

My next step was to install Apache and its required modules along with mpm-worker, mod_fcgi, php5-cgi, and the rest needed to support wordpress. I think the following command should do the trick:

sudo apt-get install apache2-mpm-worker libapache2-mod_fcgi \
php5-cgi php5-xcache php5-mysql wordpress

This won’t work out of the box. We have to set up some wrapper scripts and configuration files for fcgi and php-cgi, but more important is the tuning, which we’ll get to next.

First, create a php-cgi wrapper for FCGI to use. I called mine /usr/local/bin/php-wrapper and set up a very basic environment. Be sure to make this script executable, too.

#!/bin/sh
# Set desired PHP_FCGI_* environment variables.
# Example:
# PHP FastCGI processes exit after 500 requests by default.
PHP_FCGI_MAX_REQUESTS=1000
export PHP_FCGI_MAX_REQUESTS
# DO NOT SET PHP_FCGI_CHILDREN!
# Replace with the path to your FastCGI-enabled PHP executable
exec /usr/bin/php-cgi

As the script comment says, whatever you do, do not set PHP_FCGI_CHILDREN in some misguided attempt to conserve RAM. It will conflict with mod_fcgi and you’ll end up spawning RAM-hungry but useless child processes and killing your system!

Next, create an fcgi configuration file called /etc/apache2/conf.d/php-fcgid.conf and containing something like the following lines:

FcgidInitialEnv PHPRC=/etc/php5/cgi
FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 1000
# FcgidMaxRequestsPerProcess should be <= PHP_FCGI_MAX_REQUESTS
# The example PHP wrapper script overrides the default PHP setting.
FcgidMaxRequestsPerProcess 1000
FcgidMaxProcesses 3
FcgidMaxProcessesPerClass 3
FcgidMinProcessesPerClass 1
# Uncomment the following line if cgi.fix_pathinfo is set to 1 in php.ini:
# FcgidFixPathinfo 1
# This makes php scripts work everywhere Apache serves
<Location />
AddHandler fcgid-script .php
Options +ExecCGI
FcgidWrapper /usr/local/bin/php-wrapper .php
# Customize the next two directives for your requirements.
Order allow,deny
Allow from all
</Location>

This is a very permissive configuration, and you might want to tweak things somewhat. But you get the general idea. The important things going on here are the configuration of FCGI to launch just 3 PHP-CGI engines and the assignment of the php-wrapper script to files ending in “php” anywhere Apache finds them. This means no special configuration is needed in VirtualHost directives, or anywhere else really.

One more tweak I like is to limit Apache to just 6 worker processes. Since each is multi-threaded and under 5 MB in this configuration, this works well. Add a line to /etc/apache2/apache2.conf in the “<IfModule mpm_worker_module>” section saying the following. This is a good complement to the default setting of 25 ThreadsPerChild and  150 MaxClients.

ServerLimit             6

Limiting the number of php-cgi processes launched is critical for low-memory systems. A good-sized opcode cache (thanks to xcache) helps them perform well but grows the memory usage like crazy. Although I had just three php-cgi processes running, with one topping 225 MB, overall system performance remained good thanks to WP-SuperCache and six multi-threaded lightweight Apache HTTP servers.

So far, I’m able to serve multiple WordPress and Mediawiki sites with a few thousand pageviews per day on a 512 MB slice. Running with six Apache workers and 3 or 4 php-cgi processes just gets me under the RAM limit. Of course, I’m still using a separate 256 MB server for MySQL in addition.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | A High-Performance, Low-Memory Apache/PHP Virtual Private Server
This post was categorized as Computer History, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Check out these detailed posts on cloud storage I wrote for my other blog, Enterprise Storage Strategies:

• Cloud Storage, Storage in the Cloud, and Cloudy Storage Systems
• What Makes Cloud Storage Different from Traditional SAN and NAS?

Eventually, it dawned on me: There’s a big difference between real cloudstuff and plain old stuff in the cloud!

Lots of cloud computing offerings are startlingly conventional. They’re plain-jane IT infrastructure just like we all have today: Virtual servers, storage, and databases. Strip away the management API and self-service model and Amazon EC2 and EBS looks an awful lot like the Xen-based virtual server infrastructure you might find at any old IT shop. The same goes for Rackspace’s Mosso Cloud Servers: They’re extremely similar to Rackspace’s Slicehost virtual private servers!

This isn’t all bad, of course. As I discussed with EMC’s Barry Burke last week, you can theoretically run your ERP application on EC2 without major gyrations. Try that with Microsoft Azure or Google App Engine! And the management layers, especially those from companies like RightScale, turn these run-of-the-mill parts into something really extraordinary! VMware’s vCloud concept really hammers home this evolution-not-revolution mindset.

Real cloudstuff is completely different. Comparing a Xen instance running Linux on some disk (a-la EC2) to a programmable platform like Azure is problematic. Just about the only thing they have in common (apart from the cloud name) is the fact that they’re hosted on multi-tenant servers and offered to the public on a pay-per-usage model. Cloudstuff is the IT revolution that application developers have dreamed of!

Of course, the problem with whole-cloth reinvention is that it’s slow to take hold. Although net-new apps can be built to take advantage of full-on cloud infrastructure today, it will literally be a decade before the corporate IT applications we all rely on will run there. The early adopters will be companies like Microsoft and Google, who have a vested interest in seeing the concept succeed and the development muscle to make it happen.

Then there are the bridges between today’s world and this cloudstuff future. Consider applications like Nirvanix CloudNAS and Jungle Disk: They hide the complexity of API-driven cloud storage behind the familiar face of file server or backup application. Once the data is loaded, cloud-aware applications can access it. This is where the magic happens!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | CloudStuff Versus Stuff in the Cloud
This post was categorized as Computer History, Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I had switched from Dreamhost to Slicehost back in February to improve reliability and performance, but the meagre 256 MB of RAM in my virtual private server (VPS) “slice” proved insufficient. The time had come to completely redo my core hosting infrastructure. After some experimentation, I have settled on a simple two-server configuration based on Ubuntu Linux, MySQL, and lighttpd. I thought it would be a good idea to document this new configuration, as well as my previous experiments, for posterity.

If you’re interested in setting up a web site as outlined here, I recommend Slicehost. They’re not the cheapest, but their VPS servers are fast and reliable.

History

My web hosting environment has transitioned over the past six months. I had relied on shared hosting from Dreamhost for almost a decade, with my servers sharing infrastructure and management with thousands of others. This worked fine until I began to see significant traffic increases since creating this blog. Shared hosting just didn’t cut it once I had more than a few thousand pageviews per day.

I tried Dreamhost’s interesting and very flexible virtual private server capabilities, but could never get them working reliably. Plus, the core networking and storage performance of the Dreamhost infrastructure left something to be desired. After much research I switched to Slicehost, an all-VPS provider that has recently been acquired by Rackspace. Although they are not the cheapest or most flexible, Slicehost is a very professional service with good support and excellent infrastructure and connectivity.

I had been using a single 256 MB slice to host my entire site, and had managed to get everything well with lighttpd and MySQL, but this configuration ran into serious performance issues once traffic built again. Once I passed 10,000 pageviews per day, which happened quicker than I hoped, it was again time to upgrade.

Server Configuration

My core question was whether to go with a single 512MB or two 256 MB slices. Would resource contention in a single server be outweighed by the extra available RAM? After consulting with the experts, I decided that it was time to separate the database and web servers.

A multi-server setup delivers performance, reliability, and future capability.

As illustrated above, the database and web servers communicate via a high-speed private network, a standard Slicehost component. I created extremely restrictive iptables firewall policies to control access to both servers, disabling almost all communication. The database server in particular is inaccessible except from the web server, and even then only allows MySQL and public key ssh. Both servers are running bare-bones versions of Ubuntu 9.04.

MySQL Configuration

MySQL is much happier on its own dedicated server. It makes excellent use of its own query cache and operating system buffers and has very little disk access at all. Query performance immediately jumped, and site performance noticeably improved.

My only change to my.cnf was to enable the query cache. This works great on my 256 MB slice:

# Enable query cache
query_cache_limit       = 2M
query_cache_size        = 32M

I’m not a believer in security through obscurity. Sure, you can use a special high MySQL port, but by then you’ve probably lost anyway. Regardless, set up iptables to only allow access from your web server’s private interface using the -s parameter:

# Allow connections on our MySQL port
-A INPUT -p tcp --dport mysql -s 10.176.x.x -j ACCEPT

It’s a pretty simple configuration, but it works well. MySQL is humming along without much paging at all, using the entirety of the 256 MB available for caching. I haven’t had to resort to any tricks to keep the performance up.

One more configuration suggestion: Use a unique username and password for each database application and grant access only to that user on the private network interface. That way, your risk is segmented to a single database should an intruder use a SQL injection or something similar.

For example, say you were configuring the database “myblog_wp” for the user “me_myblog” with the password “123456abcdef”. You use the web server’s private interface address, 10.176.x.x as well so it is harder to get in even if your firewall is breached.

mysql -u root -p
Enter password:
mysql> grant all on myblog_wp.* to 'me_myblog'@'10.176.x.x' identified by '123456abcdef';

All set!

Basic Lighttpd Configuration

Back on the web server, we configure iptables to only allow connections to lighttpd on port 80 and ssh on whatever port you decide. Again, some suggest using a high port for ssh, but if you’ve configured sshd correctly then using an obscure port number is more likely to be a hassle for you than keep anyone out. Here are some good settings for /etc/ssh/sshd_config:

PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
X11Forwarding no
UsePAM no
UseDNS no

This forces all logins to use established public/private key pairs rather than passwords. Protect your keys and you’re in good shape. Key passphrases are a very good idea here! You’re also disallowing root in any case, since you’re using sudo for all administrative tasks, right?

Next we need to set up lighttpd. I’m using the basic packaged versions for ease of maintenance rather than building my own. I also loaded the stock versions of php5 and mysql-client which integrate nicely. I used to use eaccelerator but didn’t love having to recompile it myself and had some mysterious lockups with it running. So I went with XCache, which is developed by the same fine folks who created lighttpd.

sudo aptitude install mysql-client lighttpd php5-xcache

Why not use Apache? I actually tested Apache with my site before deciding to use lighttpd. I was able to tweak it to run in 256 MB of RAM by limiting the number of worker processes created to 4, but Apache just couldn’t handle the site load. Connections were stacking up and pageviews dropped as people gave up. Lighttpd with XCache is blindingly fast and fits in my 256 MB RAM envelope nicely.

The only real issue I have with lighttpd is that configuration is entirely different from Apache and remains less well-supported. It doesn’t use with .htaccess files, for example, and rewrites use a unique syntax.

Lighttpd for WordPress

Here’s a basic virtual server for /etc/lighttpd/lighttpd.conf for a WordPress domain:

$HTTP["host"] =~ "^(blog\.)?yourdomain\.com$" {
  server.name = "yourdomain.com"
  server.document-root = basedir + server.name + "/blog"
  url.access-deny = ( "wp-config.php" )
  dir-listing.activate = "disable"
  url.rewrite-final = (
    # Exclude some directories from rewriting
    "^/(wp-admin|wp-includes|wp-content)/(.*)" => "$0",
    # Uncomment to exclude Google FriendConnect files
    # "^/(canvas.html|rpc_relay.html)" => "$0",
    # Exclude .php, robots.txt, favicon.*, and sitemap.xml files at root from rewriting
    "^/(.*.php|sitemap.xml|robots.txt|favicon.*)" => "$0",
    # Handle WordPress permalinks and feeds
    "^/(.*)$" => "/index.php/$1"
  )
  accesslog.filename = basedir + "/" + server.name + "/log/blog.access.log"
}

This configuration accomplishes many of the important tasks of WordPress configuration. Most importantly, it works!

1. Permalinks and feeds are correctly redirected so no ugly index.php is used
2. Critical directories like wp-admin and files like robots.txt still work
3. Your wp-config.php file is explicitly protected
4. Directory listings are disabled
5. A site-specific access log is used (like Apache)

I’d love feedback on this configuration! One weird thing I’ve not yet figured out is how to use a site-specific error log. I haven’t configured pretty MediaWiki links in lighttpd yet, but imagine a similar configuration would work there.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Setting Up a Multi-Server Web Hosting Environment
This post was categorized as Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Dreamhost never could get their flexible resource provisioning control panel to work right with my server, and their support staff was only slightly responsive. I limped along for three months, waiting for them to get everything working right and suffering through two unexplained multi-hour server outages. After losing my server for an hour and a half yesterday without any response or explanation whatsoever from Dreamhost support, I decided that enough was enough. After nine years as a customer, I am moving my sites elsewhere.

I looked around for a rock-solid Xen VPS/Linux provider with high performance and affordable prices. After checking into many that were recommended, I narrowed my search to two: Linode and Slicehost. Both differ from Dreamhost in that they offer unconfigured servers rather than point-and-click convenience and management. But after almost 20 years in UNIX administration, I figured I was up to the job.

After examining the two, I decided that Slicehost, which was recently acquired by Rackspace, offered the best solution for me. One deciding factor was Linode’s failure to offer backup services of any kind – that’s just bizarre! When all is done, I will be spending just $5 more per month than my Dreamhost VPS cost, and will hopefully get much better performance and stability.

If you’re interested in setting up a web site as outlined here, I recommend Slicehost. They’re not the cheapest, but their VPS servers are fast and reliable.

So I spent much of Friday evening setting up Ubuntu on my slice, copying data, configuring DNS, and installing lighttpd, php, MySQL, WordPress, Mediawiki, and the rest. After some teething issues, I believe that everything is now up and running properly on my new host! Please let me know (by clicking the “Email Me” link in the upper left corner) if you see any errors or issues!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Apologies For The 404s!
This post was categorized as Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.