The Easy Way

Apache is one amazingly flexible web server. It handles multiple domains with ease, using the VirtualHost method, and the ServerAlias directive allows you to host permutations easily. Consider the following totally made-up example:

<VirtualHost *:80>
  ServerName blog.fosketts.net
  ServerAlias www.blog.fosketts.net
  ServerAlias fosketts.net
  ServerAlias www.fosketts.net
  DocumentRoot /var/www/
</VirtualHost>

This looks great, right? It tells Apache to watch any IP on port 80 for an HTTP request for a server called blog.fosketts.net and to serve it the content from /var/www/. It also tells Apache to accept plain old “fosketts.net”, “www.fosketts.net”, and even ”www.blog.fosketts.net”.

What’s Wrong With Easy?

Although accepting all these hostnames seems like the friendly and correct thing to do, it’s not in your best interest. It tells web clients that the exact same content lives on four different servers, and they’ll start linking to your content every which way. Pretty soon you’ll have incoming links for all four hostnames. So what’s wrong with this?

1. It’s confusing for users – They’ll start asking, “is your site www.fosketts.net or blog.fosketts.net?” It’s fine to segment things, but confusing to do it unnecessarily.
2. It’s hard to configure and maintain – Once your site starts getting linked to and shared around, you’re stuck supporting all possible combinations. When you switch hosts or server platforms (ahem) you have to make sure everything still works.
3. It hurts your search ranking – You might not be all that concerned with search engine placement, and it’s not as bad as some say, but splitting your traffic between multiple sites also splits your “SEO juice”.
4. Web crawls overload your servers – Search engines treat each host name as a different server. If you allow links to multiple names without a proper redirect, you’ll get multiple crawls, often at the same time.

In summary, the easy was isn’t good. ServerAlias looks friendly, but it’s not a friend when used this way.

Let’s say your name was Stephen, but some people call you Steve. Rather than insist on one or the other, you could just go through life accepting either. But imprecision can lead to issues, even in the real world. Will people know to look up Stephen in the company directory when they know you as Steve? You might start getting duplicate junk mail for both names as they find their way onto mailing lists. Then there’s the embarrassing “I always called him Steve” moment at the company party, when someone feels like they’re not part of the “in crowd” that knows your real name. It’s best to be friendly and accept anything but politely suggest that everyone uses just one name in the interest of sanity.

Redirection is Right

The best approach in life is also the correct method on the web. Your server should be set to accept any number of possible names in case someone comes in with the wrong one. But rather than blithely accepting the name, your server should issue a proper “redirect” call, instructing the browser or crawler to reload the page using the correct name from that point on.

This is simple when using Lighttpd. I just added the following lines to my lighttpd.conf file and it magically issued a proper redirect whenever someone came in using the “www” name:

$HTTP["host"] =~ "^www\.(.*)$" {
  url.redirect  = (
    "^/(.*)" => "http://%1/$1",
  )
}

I was amazed that I could locate no such universal redirect option in Apache. You can do all the RedirectMatch calls you want, but their regular expressions only operate on the path part of the URL, not the hostname. This is great for adding a “www” but makes it impossible to create a generic rule to eliminate them!

Instead, we have to use RedirectMatch on each VirtualHost domain individually. This also opens the possibility to deal with other conditions we might come across, but it’s not as simple and clean as the Lighttpd method.

Here’s where the magic is. Each VirtualHost configuration you add (in /etc/apache2/sites-available on Ubuntu) should include rules to deal with the incorrect names as well as the single correct one. Here’s the correct redirect rule for the example above:

<VirtualHost *:80>
  ServerName fosketts.net
  ServerAlias www.fosketts.net
  ServerAlias www.blog.fosketts.net
  RedirectMatch 301 (.*) http://blog.fosketts.net$1
</VirtualHost>

<VirtualHost *:80>
  DocumentRoot /var/www/
  ServerName blog.fosketts.net
</VirtualHost>

The first VirtualHost block matches all the incorrect hostnames and redirects them (with a code of 301 for “Permanent”) to the correct hostname. The “(.*)” part matches any and all paths and arguments and the “$1″ part appends them to the new hostname. Then we set up another VirtualHost block for only the correct hostname and put any and all rules in there.

This way, any clients or crawlers that hit “www.fosketts.net” or any of the other alternatives will get a proper 301 redirect to “blog.fosketts.net” and go about its business. It tells Google that there is only one proper server name for this content and encourages users (who will likely copy and paste from the address bar) to use it, too. Neat and tidy, and very friendly.

I’d love to hear alternative methods of doing this. Please leave a comment if you have a suggestion that uses a 301 redirect and works across multiple domains!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | How To Force Apache To Redirect To Canonical Hostnames, or ServerAlias Is Not Your Friend
This post was categorized as Enterprise storage, Everything. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Note: This post is about Google, because it is by far the dominant search engine, advertiser, and “portal” in the English-speaking world. Nearly everything mentioned here applies equally to other search engines and advertising providers.

Ranking Pages

Google’s relevance comes from their historical ability to present a quality searchable portal to the entire Internet. The majority of Google’s revenue is also derived from quality information, giving them the ability to present more-compelling advertising to web users.

Google’s core success is based on its ability to discover and rank the quality of Internet content. Gmail, Reader, Picasa, Apps, and the rest of the Google properties are surely excellent sources of information on the preferences of individual users, but they contribute only slightly to the other side of the coin: Information about Internet content. For that, they still rely on the core technology invented at Stanford a decade ago: PageRank.

Every time it encounters a link, Google’s software “spider” follows it, adding the content of the linked web page to an index. Google, like other early search engines, counts each link as a vote for the quality of the page. The genius of PageRank is that Google weights each vote based on the quality of the page it comes from. Although PageRank is not the entirety of Google, it is a singular key element.

Put simply, Google’s success depends on its ability to gather and rank the links we all make and match them to the data we provide about ourselves. Without this, Google will fail.

The Changing Web

The graphical Web is not the Internet. My first experiences online came well before graphical hypertext clients (what we now call browsers) dominated the user experience and became the web. Although the network we call the Internet now supports a very wide variety of traffic, Google’s preeminence comes only from the Web. They have little or no reach into the massive streams of corporate data, multimedia, and other non-hypertext content streaming across the ‘net.

When it was first developed, the web was manual and links were hand-selected and carefully put into context. It was difficult to put together a web page, and those pages that were developed were were static. The social networks of the time (USENET, IRC, and email mostly) were not integrated into the web, did not generally include links. So the first search engines, and later ones like Google, focused on this relatively small pool of pages and links.

But the web soon became automated, subsuming most other interactive services. Social (user-generated) interaction moved into the web in a big way, with blogs, wikis, and discussion forums enabling rapid content creation and reference by users. Sharing links in the social web, and through social bookmarking services, generally replaced the manual pages of old.

At first, this explosion of user-generated content was a dream scenario for Google. They could harvest the collective intelligence of us all to identify and rank content. But as the number of pages and links exploded, the notion of a “web page” was radically shifted from a stable and predictable set of data to a dynamic portal into a vast store of content. Where everyone once saw the same content at a given URL, now each of us has his own experience.

Spammers and scammers realized the value of Google placement and flooded this dynamic social web with links. This threatened not only to undermine the relevance that supports Google’s search (and advertising) business, but it also threatened these new social services themselves. Each honest, relevant link added to a Wikipedia article, included in a Slashdot comment, or shared on a service like Digg was dwarfed by the thousands or millions of spam links injected to boost the PageRank of “client” sites.

I Don’t Follow

Google and the social net fought valiantly against this wave of link spam, but it became clear that something more radical was needed. The only way to fight spam was to make it useless to the spammers. Thus was born a simple but highly-effective tool: Nofollow.

Webmasters long had the ability to tell the Google spider to ignore a certain set of hosted pages through the use of a server-side list called robots.txt. But spammers wanted the exact opposite. What was needed was a client-side way to specify that a link was not worthy of being spidered and ranked by the search engines. This would eliminate the primary benefit of link spam.

Implementing client-side spider blocking was trivial: A simple tag, “rel=nofollow”, was added alongside the url in a web link. This way, Google’s spider would simply ignore every “nofollow” link it encountered, and they would never be searched or ranked in the index.

But spammers would never put the nofollow tag in their own links. So sites quickly began implementing blanket nofollow policies: Every link submitted by users in any form would receive the tag by default. The idea would be that links that had not yet been vetted by users would get the nofollow tag and those that were deemed acceptable would not. But most sites never figured out the right process to allow the nofollow tag to be removed. Today, nearly every social service, from FaceBook to Twitter to Digg to StumbleUpon, permanently marks nearly every link this way. Even Wikipedia, a long-time holdout, finally switched to a default nofollow on all but the English site.

The Nofollow War

What does this mean for Google? If the vast majority of user-generated links are tossed into the spam category as far as the search engine is concerned, it means that their entire system of discovering and ranking links is in jeopardy. The major social services, most of which attract the majority of end-user traffic, content, and links, are rendered useless in generating relevancy.

But these are the exact sources that Google ought to be focusing on the most. Many have noted that they hear about news more rapidly through real-time sources like Twitter than through less-dynamic traditional news sites and blogs. Even if Google had the ability to spider a service like Twitter in real time, which is doubtful, they would gain no insight from the links included in these sources. Social bookmarking sites like Digg are chock full of user-vetted links and should be gold mines for Google, but the nofollow tag makes them invisible.

This scarcity of user-generated links has made the links that are followable even more valuable. Scammers constantly create fake blogs of scraped (read “stolen”) content and users are paid to include followable links anywhere they can. Sites with a high PageRank value are constantly inundated with offers and attacked by hackers to siphon off high-value “votes”.

High-profile content providers are circling their wagons, drastically cutting down on outside links in order to focus PageRank on their own properties. Smaller publishers and blogs are striking back at the big guys, decrying their dearth of external links. Some even go so far as to initiate blanket nofollow policies against these big, respected, but non-linking sites.

This leaves Google with even fewer useful links with which to examine the Web. It also leaves the biggest content providers and networks and the savviest search engine optimization (SEO) pros with a bigger slice of the valuable top-of-Google result real estate.

The Fix Is In

Google is left with a looming nightmare scenario: As smaller, alternative, social, and real-time content providers disappear from the search engine, its overall relevance and value declines. Soon, a tipping point will be reached when users would rather rely on Twitter, FaceBook, and the rest for their Internet interactions than the old-fashioned search engine, email, and RSS readers that Google currently dominates. This house-of-cards collapse can only be avoided by including user-generated content in the Google index.

Search engines could simply ignore the nofollow tag, wading into the social stream and combatting spam in other ways. But this would lead to another rapid upswing of link spam, shifting the burden to content providers once again. And it might also expose links that actually should not be followed, leading to technical and even legal trouble.

The best solution would see the social networks designing in some method of removing the nofollow attribute once links are verified to be relevant and correct. But there is no incentive for them to help drive Google traffic to other sites. Indeed, Twitter recently took the next step, arranging the titles of user pages in an attempt to SEO their way to the top page of Google searches for user’s names. Only altruistic systems like Wikipedia are likely to design in this type of response.

Another possible scenario (to be explored another day) is the usurpation of today’s social web and its content by a new next-generation service. A web-based social client like FriendFeed could rapidly siphon away both existing and net-new content and users in the guise of openness and interoperability. Although new web spiders like Cuil have failed, perhaps old-fashioned crawling capability is no longer all that valuable in the social web.

The most likely fix is both predictable and pragmatic: Google must buy all successful source of social links (like Twitter, Bit.ly, StumbleUpon, and even FaceBook) and integrate them into their search system. Owning Twitter would enable Google to decide which links to follow and which to ignore. The reward of improving search results would be the incentive needed to add “re-follow” capability. Buying these services would also give Google an open pipe of the real-time traffic flowing through these services, a critical resource that they currently lack.

Google simply can not afford not owning the real-time web, and they must continue to buy up similar sources of content as they appear. Yahoo was unable to extract value from StumbleUpon, but Google’s other competitors will certainly try to undermine the search giant. Frankly, I’m shocked that Microsoft, FaceBook, or even Baidu have not yet snapped up services like Twitter, LinkedIn, and Digg even if only to keep them and the information they contain out of Google’s hands.

If you enjoyed reading this, you’ll probably also like my Foskett Services blog!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Google Is Heading For A Cliff; What Will They Do?
This post was categorized as Computer History, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Google's first page accounts for more than two thirds of my web site traffic!

I’m not blogging to get traffic; I’m blogging because I have something to say. But, being a curious person, I do measure my blog traffic, and I’ve become interested in how the search engine optimization gurus ply their trade. So a recent tip on using a custom Google Analytics filter to determine “front page” placement on that search engine piqued my curiosity.

After just one week of monitoring, the results speak for themselves: Google’s first page of results accounts for more than two thirds of my web site traffic, and less than a quarter of my visits come from a source other than Google! It should come as no surprise that the search giant’s first page has come to dominate Internet traffic, but it was amazing for me to see it demonstrated so graphically using data from my own blog.

Since the first of the year, I’ve noticed that my traffic has been up by about 50%, too. I was puzzled, since the uptick comes across all topics and posts. My friend Louis Gray suggests that Google might be responsible for this as well: Google updated their Page Rank results, potentially moving many of my pages into the first page of results.

Of course, I also recently switched to a full-text RSS feed, so perhaps fewer people are visiting my site to read longer articles. But my examination of direct and referring traffic in Google Analytics is inconclusive on this matter.

Using Google Analytics

Google is notoriously cagey on releasing information about their ranking system. They don’t want those SEO experts gaming the system to get their pages on top. Of course this hasn’t stopped anyone, but it can be difficult to determine where exactly one’s site is ranked.

The best information I’ve found is in Google’s Webmaster Tools. Here, one can view the top search queries for one’s own (verified) site, including the absolute position for both impressions (views in the search results) and traffic generated. Although this information is useful, it is not as quantitative as one might desire: I can see that a search for “iphone exchange setup” puts my iPhone ActiveSync setup page on top, but I can’t compare this to Yahoo or direct visits.

Google’s Analytics service is another great resource, with excellent recording and reporting of web site metrics over time. But the company seems reluctant to link this information to their search data, perhaps worried that they’ll make life too easy for those who are trying to game the system to get more traffic.

The Google Analytics SEO Filter

The SEO filter proposed by André Scholten certainly is clever: Although Google doesn’t include absolute ranking data in Analytics, their own URLs do include an indication of which page of results a user was viewing when they clicked to visit a site. When one is not satisfied with the results on the first page of Google’s search, they click “Next” and are taken to another page of results. Google tracks which page to display using an element in the URL called “start”. For example, the second page of results is called by including “&start=10″ in the URL, while the 10th page includes “&start=90″.

Since Google Analytics normally records the full referring URL, we can use an advanced feature called filtering to tag Google referrals with an indication of which page the user was looking at when they clicked. André doesn’t include the whole recipe, but it’s not difficult for a novice to implement:

1. Get Google Analytics up and running for your site. I’m not going to go into how to do this here.
2. In the main Analytics page listing Website Profiles for your user, click “+ Add new profile” in the gray bar. It is important that anything you do is done in a new profile, since this is a filter, and will remove (filter out) things that don’t match.
3. Select “Add a Profile for an existing domain” (since you already set this domain up in step 1) and give it a Profile Name like “blog SEO”.
4. Click Finish, and now we have a profile to work with without messing up our main Analytics results.
5. Find this new profile in the Website Profiles list and click “Edit” under the “Actions” heading.
6. The third box is “Filters” – this is where the magic happens. We will create three filters to only collect information on Google searches, and once you create them you can re-apply them in other profiles for other sites without reinventing the wheel.
7. Click “+ Add Filter” and add a new one called “Ranking 1″. “Filter Type” will be “Custom filter”, and you will select “Include” under this. Then use “Campaign Source” as the “Filter Field” and enter “Google” as the “Filter Pattern”. This will filter out any traffic that didn’t come from a Google referral.
8. Click “+ Add Filter” and add a new one called “Ranking 2″. “Filter Type” will be “Custom filter”, and you will select “Include” under this. Then use “Campaign Medium” as the “Filter Field” and enter “organic” as the “Filter Pattern”. This will filter out any paid traffic from Google ads.


The magic Google Analytics SEO filter

9. Now comes the big one. Click “+ Add Filter” and add a new one called “Ranking 3″. “Filter Type” will be “Custom filter”, and you will select “Advanced” under this.
10. For “Field A -> Extract A”, select “Referral” and enter “(\?|&)q=([^&]*)” in the box (no quotes!) This matches the key/value sets in the URL.
11. Next, for “Field B -> Extract B”, select “Referral” again and enter “(\?|&)start=([^&]*)” in the box (no quotes again!) This extracts the value for the “start” key, if present.
12. Finally, for “Output To -> Constructor”, select “User Defined” and enter “$A2 (page: $B2)” in the box (think you include quotes? Think again!) This adds a custom tag with the full URL and the value of the “start” key.
13. Finish it off by checking “Yes” for “Field A Required”, “No” for “Field B Required”, and “Yes” for “Override Output Field”.
14. Now give it some time – at least a day for Google to start adding data to this profile. Assuming you did everything correctly, you shouldn’t have any trouble with the statistics recording in your main profile.

Watching the Results

Assuming it worked, you can now add the correct chart to Analytics in the new profile:

Google's first page dominates their referrals

1. Enter the new profile’s Dashboard by clicking “View report”.
2. Select “Traffic Sources” on the left side.
3. Click “google (organic)” in the “Top Traffic Sources” table – it should be the only item there!
4. In the “Dimension” drop-down box, select “User Defined Value”. This is the payoff for all this work!
5. Add this to the dashboard by clicking the “Add to Dashboard” button at the top.
6. Now use your creativity to examine the data! For example, enter “page: 10″ in the “Find User Defined Value” box. These are search terms that showed up on the second page for some reason, so you might try some SEO kung fu to improve their ranking…

You will notice a number of visits coming up as “(not set)”. This means that the Google referral didn’t include the long URL with keys and values, so you don’t know where the user was when they clicked.

With some creativity, you could adapt this filter to grab Yahoo, Microsoft, or other SEO information. But Google is definitely the 800 lb gorilla right now, and page 1 is the target – it accounts for 90 to 95% of Google referrals to my site!

If you enjoyed reading this, you’ll probably also like my Foskett Services blog!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Measuring the Importance of Google’s First Page
This post was categorized as Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.