Mod Rewrite Rules for Uniform Spidering

Friday, September 14th, 2007

I mention in my earlier post that you should choose to use www or not, but regardless, stick with it. Another issue that arises is that you can’t always control how someone else will link to your site. While most will include www. others wil not. To insure that all you links are cataloged in the search engines uniformly, you should implement a couple of rules in your .htaccess file.

  1. A rule to insure www is (or is not) always used. This exmple always adds the www…
    Options +FollowSymLinks
    RewriteEngine On
    RewriteBase /
    RewriteCond %{HTTP_HOST}!^somewebsite\.com$ [NC]
    RewriteRule ^(.*)$ http://www.somewebsite.com/$1 [R=301,L]
  2. A rule to always redirect your home page (index.html, index.php, default.htm, etc) to just root (http://www.yourwebsite.com/)
    REDIRECT 301 “index.htm” http://www.somewebsite.com

     

Thers is a good guide on mod rewrite here at the source.

While it won’t apply to most of the people reading this blog, it is also important not to have multiple views of the same information. So if you have a site with a page of toasters and you give the browser the option to sort by price, color, etc, be sure that the spiders only see one sort of the information.

Never delete or rename a page…

Friday, September 14th, 2007

Well, almost never… I talk about the importance of page names on my SEO page. The trick is to design your site so that you have the best page names with the right content in those pages for top organic results. But what happens if you inherit a poorly designed site, the content becomes obsolete, or a generic term like “MP3 player” is suddenly secondary to a more popular phrase like “iPod”?

In those cases, you should rename the page and use a 301 redirect (moved) or 410 (deleted) to ensure any traffic to the old page finds the new page. This is critical, unless the page has no traffic and no incoming links, you should never just delete or rename a page and move on.

Here’s a good article on 301 redirects and how to implement them.

Good 301/410 discussion here