* * * * *

                              The Archive Effect

I just implemented a small change to mod_blog, the code that runs this site.
I added a new template command, %{robots.index}% that runs the following
fragment of code:

> if (m_navunit == PART)
>   fprintf(fpout,"index");
> else
>   fprintf(fpout,"noindex");
>

To use it:

> <META NAME="robots" CONTENT="%{robots.index}%,follow">
>

The effect is to generate:

> <META NAME="robots" CONTENT="noindex,follow">
>

or

> <META NAME="robots" CONTENT="index,follow">
>

Depending on if the page contains multiple entries or a single entry.

So far, nothing earth shattering and to tell the truth, pretty trivial as
features go. Not to mention pretty specific. This particular tag is used to
control the behavior of search engine robots [1] (at least, those that
understand this convention) and it may seem strange that I am instructing
these web robots to skip indexing my site unless they pull up a single entry.

But it's not that strange when you think about it. And the feature, however
trivial it is to implement (and it's been a feature I've been wanting to add
for some time now) is a form of search engine optimization. And is something
that other blogging software should do, but doesn't.

First off, the chance of someone coming across my pages with a disturbing
search request [2] is pretty high, given that up to a full year's worth of
text can be indexed (actually, you can get the every entry I've written here
in one shot, but I'll leave that up as an exercise for the reader) so that I
get a search like:

> the chipper that is configured like a triangle with two hitting areas is
> illegal
>

with the resultant page being all the entries I've written for 2001; I'm
guessing that most, if not all, of those words do appear within the entries
of 2001, but not in a single entry.

The result isn't just to solely remove the posibility of distrubing search
requests from my site (“aggresiveness of college students in manila”—I'm not
a college student, nor am I in Manila, but hey, if they're female, I wouldn't
mind them being aggressive) but to actually help the search engines generate
good results for people (I'm sure the person looking for “door king 1812 dsl
technical information” didn't appreciate being fed every entry for 2002).

Then again, I will miss those that come looking for “mitubishi polyester film
corp.”

[1] http://www.robotstxt.org/wc/exclusion.html#meta
[2] http://searchrequests.weblogs.com/

Email author at [email protected]