* * * * *
It's really been over ten years since I wrote “An Extended Standard for Robot
Exclusion”? Wow …
> Around the same time the IETF (Internet Engineering Task Force) draft was
> being discussed, Sean “Captain Napalm” Connor [sic] proposed his own
> extension [1] to the Robots Exclusion Protocol, which included Allow rules
> as well as regular expression syntax for rules, and new Robot-version,
> Visit-time, Request-rate, and Comment rules. Less than 100 of the sites I
> visited use rules unique to this spec.
>
Via email from Steve Smith [2], “robots.txt Adventure [3]”
[As a small aside, I don't know why people insist on spelling my last name
with an “O-R” instead of an “E-R”. It's not like I misppelled [4] my own name
on that page [5]. Sigh. —Editor]
That's not the only place “An Extended Standard for Robot Exclusion” has been
referenced—it's also mentioned in O'Reilly's [6] _HTTP: The Definitive Guide_
[7], but until Steve reminded me of it, I basically forgot about it.
Understandable since the last time it was edited was November of 2002 [8]
(and even then, the previous time it was edited was six years earlier—it's
old).
This probably means it's time once again to check the links and make sure
they all work.
And maybe clean up the HTML (HyperText Markup Language) while I'm at it.
Just as soon as I can reproduce that insipid Heisenbug [9].
[1]
http://www.conman.org/people/spc/robots2.html
[2]
http://www.steventalcottsmith.com/
[3]
http://www.nextthing.org/archives/2007/03/12/robotstxt
[4]
gopher://gopher.conman.org/0Phlog:2007/09/11.2
[5]
http://www.conman.org/people/spc/robots2.html
[6]
http://www.oreilly.com/
[7]
http://www.amazon.com/exec/obidos/ASIN/1565925092/conmanlaborat-
[8]
gopher://gopher.conman.org/0Phlog:2002/11/05.1
[9]
gopher://gopher.conman.org/0Phlog:2007/09/23.1
Email author at
[email protected]
