Subj : Last couple of days with my binkp mailer..
To : Jame Clay
From : Janis Kracht
Date : Mon Aug 23 2010 09:14 am
Hi Jame,
> Are you sure that it was actually googlebot? It only follows http links,
I guess since the web version of the bbs is http, it went there on my port
8080.. and from there, there are links on that page for ftp, etc. and yes I'm
sure it was google bot. MSN-Bot is another one that likes to run all over my
site - I've seen them in my bbbs inet.log as well <grin>.
> which won't lead to that binkp port... (How'd the IP show up in a reverse
> dns?)
I'm sure it was Google-bot :)
janis:~$ host 66.249.71.53
53.71.249.66.in-addr.arpa domain name pointer crawl-66-249-71-53.googlebot.com.
That is for sure the google spider :)
> Also; do you have a robots.txt? I use it to disallow "/bbbs"...
I haven't put that in there, but I will.. Should work, though there's no need
right now. Good idea for the bbs, though, thanks :)
Google called me on the phone quite a while back and their representative asked
me to allow Google-bot to index my main site that runs apache, with robots.txt
- I don't mind that since it doesn't involve the bbs nodes.. this is how I
allow them to scan my main site on port 80:
