Subj : Web server and BOT's
To   : nelgin
From : MRO
Date : Thu Jun 26 2025 03:24 pm

 Re: Web server and BOT's
 By: nelgin to Richard Vonzel on Wed Jun 25 2025 11:26 pm

>> guess a BOT got on my site and jst downloaded all the file I had
>> available. Here's the log entry for this,

ne> I have found gptbot ignores a global disallow rule in robots.txt

ne> User-agent: *
ne> Disallow: /

ne> But you could try adding


dude where have you been? most of those bots ignore robots.txt

google bot, archive.org's bots, all of them.  there's no law that they
have to obey robots.txt
---
� Synchronet � ::: BBSES.info - free BBS services :::

You are viewing proxied material from cvs.synchro.net. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.