Subj : Web server and BOT's
To : Richard Vonzel
From : nelgin
Date : Wed Jun 25 2025 11:26 pm
Re: Web server and BOT's
By: Richard Vonzel to All on Wed Jun 25 2025 04:25:24
> I turned on the Synchronet Web server, works great, hsad my website running.
> Had my BBS's files available for access thru it. Yesterday I guess a BOT got
> on my site and jst downloaded all the file I had available. Here's the log
> entry for this,
I have found gptbot ignores a global disallow rule in robots.txt
User-agent: *
Disallow: /
But you could try adding
User-agent: gptbot
Disallow :/
It could take a few days but you'll see when they start to pickup your robots.txt file.
Another option would be to just block the ip ranges altogether.
Another option would be to block the published subnets. I have these blocked using ipsets
