---------------------------------------- | |
robots.txt | |
May 24th, 2019 | |
---------------------------------------- | |
I've run a little disallow robots.txt file on this gopher hole for | |
a while. Now there's been more discussion about its inclusion on | |
gopher for gopher search engines. This leads me to a little | |
problem. | |
My robots.txt file is not meant for gopher crawlers. I created it | |
because I do not want my site crawled by web crawlers trawling | |
through proxies to gopher. There's some question as to whether | |
a robots.txt file is effective in that way (technically the proxy | |
should have one, not the site itself), but I thought it better | |
than nothing. | |
Now I have a dilemma. I would very much like my site to be crawled | |
by Veronica-2 and other gopher-centric search engines, but I don't | |
want Google plopping my burrowy-thoughts on the web. What's | |
a gopher to do? |