Introduction
Introduction Statistics Contact Development Disclaimer Help
----------------------------------------
robots.txt
May 24th, 2019
----------------------------------------
I've run a little disallow robots.txt file on this gopher hole for
a while. Now there's been more discussion about its inclusion on
gopher for gopher search engines. This leads me to a little
problem.
My robots.txt file is not meant for gopher crawlers. I created it
because I do not want my site crawled by web crawlers trawling
through proxies to gopher. There's some question as to whether
a robots.txt file is effective in that way (technically the proxy
should have one, not the site itself), but I thought it better
than nothing.
Now I have a dilemma. I would very much like my site to be crawled
by Veronica-2 and other gopher-centric search engines, but I don't
want Google plopping my burrowy-thoughts on the web. What's
a gopher to do?
You are viewing proxied material from gopher.black. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.