__________________
DYNAMIC MARMOTTE
Nicolas Herry
__________________
2022/10/31
1 Dynamic marmotte
==================
This is the second article on the Gopher server I'm currently
working on, marmotte. After I introduced the project in [the
first article], I have worked some more on the project and
present here some of the new features.
[the first article] <
gopher://gopher.beastieboy.net/0/marmotte.txt>
1.1 Articles in this series
~~~~~~~~~~~~~~~~~~~~~~~~~~~
[Marmotte]
[Marmotte] <
gopher://gopher.beastieboy.net/0/marmotte.txt>
1.2 Dynamic responses
~~~~~~~~~~~~~~~~~~~~~
These last few days' efforts were all about adding dynamic
rendering to marmotte. The original [RFC1436] describes Gopher in
its abstract in these words:
The Internet Gopher protocol is designed for
distributed document search and retrieval. This
document describes the protocol, lists some of the
implementations currently available, and has an
overview of how to implement new client and server
applications. This document is adapted from the
basic Internet Gopher protocol document first issued
by the Microcomputer Center at the University of
Minnesota in 1991.
Apart from the part on searching, this description doesn't
mention any dynamic aspect of Gopher, and the rest of the
document doesn't address this either, apart from one little
sentence in the rather mundanely named section 3.6, /Building
ordinary internet Gopher servers/:
The retrieval string sent to the server might be a
path to a file or directory. It might be the name of
a script, an application or even a query that
generates the document or directory returned.
That's all. Dynamic content? Oh, let's just not cover it and only
briefly hint at the possibility... This is very typical of
Gopher, how it's been designed, and how it's been documented. To
me, this makes Gopher very much a protocol for hackers: it leaves
many things to the creativity of the coders. I intend to get back
to the topic of hackability in software in a later essay; for
now, let's focus on marmotte, and more specifically, on the three
ways marmotte can now produce dynamic responses.
[RFC1436] <
https://www.rfc-editor.org/rfc/rfc1436>
1.2.1 Rendering pipeline
------------------------
At the heart of marmotte are two pipelines: one for processing
requests, and one for preparing responses to these requests. Both
pipelines are implemented in the same manner: a collection of
transformations applied to a piece of data describing a request
or a response. Each transformation is made up of a predicate and
a function. To understand how this works, let's follow a simple
exchange between a client and a server.
A client sends a request to the server, in the form of a selector
(a string). Upon receving the request, the server simply goes
through the collection of request transformations, asking each,
via its associated predicate, if it should run, and if so, calls
the function with the request. The modified request is then
passed to the next transformation in the pipeline, and so on
until all candidate transformations have got a chance to run. The
final version of the request is then passed to the response
pipeline the same way: each transformation lets its predicate
decide whether it should run, in which case its function is run
against the request and the current version of the response, and
both are passed down the rest of the collection. The final
response is what is returned to the client.
This generic approach allows to satisfy basic cases, where a
request contains a path to a file, for example, and doesn't need
any change, and where a response is handled by a single
transformation, which simply loads the data from the
corresponding file and stores it in the response. But the same
architecture caters to more complex scenarios. One is that of
selector rewriting: it is similar to URL rewriting, where a path
in a selector is changed to map onto a real, physical directory,
and is an easy way of implementing virtual directories. Another
is transforming the data itself: adding headers, footers,
modifying the data, compressing it... All this can be implemented
via instances of marmotte's RequestTransformers and
ResponseTransformers.
The current version of marmotte implements dynamic gophermaps
that way: when handling a directory where no gophermap file is
present, the ResponseTransformer simply generates one. Headers
and Footers are also available and multiple instances can be
defined: one can devise a Footer for all gophermaps, a Header for
textfiles in a specific directory, another Header for files
maching specific names, and these can be combined or remain
exclusive. This is made possible thanks to having the functions
being separate from the predicates: a given transformation
function can be combined with any predicate and packaged in a
ResponseTransformer any number of times. In the examples above,
we would have three ResponseTransformers, each having the same
functions (Header or Footer) and different predicates. Some would
only match on gophermaps, some on files with a specific name, or
in a specific path...
Over time, I will keep adding to these transformation functions
and predicates to implement gateways to other protocols, file
format conversion, file compression, responses that would change
depending on the time of the day, and so on.
1.2.2 Executables
-----------------
marmotte now also has a new ResponseTransformer, the
ExecutableTransformer. As the name suggests, it allows marmotte
to call external programs and return the data generated by these
programs. The programs in question can be anything that marmotte
can call: shell scripts, Perl scripts, binaries... The main
contract between marmotte and a program is in two parts: the
latter will receive a root path where to look for and generate
files and directories, and the former will get in return a single
string, which can be either a directory or a file. The response
crafted by this transformation is then picked up by the rest of
the pipeline and either presented as a gophermap (generated or
loaded from a gophermap file the program would have placed in the
directory), or loaded from a file on the disk.
Below is a simple example:
,----
| #!/bin/sh
|
| # Generate the filename
| FILENAME=`cat /dev/random | env LC_CTYPE=C tr -cd "[:alnum:]" | head -c 32`
| FILEPATH=/tmp/$FILENAME
|
| # Generate a file with some text contents
| if [ ! -d $1/tmp ]; then
| mkdir $1/tmp
| fi
|
| echo "This is an automatically generated file, named $FILENAME." > $1/$FILEPATH
|
| # Return the filepath to marmotte
| echo $FILEPATH
|
`----
This shell script generates a file with a random name (tapping
into the power of `/dev/random'), places it in the subdirectory
`/tmp' of the gopherhole and returns the path to it (relative to
the server's root). Nothing fancy, but anything can happen
between the first line and the last line...
Having the ability to run executables opens many, many doors:
this is one of the main reasons the web exploded in the mid-90s,
and is akin to the `cgi-bin' that were popular then (and still
are everywhere today, in the more modern implementation of
`fast-cgi'). A file can be altered before being served, or
generated from scratch, and entire directory hierarchies can even
be generated in one go and served to the client.
One thing that I will need to implement in the coming weeks is
sandboxing, to restrict the programs to only safe portions of the
filesystem, instead on relying on the executables to politely
place everything under the server root given in argument.
1.2.3 Templates
---------------
The last addition in terms of dynamic rendering is
templates. This is another feature that propelled the early Web
forward in the 90s, and opens at least as many doors as the
ability to call external executables. marmotte being written in
Go, I've made the simple choice here to leverage the power of the
template engine that comes with this language.
The current implementation is still in its very early stages, and
is only used for dynamic error reporting (more on that in a later
article). These error documents are designed as Go templates that
accept from the server a structure of data containing arbitrary
informations.
Here's an example below:
,----
| You have the wrong permissions to access the resources behind the selector {{ .Selector }}. If you believe you should have access to these resources, please contact the administrator.
`----
Here, the selector originally sent by the client will be
displayed, coming from the data structure provided by the
server. This might seem fairly basic, but this opens all kinds of
possibilities, as this structure can contain any type of data,
from any provenance, and even data contributed to by other
RequestTranformers or ResponseTransformers, and be used in any
kind of document, not only error messages.
In the coming weeks, I will probably introduce these templates to
the existing Headers and Footers facilities, to make these even
more interesting, and simply generalise and promote the use of
templates as a very simple way to prepare dynamic content. Again,
ResponseTransformers will do the work here, simply running
against any file with a `.gt' extension, for Gopher Template.
1.3 In the next episode...
~~~~~~~~~~~~~~~~~~~~~~~~~~
I have left some features out for today and will cover these in
future articles: virtual gopherspaces (aka, /tildes/, as in
~bboy) and errors and errorcodes. Once again, stay tuned!
