Dilip

Dilip
<[email protected]> 27 Nov 2020
========================================================================
Shovel

A collection of python scripts used to dig out this gopher hole.
Written by Dilip <[email protected]>.

This system of python scripts ingest a variant of the markdown script
and parses it into the raw text you see here.

Syntax Guide.
=============
Markdown syntax:

--------------------------------------------------------------------------
| # Title |
--------------------------------------------------------------------------
Sets the title for the page. For example, see Shovel at the top of this
page. Must be the only thing in that line. If a date is available, it
is added to the title. This is processed by the script md_parser.

--------------------------------------------------------------------------
| ## Header |
--------------------------------------------------------------------------
Formats as a section header. For example, see "Synatax Guide" at the
top of this page. Must be the only thing in that line. This is
processed by the script md_parser.

--------------------------------------------------------------------------
| ### Header |
--------------------------------------------------------------------------
Formats as a subsection header. See the details of the scripts below
for an example. Must be the only thing in that line. This is processed
by the script md_parser.

--------------------------------------------------------------------------
| [Link Text](Link) |
--------------------------------------------------------------------------
Adds a link as a footnote with the provided text. A footnote number
will replace the link where it was first encountered. If the same link
is encountered again in the page, the same footnote number will be
used. This is turned into %%FOOTNOTELINK by the md_parser to be later
dealt with by the annotate script and the linker script.

```
Code block.
```
The triple backtick (the sign below the ~ key, to the left of 1 on most
keyboards will produce text as code. This was used to show the markdown
syntax above for example. This is performed by the md_parser script.

------------------------------------------------------------------------------------------------
| - Bullet Lists |
| - These are not numbered |
| - Bullet points with the same number of left spaces will be indented |
| together. |
| - A tab is treated as 2 spaces for you barbarians who mix tabs and spaces. |
| - Bullets must begin with a "-" and a space (a minus sign on most |
| keyboards). |
| |
| 1. Numbered Lists |
| 2. The script is fairly dumb. If you skip a number, the script will not |
| correct it for you. |
| 1. Numbered lists must begin with a number and a period. |
| 2. Roman numarals, alphabets, hexadecimal / binary / imaginary / |
| complex / quaternion / octonion numbers and anything else that is not a |
| number according to a kindergartener will not work. |
| 3. As with bullet points, points with the same number of |
| left spaces are indented together. |
------------------------------------------------------------------------------------------------
* Bullet Lists
* These are not numbered
* Bullet points with the same number of left spaces will be
indented together.
* A tab is treated as 2 spaces for you barbarians who mix
tabs and spaces.
* Bullets must begin with a "-" and a space (a minus sign on most
keyboards).
1. Numbered Lists
2. The script is fairly dumb. If you skip a number, the script
will not correct it for you.
1. Numbered lists must begin with a number and a period.
2. Roman numarals, alphabets, hexadecimal / binary / imaginary
/ complex / quaternion / octonion numbers and anything else
that is not a number according to a kindergartener will not
work.
3. As with bullet points, points with the same number of left
spaces are indented together.
Extended syntax:
================
--------------------------------------------------------------------------
| %%TITLE{Title Text} |
| %%H1{Header 1 text} |
| %%H2{Header 2 text} |
--------------------------------------------------------------------------
The markdown parser will emit these. They can also be emitted manually.
The layout script will process these and format them in it's output.

The %%TITLE is also used to identify links to this file. If multiple
%%TITLE lines are found, only the first one will be used. The md_parser
script records the %%TITLE as it is emitted.

--------------------------------------------------------------------------
| %%BULLET_LIST{indent}{Text} |
| %%NUMBERED_LIST{indent}{Text} |
--------------------------------------------------------------------------
The markdown parser will emit these. They can also be emitted manually.
The layout script will process these and format them in it's output.

--------------------------------------------------------------------------
| %%DATE{MM-DD-YYYY} |
--------------------------------------------------------------------------
If found in a file, all scripts will use this date with this file. This
is supported by the utility script, all other scripts might use this.
This will be cleared by the cleanup script.

--------------------------------------------------------------------------
| %%QUOTE{Quoted text}{Author} |
--------------------------------------------------------------------------
Formats the quoted text and attributes it to the author. The Author is
optional. This must be on a line by itself. Here is an example:

I am going to Nebraska for two weeks. What am I going to
do?
- Random girl sitting next to me in the
park while I wrote this.
This is managed by the layout script.

--------------------------------------------------------------------------
| %%START_QUOTE{Title} |
| %%END_QUOTE{Author} |
--------------------------------------------------------------------------
Allows quoted text to span multiple lines. Whitespaces will be
respected (if possible). Great for poems and the like. Start and End
must be the only things on their lines. This is managed by the layout
script.

When using this syntax, let the script align the whitespace.

Shake Prayer

Open to them your hand to the shore,
Watch them walk into the sea.

Press upon them all they need,
see them yearn for all they want.

Gift to them the calm pool of words,
watch them draw the sword.

Bless upon them the satiation of peace,
see them strave for war.

Grant them darkness and they will lust for light.
Deliver to them death and hear them beg for life.
Beget life and they will murder your kin.
Be as they are and they see you different.
Show wisdom and you are a fool.
The shore gives way to the sea.
And the sea, my friends,

Does not dream of you.
- Steven Erikson in Reaper's Gale,
chapter 21
--------------------------------------------------------------------------
| %%DISABLE_MD to disable markdown. |
| %%ENABLE_MD to re-enable markdown. |
--------------------------------------------------------------------------
This will disable markdown processing. For example, this was used to
show how backticks are used format code blocks. This is cleared by the
cleanup script.

--------------------------------------------------------------------------
| %%FOOTNOTE{Footnote text} |
| %%FOOTNOTELINK{Footnote text}{link} |
--------------------------------------------------------------------------
Adds a footnote [1]. Footnotes can be on a line along with other text.
This is emitted by the annotate script, it can also be added manually.
This is managed by the annotate script.

--------------------------------------------------------------------------
| %%CATEGORY{category} |
--------------------------------------------------------------------------
Request to be identified under a category. This is processed by the
metadata script. If the paginator script finds one of the %%CHILDREN
category in a page, a link will be added to this page.

--------------------------------------------------------------------------
| %%CHILDREN{category} |
| %%CHILDREN_WITH_NEWEST_FIRST{category} |
| %%CHILDREN_WITH_OLDEST_FIRST{category} |
--------------------------------------------------------------------------
When the paginator finds a %%CHILDREN tag on a .page file, it will scan
through all the children that have marked the current category and
generate links to them.

--------------------------------------------------------------------------
| %%ROOT |
--------------------------------------------------------------------------
Mark the file containing this as the top level file in the gopher
space.

--------------------------------------------------------------------------
| %%DISABLE_LAYOUT |
| %%ENABLE_LAYOUT |
--------------------------------------------------------------------------
These are mostly emitted by other scripts in the collection to tell the
layout script not to disturb something. They can also be manually added
- the layout script doesn't care where this came from.

--------------------------------------------------------------------------
| %%LINK{Text}{Link} |
--------------------------------------------------------------------------
The linker script translates these into links.

--------------------------------------------------------------------------
| %%NO_HEADER |
| %%NO_FOOTER |
--------------------------------------------------------------------------
The layout script is responsible for adding the header and a footer to
each page. This tag tells the layout script to not perform this step.
The tags are cleared by the metadata script.

--------------------------------------------------------------------------
| %%INSERT{textfile.txt} |
| %%BANNER{banner.txt} |
--------------------------------------------------------------------------
The insert script looks for the specified file and pastes its content
over to the current file. %%BANNER is like %%INSERT except that it is
surrounded by %%DISABLE_LAYOUT and %%ENABLE_LAYOUT so that the layout
script doesn't change the inserted text or ascii art.

--------------------------------------------------------------------------
| \%% |
--------------------------------------------------------------------------
For the rare occasion where want to have %% in your text. Like in the
previous sentence for example.

Modules and extending them.
===========================
NewlineKiller
-------------
This is the first script that is run. It attempts to remove as many new
lines as possible using simple rules:

1. It respects blank lines added in the original Markdown source.
2. It forces a newline when the next line begins with a markdown
header, a code block or a %% symbol (a good indicator that the next
line begins with a keyword).
3. It copies over anything inside a multiline quote or a code block as
is.
4. It does not modify anything between %%DISABLE_LAYOUT and
%%ENABLE_LAYOUT
The primary purpose of this script is to simplify parsing for all other
scripts. The primary concern without this script is a %%QUOTE{ .... }
that has several newlines in it - all scripts parse the source markdown
line by line, multiple line breaks will make the code for matching the
curly braces too complicated.

Metadata
--------
This is the second script that is run. If it finds any of these in the
file, it will remove them in it's output after recording the relavant
metadata.

--------------------------------------------------------------------------
| %%DATE{MM-DD-YYYY} |
| %%CATEGORY{category} |
| %%ROOT |
| %%NO_HEADER |
| %%NO_FOOTER |
| %%CATEGORY{category} |
--------------------------------------------------------------------------
This script only records these keywords, other scripts will do
something with these.

MdParser
--------
This is the first script that is run. It parses the following markdown
syntaxes:

--------------------------------------------------------------------------
| # Title |
| ## Header 1 |
| ### Header 2 |
| [Link Text](link) |
| - Bullet list |
| 1. Numbered List |
--------------------------------------------------------------------------
and
```
code block
```
The MdParser script is the third script that is run. It ignores
everything between %%DISABLE_MD and %%ENABLE_MD. For all other markdown
syntaxes that it supports (see list above) and translates them into
other keywords that is understood by the layout or the linker scripts.
Specifically, it emits these keywords:

--------------------------------------------------------------------------
| %%TITLE{Title Text} |
| %%H1{Heading 1} |
| %%H2{Heading 2} |
| %%FOOTNOTELINK{text}{link} |
| %%BULLET_LIST{indent}{Text} |
| %%NUMBERED_LIST{indent}{Text} |
--------------------------------------------------------------------------
Paginator
---------
The paginator script looks for one of these keywords and expands them
with entries with matching categories.

Indexer
-------
The indexer script is currently a pass-through. It was intended to
generate Indexes like those found at the back of a textbook, but I am
not sure if I really want that right now. This document will be updated
if I decide to go ahead and implement it.

Annotate
========
The annotate script is responsible for annotating a file with the
footnotes. It numbers footnotes and links separately. It adds a Notes
section and a Links section at the bottom of the file if any footnotes
or links are found respectively.

Change Log.
===========
2020, Nov 27: Initial Version.

Notes:
------
1. This is an example