---
layout: ../Site.layout.js
---
# `:screwlisps-knowledge/tangle` common lisp asdf package markdown tangling and lisp pathnames

Here I exhibit a simple use of [`cl-series`](https://gitlab.common-lisp.net/rtoy/cl-series/\-/wikis/Series-User's-Guide)' [`scan-file`](https://gitlab.common-lisp.net/rtoy/cl-series/\-/wikis/Series-User's-Guide#scan-file-function) and [`collect-file`](https://gitlab.common-lisp.net/rtoy/cl-series/\-/wikis/Series-User's-Guide#collect-file-function) to tangle a markdown file into a lisp package in a specified lisp system. Further, I add a system stubbing function which uses [lisp `pathname`](https://www.lispworks.com/documentation/HyperSpec/Body/19_b.htm)s, which are a tricky topic.

My motivation was to at least in common lisp reconcile my small-web markdown and lisp systems, by defining my lisp systems as tangles of [my](https://codeberg.org/tfw/screwlisps-kitten/) [small-web markdown](https://kitten.small-web.org/tutorials/markdown/). Edrx showed me that something like this could be done by modifying [`emacs` `eev`](https://anggtwu.net#eev)'s `code-c-d`, but, well, I know one hundred percent how what I just wrote here works.

It is a little hard to remember that common-lisp pathname stuff, as Kent fairly frequently says: it was written for back when there was more than one style of filesystem. However, after thinking a bit about it, I am fairly happy with it. the gist is that I have

1. Always used [`make-pathname`](https://www.lispworks.com/documentation/HyperSpec/Body/f_mk_pn.htm).
1. Represented unknown paths as [relative and wild (`(:relative :wild)`)](https://www.lispworks.com/documentation/HyperSpec/Body/26_glo_v.htm#valid_pathname_directory).
1. Made a new pathname when you know more of the path.

Wild pathnames can have more advanced uses, but this is not a bad idiom on modern systems anyway I think. The modern string convention is I think, more primitive.

# eev setup

This is a block comment, prior to the required package specification. Elsewhere I have talked about my focus on [embeddable common lisp](https://gitlab.com/embeddable-common-lisp/ecl).

```
#|
;; Allowing, the quicklisp links are for sbcl and sly. Still, it gives an idea.
 (find-quicklisp-links "series")
 (setq inferior-lisp-program "ecl")
 (slime)
 (setq eepitch-buffer-name "*slime-repl ECL*")

(require :series)
|#
```

# :screwlisps-knowledge/tangle package definition

This *must* be the first lisp expression in the file (since from our [`:package-inferred-system` useage](https://github.com/fare/asdf/blob/master/doc/best_practices.md#package_inferred), experientially files won't exist but packages do).

```
(uiop:define-package :screwlisps-knowledge/tangle
   (:import-from :series install)
 (:mix :asdf :uiop :cl)
 (:export :tangle)
 (:nicknames :sk/tangle :screwniverse/tangle))
(in-package :sk/tangle)
(eval-when
   (:compile-toplevel :execute)
 (install))
```

For those who aren't used to common-lisp, asdf, and uiop:

## [ASDF](https://asdf.common-lisp.dev/asdf.html) `:package-inferred-system` details

This is in my view a modern residential source filesystem in the sense that there is no "file" - only packages, where packages are inside systems. In order to make this work, each source file created in a system starts with either `defpackage` or [`uiop:define-package`](https://asdf.common-lisp.dev/uiop.html#index-define_002dpackage), and the package generally needs to be named `:system/dir1/dir2/dir3/filename` (downcased).

Then, in the absense of a `(:mix ..)`, `(:mix :cl)` is assumed if I recall correctly, however I specifically `import` `install` from `series`.

In the new package=file, at `compile-toplevel`, I call cl-series' `install` (which temporarily interns series' exported symbols into the current or specified package).

## Interactively muddling through it once

### Setup a test input and output file

Let's do this in a comment in the file. In some sense, it is documentation of what we did. Let's also use the [common lisp formatted output](https://www.lispworks.com/documentation/HyperSpec/Body/22_c.htm) [`~@\newline` directive](https://www.lispworks.com/documentation/HyperSpec/Body/22_cic.htm) which will let us use indented `'''`s safely.

```
(with-open-file
   (*standard-output* #p"/tmp/test-tangle.md"
                      :direction :output
                      :if-does-not-exist :create)
 (format *standard-output*
         "# An MD file~@
          having~@
          ```~@
          (lisp code)~@
          ```~@
          blocks~@
          ## Such as~@
          ```~@
          ;; this.~@
          ```~@
           ``` but not this.~@
          ."))

```

### Let's read the codes of that using cl-series

cl-Series provides working fakes of its static-analysis-generated tight, lazy, efficient functionality. However, the fakes are not pure common lisp. But you can use them to explore interactively in series' declarative style, and see what-you're-working-with while you're working on it, as [Hairylarry for example mentioned](https://gamerplus.org/@hairylarry/114760934821459035) doing.

```
(defparameter *filein* (scan-file #p"/tmp/test-tangle.md"
                                 #'read-line))
*filein*
```

Using eev-mode, on my right side I see:

```
#|
SK/TANGLE> (defparameter *filein* (scan-file #p"/tmp/test-tangle.md"
                                                                         #'read-line))
*FILEIN*
SK/TANGLE> *filein*
#Z("# An MD file " "having " "```" "(lisp code)" "```" "blocks" "## Such as" "```" ";; this." "```" ".")
SK/TANGLE>
|#
```

I guess, I would like to know where lines starting with `'''` are.

```
(defparameter *codeblock-toggles*
 (#Mstring= (series "```") *filein*))
*codeblock-toggles*
```

Series' `series` creates an infinite series of whatever its arguement is, which I compare to the finite series of lines [`read-line`](https://www.lispworks.com/documentation/HyperSpec/Body/f_rd_lin.htm)ed from the file. We don't have to personally manage the end of file stuff.

Outputs:

```
#|
SK/TANGLE> (defparameter *codeblock-toggles*
              (#Mstring= (series "```") *filein*))
*CODEBLOCK-TOGGLES*
SK/TANGLE> *codeblock-toggles*
#Z(NIL NIL T NIL T NIL NIL T NIL T NIL)
SK/TANGLE>
|#
```

Looking good. Now here we come to an annoying-ish point for my personal intuitions about series. Sometimes we need to use a toggle as I am about to, because we-don't-know-what-happens-in-the-future. This is quite different to eagerly evaluated code we may be used to.

```
(defparameter *blocksp*
 (let* ((in-block nil))
   (mapping
    ((maybe-latch *codeblock-toggles*)
     (line *filein*))
    (if in-block
        (if maybe-latch
            (setq in-block (not in-block))
            line)
        (and maybe-latch
             (setq in-block (not in-block))
             nil)))))
```

Well, let's do an [assert](https://www.lispworks.com/documentation/HyperSpec/Body/m_assert.htm)ion.

```
(assert (equal "(lisp code)"
              (collect-nth 3 *blocksp*))
       nil
       "~a should be \"(lisp code)\" in the test example."
       (collect-nth 3 *blocks*))
```

I guess we just want the lines, and [`write-line`](https://www.lispworks.com/documentation/HyperSpec/Body/f_wr_stg.htm)ed to a file.

```
(collect-file
"/tmp/test-tangle.lisp"
(choose *blocksp* *blocksp*)
'write-line)
```

# The [tangle](https://www.cs.tufts.edu/~nr/cs257/archive/literate-programming/01-knuth-lp.pdf) function

Let's just trivially collect our test assert case into a function.

```
(defun tangle
   (md-path system
    &aux
      (fileout (make-pathname
                :name (pathname-name md-path)
                :type "lisp"
                :directory
                (pathname-directory
                 (system-source-directory system)))))
 (let* ((filein (scan-file md-path #'read-line))
        (codeblock-toggles (#Mstring= (series "```")
                                      filein))
        (blocksp
          (let ((in-block nil))
            (mapping
             ((maybe-latch codeblock-toggles)
              (line filein))
             (if in-block
                 (if maybe-latch
                     (setq in-block (not in-block))
                     line)
                 (and maybe-latch
                      (setq in-block (not in-block))
                      nil))))))
   (collect-file
    fileout
    (choose blocksp blocksp)
    'write-line)))
```

# System directory stubbing

I guess one should probably not want this. Still, here it is.

```
(defun stub-system
   (namestring)
 (let* ((homedir (user-homedir-pathname))
        (dir (make-pathname :directory
                            `(:relative ,namestring)))
        (lispdir (make-pathname :directory
                                '(:relative "common-lisp")))
        (sysdir
          (reduce 'merge-pathnames
                  (list dir lispdir homedir)))
        (asd
          (make-pathname :name namestring
                         :type "asd"
                         :directory '(:relative :wild)))
        (system-asd
          (make-pathname
           :directory (pathname-directory sysdir)
           :name (pathname-name asd)
           :type (pathname-type asd))))
   (ensure-directories-exist sysdir)
   (with-open-file
       (*standard-output* system-asd
                          :direction :output
                          :if-does-not-exist :create
                          :if-exists :error)
     (format t
             "(defsystem ~a ~
               :class :package-inferred-system)"
             (intern namestring :keyword))
     (probe-file
      (make-pathname
       :directory (pathname-directory sysdir)
       :name (pathname-name asd)
       :type (pathname-type asd))))))
```

# Example

```
(eval-when
   ()
 (stub-system "foo")
 (tangle #p"/tmp/test-tangle.md"
         "foo")
 )
```

it will error if `#p"~/common-lisp/foo/foo.asd"` already exists. It works 😺

# Conclusion

While my intended focus was the first two, I found I usefully covered three topics here:

1. Tangling markdown into an `asdf` `:class :package-inferred-system` lisp system
1. Doing so with `scan-file` and `collect-file` from `series`
1. Working with lisp's `make-pathname` directories.

I do think that `make-pathname` is more sophisticated than primitive string-processing style modern convention pathnames. Hope it helps someone get used to these three topics.

Currently works on my machine!

# Fin.

I guess this article is somehow both very simple and very intricate. I would like to hear your improvements, alternatives, or memories of what-you-would-do on [the mastodon thread](https://gamerplus.org/@screwlisp/114771339388871919).

Because our article shows all of

- interactive, incremental, reflective functional programming with cl-series
- a reasonable useage idiom of lisp's sophisticated native pathname system
- working with asdf systems in a literate residential source manner

as always, I would appreciate you sharing it in any particular means and manner that occurs to you personally.

Small note: When I used the system on itself, my markdown file was named `#p"tangle.page.md"` so the lisp file it generated from itself used metacircularly was `#p"~/common-lisp/screwlisps-knowledge/tangle.page.lisp"` - I didn't think of how I want to deal with other programms "subtyping" file types like this, so I just fixed it directly for now. If you don't subtype files inside the filename, I guess it just works.