* * * * *

 Why one might want to load dynamic objects from memory instead of from disk

Years ago I started on a little project entitled the LEM Project [1]. It was
a project to make distribution of applications written in Lua [2] easier
(It's named after the Apollo Lunar Module [3]—aka (also known as) the Lunar
Excursion Module; Lua is “moon” in Portuguese). I had a few goals:

 1. allow distribution of multiple versions of a Lua module;
 2. allow for different architectures for the same module written in C;
 3. run the entire application from the distribution file (nothing to
    install).

And I almost hit all the goals.

The first two were easy. I read over the ZIP file specification [4] and saw
that

 1. multiple files of the same name were entirely possible—nothing in the
    specification disallows it;
 2. additional metadata can be added to each file beyond what is defined in
    the specification.

So armed, I created zip file with some addtional metadata and …

-----[ shell ]-----
[spc]lucy:~/projects/LEM>./zipf.lua sample.lem
The Eagle Has Landed
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.env
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.errno
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.1    MODULES/org.conman.iconv
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.sys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.env
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.errno
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys.magic
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
Linux     x86       LGPL3+       Lua 5.1 5.1 1.1.1  MODULES/org.conman.iconv
Linux     x86       LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net.ipacl
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
Linux     x86       LGPL3+       Lua 5.1 5.1 1.2.0  MODULES/org.conman.sys
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.2  MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.tcc
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.cc
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.date
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.debug
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.dns.resolv
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.getopt
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.string
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.table
                   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.unix
                   MIT          Lua 5.1 5.1 0.10   MODULES/lpeg
                   MIT          Lua 5.1 5.1 0.10   MODULES/re
Linux     x86       MIT          Lua 5.1 5.1 0.12   MODULES/lpeg
Linux     x86       MIT          Lua 5.2 5.2 0.12   MODULES/lpeg
                   MIT          Lua 5.1 5.2 0.12   MODULES/re
Linux     x86       MIT/X11      Lua 5.1 5.1 0.4.work3 MODULES/zlib
                   MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket
Linux     x86       MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.core
                   MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.ftp
                   MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.http
                   MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.smtp
                   MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.tp
                   MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.url
                   MIT/X11      Lua 5.1 5.1 1.0.1  MODULES/ltn12
                                                   FILES/APPNOTE.TXT
                                                   FILES/COPYING
                                                   FILES/README
                                                   FILES/Miscellaneous Things About Nothing
[spc]lucy:~/projects/LEM>
-----[ END OF LINE ]-----

You can see there are several copies of each module. The modules listed
without a system (like Linux or SunOS) are written in Lua; the other ones are
in C and both the system and architecture are listed. The license is
included, along with the Lua verions the module will work for (it's “minimum
version” and “maximum version”) as well as the version of the module (if
known).

It's even a valid zip file:

-----[ shell ]-----
[spc]lucy:~/projects/LEM>unzip -l sample.lem
Archive:  sample.lem
The Eagle Has Landed
 Length     Date   Time    Name
--------    ----   ----    ----
   25472  05-24-14 17:10   MODULES/org.conman.base64
   13448  05-24-14 17:10   MODULES/org.conman.crc
   12200  05-24-14 17:10   MODULES/org.conman.env
   18688  05-24-14 17:10   MODULES/org.conman.errno
   57032  05-24-14 17:10   MODULES/org.conman.fsys
   24952  05-24-14 17:10   MODULES/org.conman.hash
   17664  05-24-14 17:10   MODULES/org.conman.iconv
   17648  05-24-14 17:10   MODULES/org.conman.math
   77944  05-24-14 17:10   MODULES/org.conman.net
   26296  05-24-14 17:10   MODULES/org.conman.pollset
   88256  05-24-14 17:10   MODULES/org.conman.process
   37848  05-24-14 17:10   MODULES/org.conman.strcore
   15392  05-24-14 17:10   MODULES/org.conman.sys
   24312  05-24-14 17:10   MODULES/org.conman.syslog
   14175  06-12-14 22:04   MODULES/org.conman.base64
    8214  06-12-14 22:04   MODULES/org.conman.crc
    7193  06-12-14 22:04   MODULES/org.conman.env
   10690  06-12-14 22:04   MODULES/org.conman.errno
   31885  06-12-14 22:04   MODULES/org.conman.fsys
   15567  06-12-14 22:04   MODULES/org.conman.fsys.magic
   14067  06-12-14 22:04   MODULES/org.conman.hash
   10197  06-12-14 22:04   MODULES/org.conman.iconv
   10816  06-12-14 22:04   MODULES/org.conman.math
   43651  06-12-14 22:04   MODULES/org.conman.net
   25248  04-18-14 20:06   MODULES/org.conman.net.ipacl
   15914  06-12-14 22:04   MODULES/org.conman.pollset
   49607  06-12-14 22:04   MODULES/org.conman.process
   15666  06-12-14 22:04   MODULES/org.conman.strcore
    9763  06-12-14 22:04   MODULES/org.conman.sys
   13303  06-12-14 22:04   MODULES/org.conman.syslog
   21218  06-12-14 22:04   MODULES/org.conman.tcc
    5617  06-12-14 22:04   MODULES/org.conman.cc
    2500  06-12-14 22:04   MODULES/org.conman.date
    3829  06-12-14 22:04   MODULES/org.conman.debug
    2966  06-12-14 22:04   MODULES/org.conman.dns.resolv
    3260  06-12-14 22:04   MODULES/org.conman.getopt
    2464  06-12-14 22:04   MODULES/org.conman.string
    5243  06-12-14 22:04   MODULES/org.conman.table
    2732  06-12-14 22:04   MODULES/org.conman.unix
   40081  05-25-14 15:17   MODULES/lpeg
    6029  05-24-14 00:36   MODULES/re
   40045  05-28-14 15:24   MODULES/lpeg
   40045  05-28-14 15:24   MODULES/lpeg
    6286  05-28-14 15:24   MODULES/re
   19794  05-30-14 21:29   MODULES/zlib
    4451  05-28-14 14:52   MODULES/socket
   55449  05-28-14 14:52   MODULES/socket.core
    9243  05-28-14 14:52   MODULES/socket.ftp
   12330  05-28-14 14:52   MODULES/socket.http
    8074  05-28-14 14:52   MODULES/socket.smtp
    3612  05-28-14 14:52   MODULES/socket.tp
   11036  05-28-14 14:52   MODULES/socket.url
    8331  05-28-14 14:52   MODULES/ltn12
  161412  05-09-14 01:24   FILES/APPNOTE.TXT
The ZIP file format
    7651  05-25-14 18:53   FILES/COPYING
    9789  06-07-14 22:13   FILES/README
Much Ado About Nothing
    3946  05-10-99 23:00   FILES/Miscellaneous Things About Nothing
If you this this has significance, think otherwise
--------                   -------
 1250541                   57 files
[spc]lucy:~/projects/LEM>
-----[ END OF LINE ]-----

(Although if you try to unzip this file, the unzip program will ask you what
you want to do with the duplicate files.)

The intent was that only those modules that match the Lua version, system (if
a C-based Lua module) and architecture (again, C-base Lua module) match, load
the module; otherwise, it would be ignored.

And it worked. For the most part.

I could load Lua modules written in Lua directly from the zip file with no
problem. I even got LuaRocks [5] to run completely from the zip file (with
some hacks so it could load the configuration file). But I could not do that
for the C-based Lua modules. For those, you had to write them out to disk to
load them up. And because of that, I lost interest in continuing the project.

Until now. I just came across MojoELF [6] (via Project: 2ine [7] which was
linked via Hacker News [8])—a project to load ELF (Executable and Linkable
Format) binaries directly from memory (and those C-based Lua modules I have
are ELF binaries), which is exactly what I wanted all those years ago.

[1] https://github.com/spc476/LEM
[2] https://www.lua.org/
[3] https://en.wikipedia.org/wiki/Apollo_Lunar_Module
[4] https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
[5] https://luarocks.org/
[6] https://hg.icculus.org/icculus/mojoelf/file/tip/README.txt
[7] https://www.patreon.com/posts/project-2ine-16513790
[8] https://news.ycombinator.com/item?id=16409584

Email author at [email protected]

You are viewing proxied material from gopher.conman.org. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.