Last week I upgraded the OS at the Zaibatsu (if any sundogs have seen
scary warnings from ssh/scp about changed keys, this is the reason
why! It's safe to connect - email me if you are extra paranoid and
want to confirm the new fingerprint). We had been running the stable
release of Debian 8 (Stretch) since the server was setup over two
years ago. Stretch is pretty darn old now, and with Debian stable
typically shipping slightly old software from day one, we were well
behind the times. Which, for the most part, I imagine was absolutely
fine by most people involved. We're not exactly all about being at
the cutting edge here!
But with very old versions of programming languages installed it was
becoming increasingly common for new projects to not run smoothly,
and upgrades to security libraries like OpenSSL are always a good
thing. So I've been ready to uprade for a while, but have had to
wait for the VPS provider RamNode to be ready as well. The Zaibatsu
is an OpenVZ VPS, meaning it shares a running Linux kernel with
several other VPSes (virtualisation magic keeps the processes,
filesystems etc. of one VPS invisible from the others). Because we
don't have our own independent kernel, we can't upgrade whenever we
like (or fiddle with most sysctl settings, either). We can only run
the kernels that RamNode provide. However, RamNode semi-recently
updated their OpenVZ infrastructure to a non-legacy version, meaning
they were at last able to offer some more modern OS options, including
Debian 10 (Buster). However, there's no possibility for a typical
"in place" upgrade via the standard `apt` tooling. I basically had
to back the whole thing up, re-image the Zaibatsu with Buster,
reinstall all the software and restore our files. The recent May Day
long weekend gave me a good opportunity to do this with enough time to
hopefully fix any issues if things really went South.
For the most part, things went pretty smoothly. No data was lost, and
I prioritised getting the gopher and mail servers running again ASAP
so that the visible downtime to the rest of the world with regard to
our basic services was very short. I'm glad I did this with a lot of
spare time available, though, because it took me what seemed like an
eternity to get our BBS working again.
The Circumlunar BBS works as follows (it's not proprietary software
with unknown inner workings like SDF's BBOARD is!): all posts are
stored in a board/thread/post directory structure in /var/bbs, which
is owned by a "bbs" user. The content is world readable, so if you
really want you can poke around and read everything with cd, ls and
cat/less/more. Or write your own read-only client, or search engine,
or whatever. To post, you have to use one of the approved clients,
which are setuid and owned by the bbs user, enabling them to create
new files in /var/bbs. These clients ensure that people's correct
usernames are attached to posts - if we just made /var/bbs
world-writable, people could impersonate other users, or vandalise
other user's posts. Not that I think any sundogs would do that, but
at some point we hoped other pubnixes would pick this software up so
we didn't want to necessarily assume a small, high-trust society.
There's nothing super innovative about this - take a look in
/usr/games and you'll find many of those binaries are setgid and have
their group set to "games". This is precisely to allow any user on a
multiuser unix system update a shared high score file in a controlled
way. Even though the games (and our BBS clients) are free software,
if a user cloned the repo and modified the game to give them free
points, without root access they can't make their new hax0red binary
owned by the "games" group, so they can't cheat their way to the top
of the scoreboard.
The main BBS client we use is written in Lua, because of its low
memory footprint (we have 128MB of RAM to share amongst all logged in
users!). Now, running interpreted programs (or shell scripts) set
setuid or setgid is not straightforward. Because setuid programs
which are owned by the root user are potentially big security holes,
and because shell scripts in particular are hard to guarantee the
security of because users can easily influence their behaviour via
environment variables, aliases, etc., most modern unixes do not allow
shells or interpreters to run setuid, and this extends to the Lua
interpreter. The way around this is to write a small "wrapper" in a
compiled language like C, which does nothing but call the interpeter
you want with the required arguments. The resulting binary wrapper
can be made setuid/setgid no problems.
People online will scream at you that you are going directly to
sysadmin hell if you attempt this, but they are being lazy: many, many
people seem to have forgotten that setuid binaries are not necessarily
owned by root. If they are owned by a low-privilege user/group like
"games" or "bbs" then the risk is comparatively minimal and there's no
reason to freak out. Most likely, in this day and age where using
unix as an actual, genuine multi-user system is seen more as quaint
historical re-enactment than serious computing, they're unable to
conceive of what the point would be in a non-root setuid program.
Heck, maybe they don't even have a /usr/games/ directory!!!
Anyway, we have just such a setuid binary wrapper for our BBS client,
and for whatever reason it was not working correctly after the
upgrade, leaving the BBS in a read-only state. I went nuts trying to
figure out why it had stopped working, verifying that it was owned by
the correct user, that the setuid bit was set, etc. Eventually I
discovered that between the versions of dash (Debian's default
/bin/sh) shipped with Stretch and with Buster, a change was made where
the first thing the shell does is check whether its true and effective
uid are equal and, if they aren't, drops priveleges so that they are.
Because the binary wrapper used the C standard library's `system()`
function to lauch the Lua interpeter, and because `system()` uses
/bin/sh to launch things, we were ending up with a non-priveleged Lua.
This change to dash was made "for security reasons". It really
pisses me off that nobody thought to implement this feature in such a
way that the shell checks whether its effective uid is 0 and only
drops privileges in such a case, realising that the threat from
non-root setuids is minior.
At first I thought this would be an easy fix - I changed the wrapper
to use one of the `exec()` functions instead of `system()` so that
/bin/sh was not invoked. This resulted in a privelged Lua interpeter!
However, because Lua has no built-in filesystem support, the client
relies crucially on launching standard unix tools like cp, mv, rm,
chown and chmod to maintain all the correct permissions in /var/bbs in
a secure way. And Lua launches those tools...using `system()`, and
hence /bin/sh. So the problem was only partially solved.
In the end, I hunted down and installed an older version of dash which
didn't drop privileges and everything immediately just worked. It's
not an ideal solution, but the BBS is a central part of our community
and I wanted it up and running ASAP. However, this situation does
pretty well rule out the notion of this system becoming widely
deployed at other pubnixes. I will have to see if I can either get
sudo to function as an alternative binary wrapper (this is widely
advised online, but I have tried previously to apply it to our
situation and although I've forgotten the details I recall that there
was a very real and principled reason why it didn't work for us), or
if we can reimplement the client in a language which compiles to a
binary. I'm not optimistic on that last part, though. Nobody wants
to write something like this in C in this day and age. However,
modern compiled languages tend to have extremely smart and
high-powered toolchains (in order to manage the dependency hell that
modern software development fashion seems to eagerly embrace) which
don't play well in a low-resource pubnix environment. Rust wants
every single user to have a multi-hundred megabyte copy of the
toolchain in their home directory, and "Hello, world" has about 250
dependencies - one of which was always released only yesterday. Go
can be installed system wide in the traditional sane manner, but
128MB of RAM is not enough to compile anything I've tried. Sigh...
Aside from this saga, all other problems have been pretty minor.
Systemd adoption is much more widespread in Buster than it was in
Stretch, but I'm trying to see this as a learning opportunity rather
than a hassle. We are now no longer using xinet to run some things
that we previously were, they are now implemented as systemd
socket-activated services. I still think the death of /etc/rc.local
is as a dirt-simple place-of-last-resort to stick system
initialisation code is a tragedy, but I'll spare any further
complaining until a later post at least. Basically I think we are
now right back to where we were, in terms of everything working, but
with a much more modern environment which isn't frighteningly close
to EOL. Hopefully I don't have to do this again for another few
years!
