Title: Introduction to immutable Linux systems | |
Author: Solène | |
Date: 12 July 2023 | |
Tags: immutability linux | |
Description: In this article, you will learn what stands behind the | |
label immutable when it comes to Linux systems | |
# Introduction | |
If you reach this page, you may be interested into this new category of | |
Linux distributions labeled "immutable". | |
In this category, one can find by age (oldest → youngest) NixOS, | |
Guix, Endless OS, Fedora Silverblue, OpenSUSE MicroOS, Vanilla OS and | |
many new to come. | |
I will give examples of immutability implementation, then detail my | |
thoughts about immutability, and why I think this naming can be | |
misleading. I spent a few months running all of those distributions on | |
my main computers (NAS, Gaming, laptop, workstation) to be able to | |
write this text. | |
# What's immutability? | |
The word immutability itself refers to an object that can't change. | |
However, when it comes to an immutable operating system, the definition | |
immediately become vague. What would be an operating system that can't | |
change? What would you be supposed to do with it? | |
We could say that a Linux LIVE-CD is immutable, because every time you | |
boot it, you get the exact same programs running, and you can't change | |
anything as the disk media is read only. But while the LIVE-CD is | |
running, you can make changes to it, you can create files and | |
directories, install packages, it's not stuck in an immutable state. | |
Unfortunately, this example was nice but the immutability approach by | |
those Linux distribution is totally different, so we need to think a | |
bit further. | |
There are three common principles in these systems: | |
* system upgrades aren't done on the live system | |
* packages changes are applied on the next boot | |
* you can roll back a change | |
Depending on the implementation, a system may offer more features. But | |
this list is what a Linux distribution should have to be labelled | |
"immutable" at the moment. | |
# Immutable systems comparison | |
Now we found what are the minimum requirements to be called immutable, | |
let's go through each implementation, by their order of appearance. | |
## NixOS / Guix | |
In this section, I'm mixing NixOS and Guix as they both rely on the | |
same implementation. NixOS is based on Nix (first appearance in 2003), | |
which has been forked into early 2010s into the Guix package manager to | |
be 100% libre, which gave birth to an eponym operating system also 100% | |
free. | |
NixOS official project website | |
Guix official project website | |
Jonathan Lorimer's blog post explaining Eelco Dolstra's thesis about Nix | |
These two systems are really different than a traditional Unix like | |
system we are used to, and immutability is a main principle. To make | |
it quick, they are based on their package manager (being Nix or Guix) | |
that contains every package or built file into a special read-only | |
directory (where only the package manager can write) where each package | |
has its own unique entry, and the operating system itself is a | |
byproduct of the package manager. | |
What does that imply? If the operating system is built, this is | |
because it's made of source code, you literally describe what you want | |
your system to be in a declarative way. You have to list users, their | |
shells, installed packages, running services and their configurations, | |
partitions to mount with which options etc... Fortunately, it's made a | |
lot easier by the use of modules which provide sane defaults, so if you | |
create a user, you don't have to specify its UID, GID, shell, home | |
etc... | |
So, as the system is built and stored in the special read-only | |
directory, all your system is derived from that (using symbolic links), | |
so all the files handled by the package manager are read-only. A | |
concrete example is that /etc/fstab or /bin/sh ARE read-only, if you | |
want to make a change in those, you have to do it through the package | |
manager. | |
I'm not going into details, because this store based package manager is | |
really different than everything else but: | |
* you can switch between two configurations on the fly as it's just a | |
symlink dance to go from a configuration to another | |
* you can select your configuration at boot time, so you can roll back | |
to a previous version if something is wrong | |
* you can't make change to a package file or system file as they are | |
read only | |
* the mount points except the special store directory are all mutable, | |
so you can write changes in /home or /etc or /var etc... You can remove | |
the system symlinks by a modified version, but you can't modify the | |
symlink source itself. | |
This is the immutability as seen through the Nix lens. | |
I've spent a few years running NixOS systems, this is really a blast | |
for me, and the best "immutable" implementation around, but | |
unfortunately it's too different, so its adoption rate is very low, | |
despite all the benefits. | |
NixOS forum: My issues when pushing NixOS to companies | |
## Endless OS | |
While this one is not the oldest immutable OS around, it's the first | |
one to be released for the average user, while NixOS and Guix are older | |
but for a niche user category. The company behind Endless OS is trying | |
to offer a solid and reliable system, free and open source, that can | |
works without Internet, to be used in countries with a low Internet / | |
powergrid coverage. They even provide a version with "offline internet | |
included" containing Wikipedia dumps, class lessons and many things to | |
make a computer useful while offline (I love their work). | |
Endless OS official project website | |
Endless OS is based on Debian, but uses the OSTree tool to make it | |
immutable. OSTree allows you to manage a core system image, and add | |
layers on top of it, think of packages as layers. But it can also | |
prepare a new system image for the next boot. | |
With OSTree, you can apply package changes in a new version of the | |
system that will be available at next boot, and revert to a previous | |
version at boot time. | |
The partitions are mounted writable, except for `/usr`, the land of | |
packages handled by OSTree, which is mounted read-only. There are no | |
rollbacks possible for `/etc`. | |
Programs meant to be for the user (not the packages to be used by the | |
system like grub, X display or drivers) are installed from Flatpak | |
(which also uses OSTree, but unrelated to the system), this avoids the | |
need to reboot each time you install a new package. | |
My experience with Endless OS is mixed, it is an excellent and solid | |
operating system, it's working well, never failed, but I'm just not the | |
target audience. They provide a modified GNOME desktop that looks like | |
a smartphone menu, because this is what most non-tech users are | |
comfortable with (but I hate it). And installing DevOps tools isn't | |
practical but not impossible, so I keep Endless OS for my multimedia | |
netbook and I really enjoy it. | |
## Fedora Silverblue | |
This linux distribution is the long descendant of Project Atomic, an | |
old initiative to make Fedora / CentOS/ RHEL immutable. It's now part | |
of the Fedora releases along with Fedora Workstation. | |
Project Atomic website | |
Fedora Silverblue project website | |
Fedora Silverblue is also using OSTree, but with a twist. It's using | |
rpm-OSTree, a tool built on top of OSTree to let your RPM packages | |
apply the changes through OSTree. | |
The system consists of a single core image for the release, let's say | |
fedora-38, and for each package installed, a new layer is added on top | |
of the core. At anytime, you can list all the layers to know what | |
packages have been installed on top of the core, if you remove a | |
package, the whole stack is generated again (which is terribly SLOW) | |
without the package, there is absolutely no leftover after a package | |
removal. | |
On boot, you can choose an older version of the system, in case | |
something broke after an upgrade. If you install a package, you need | |
to reboot to have it available as the change isn't applied on the | |
current booted system, however rpm-OSTree received a nice upgrade, you | |
can temporarily merge the changes of the next boot into the live system | |
(using a tmpfs overlay) to use the changes. | |
The mountpount management is a bit different, everything is read-only | |
except `/etc/`, `/root` and `/var`, but your home directory is by | |
default in `/var/home` which sometimes breaks expectations. There are | |
no rollbacks possible for `/etc`. | |
As installing a new package is slow due to rpm-OSTree and requires a | |
reboot to be fully usable (the live change back port store the extra | |
changes in memory), they recommend to use Flatpak for programs, or | |
`toolbox`, some kind of wrapper that create a rootless fedora container | |
where you can install packages and use it in your terminal. toolbox is | |
meant to provide development libraries or tool you wouldn't have in | |
Flatpak, but that you wouldn't want to install in your base Fedora | |
system. | |
toolbox website | |
My experience with Fedora Silverblue has been quite good, it's stable, | |
the updates are smooth even if they are slow. `toolbox` was working | |
fine despite I don't find this practical. | |
## OpenSUSE MicroOS | |
This spin of OpenSUSE Tumbleweed (rolling-release OpenSUSE) features | |
immutability, but with its own implementation. The idea of MicroOS is | |
really simple, the whole system except a few directories like `/home` | |
or `/var` lives on a btrfs snapshot, if you want to make a change to | |
the system, the current snapshot is forked into a new snapshot, and the | |
changes are applied there, ready for the next boot. | |
OpenSUSE MicroOS official project website | |
What's interesting here is that `/etc` IS part of the snapshots, and | |
can be roll backed, which wasn't possible in the OSTree based systems. | |
It's also possible to make changes to any file of the file system (in a | |
new snapshot, not the live one) using a shell, which can be very | |
practical for injecting files to solve a driver issue. The downside | |
it's not guaranteed that your system is "pure" if you start making | |
changes, because they won't be tracked, the snapshots are just | |
numbered, and you don't know what changes were made in each of them. | |
Changes must be done through the command `transactional-update` which | |
do all the snapshot work for you, and you could either manipulate | |
package by adding/removing a package, or just start a shell in the new | |
snapshot to make all the changes you want. I said `/etc` is part of | |
the snapshots, it's true, but it's never read-only, so you could make a | |
change live in `/etc`, then create a new snapshot, the change would be | |
immediately inherited. This can create troubles if you roll back to a | |
previous state after an upgrade if you also made changes to `/etc` just | |
before. | |
The default approach of MicroOS is disturbing at first, a reboot is | |
planned every day after a system update, this is because it's a | |
rolling-release system and there are updates every day, and you won't | |
benefit from them until you reboot. While you can disable this | |
automatic reboot, it makes sense to use the newest packages anyway, so | |
it's something to consider if you plan to use MicroOS. | |
There is currently no way to apply the changes into the live system | |
(like Silverblue is offering), it's still experimental, but I'm | |
confident this will be doable soon. As such, it's recommended to use | |
`distrobox` to use rootless containers of various distributions to | |
install your favorite tools for your users, instead of using the base | |
system packages. I don't really like this because this adds | |
maintenance, and I often had issues of distrobox refusing to start a | |
container after a reboot, I had to destroy and recreate it entirely to | |
solve. | |
distrobox GitHub project page | |
My experience with OpenSUSE MicroOS has been wonderful, it's in | |
dual-boot with OpenBSD on my main laptop, it's my Linux Gaming OS, and | |
it's also my NAS operating system, so I don't have to care about | |
updates. I like that the snapshots system doesn't restrict me, while | |
OSTree systems just doesn't allow you to make changes without | |
installing a package. | |
## Vanilla OS | |
Finally, the really new (but mature enough to be usable) system in the | |
immutable family is Vanilla OS based on Ubuntu (but soon on Debian), | |
using ABroot for immutability. With Vanilla OS, we have another | |
implementation that really differs from what we saw above. | |
Vanilla OS project website | |
ABroot named is well thought, the idea is to have a root partition A, | |
another root partition B, and a partition for persistent data like | |
`/home` or `/var`. | |
Here is the boot dance done by ABroot: | |
* first boot is done on A, it's mounted in read-only | |
* changes to the system like new packages or file changes in `/etc` are | |
done on B (and can be applied live using a tmpfs overlay) | |
* upon reboot, if previous boot was A, you boot on B, then if the boot | |
is successful, ABroot scan for all the changes between A and B, and | |
apply all the changes from B to A | |
* when you are using your system, until you make a change, A and B are | |
always identical | |
This implementation has downsides, you can only roll back a change | |
until you boot on the new version, then the changes are also applied on | |
the previous boot, and you can't roll back. This implementation mostly | |
protects you from a failing upgrade, or if you made changes and tried | |
them live, but you prefer to rollback. | |
Vanilla OS features the package manager apx, written by distrobox | |
author. That's for sure an interesting piece of software, allowing | |
your non-root user to install packages from many distributions (arch | |
linux, fedora, ubuntu, nix, etc...) and integrates them into the system | |
as if they were installed locally. I suppose it's some kind of layer | |
on top of distrobox. | |
apx package manager GitHub project page | |
My experience wasn't very good, I didn't find ABroot to be really | |
useful, and the version 22.10 I tried was using an old Ubuntu LTS | |
release which didn't make my gaming computer really happy. The overall | |
state of Vanilla OS, ABroot and apx is that they are young, I think it | |
can become a great distribution, but it still has some rough edges. | |
## Alpine Linux (with LBU) | |
I've been told that it was possible to achieve immutability on Alpine | |
Linux using the "lbu" command. | |
Alpine Linux wiki: Local backup | |
I don't want to go much into details, but here is the short version: | |
you can use Alpine Linux installer as a base system to boot from, and | |
create tarballs of "saved configurations" that are automatically | |
applied upon boot (it's just tarred directories and some automation to | |
install packages). At every boot, everything is untarred again, and | |
packages are installed again (you should use an apk cache directory), | |
everything in live memory, fully writable. | |
What does this achieve? You always start from a clean state, changes | |
are applied on top of it at every boot, you can roll back the changes | |
and start fresh again. Immutability as we defined above here isn't | |
achieved because changes are applied on the base system, but it's quite | |
close to fulfill (my own) requirements. | |
I've been using it a few days only, not as my main system, and it | |
requires a very good understanding of what you are doing because the | |
system is fully in memory, and you need to take care about what you | |
want to save/restore, which can create big archives. | |
On top of that, it's poorly documented. | |
# Pros, Cons and Facts | |
Now I gave some details about all the major immutable systems (Linux | |
based) around, I think it's time to list the real pros and cons I found | |
from my experimentation. | |
## Pros | |
* you can roll back changes if something went wrong. | |
* transactional-updates allows you to keep the system running correctly | |
during packages changes. | |
## Cons | |
* configuration management tool (ansible, salt, puppet etc..) integrate | |
VERY badly, they received updates to know how to apply package changes, | |
but you will mostly hit walls if you want to manage those like regular | |
systems. | |
* having to reboot after a change is annoying (except for NixOS and | |
Guix which don't require rebooting for each change). | |
* OSTree based systems aren't flexible, my netbook requires some extra | |
files in alsa directories to get sound (fortunately Endless OS have | |
them!), you just can't add the files without making a package deploying | |
them. | |
* blind rollbacks, it's hard to figure what was done in each version of | |
the system, so when you roll back it's hard to figure what you are | |
doing exactly. | |
* it can be hard to install programs like Nix/Guix which require a | |
directory at the root of the file system, or install non-packaged | |
software system-wide (this is often bad practice, but sometimes a | |
necessary evil). | |
## Facts | |
* immutability is a lie, many parts of the systems are mutable, | |
although I don't know how to describe this family with a different word | |
(transactional something?). | |
* immutable doesn't imply stateless. | |
* NixOS / Guix are doing it right in my opinion, you can track your | |
whole system through a reliable package manager, and you can use a | |
version control system on the sources, it has the right philosophy from | |
the ground up. | |
* immutability is often associated with security benefits, I don't | |
understand why. If someone obtains root access on your system, they | |
can still manipulate the live system and have fun with the `/boot` | |
partition, nothing prevent them to install a backdoor for the next | |
boot. | |
* immutability requires discipline and maintenance, because you have to | |
care about the versioning, you have extra programs like apx / distrobox | |
/ devbox that must be updated in parallel of the system (while this is | |
all integrated into NixOS/Guix). | |
# Conclusion | |
Immutable operating systems are making the news in our small community | |
of open source systems, but behind this word lies many implementations | |
with different use cases. The word immutable certainly creates | |
expectations from users, but it's really nothing more than | |
transactional updates for your operating system, and I'm happy we can | |
have this feature now. | |
But transactional updates aren't new, I think it started a while ago | |
with Solaris and ZFS allowing you to select a system snapshot at boot | |
time, then I'm quite sure FreeBSD implemented this a decade ago, and it | |
turns out that on any linux distribution with regular btrfs snapshots | |
you could select a snapshot at boot time. | |
Previous blog post about booting on a BTRFS snapshot without any special setup | |
In the end, what's REALLY new is the ability to apply a transactional | |
change on a non-live environment, integrates this into the bootloader, | |
and give the user the tooling to handle this easily. | |
# Going further | |
I recommend reading the blog post "“Immutable” → reprovisionable, | |
anti-hysteresis" by Colin Walters. | |
“Immutable” → reprovisionable, anti-hysteresis |