Title: How to get NixOS hosted at OpenBSD Amsterdam | |
Author: Solène | |
Date: 07 August 2022 | |
Tags: nixos openbsd hosting | |
Description: This article explains how to host a NixOS VM at the | |
hosting company OpenBSD Amsterdam which only provides OpenBSD systems. | |
# Introduction | |
In this guide, I'll explain how to create a NixOS VM in the hosting | |
company OpenBSD Amsterdam which only provides OpenBSD VMs hosted on | |
OpenBSD. | |
I'd like to thank the team at OpenBSD Amsterdam who offered me a VM for | |
this experiment. While they don't support NixOS officially, they are | |
open to have customers running non-OpenBSD systems on their VMs. | |
OpenBSD Amsterdam hosting service website | |
# The steps from OpenBSD to NixOS | |
Here is a short description of the steps required to get NixOS | |
installed on OpenBSD Amsterdam. | |
1. Generate a NixOS VM disk file or use the one I provide | |
2. Rent a VM at OpenBSD Amsterdam (5€ / month for 1 vCPU, 1GB of | |
memory and 50 GB of hdd, with a dedicated IPv4, working IPv6 and | |
reverse DNS) | |
3. Connect to the hypervisor in order to get the serial console access | |
to your VM | |
4. Connect with ssh to your VM to reboot it | |
5. In the serial console, upon reboot, boot on bsd.rd (the OpenBSD | |
installer ramdisk) | |
6. Overwrite the local disk by fetching your NixOS VM disk file through | |
http/ftp and writing it on the disk file | |
7. Reboot on NixOS | |
8. Configure the network from the serial console, rebuild the system | |
9. Enjoy | |
# How to proceed | |
You need to order a VM at OpenBSD Amsterdam first. You will receive an | |
email with your VM name, its network configuration (IPv4 and IPv6), and | |
explanations to connect to the hypervisor. We will need to connect to | |
the hypervisor to have a serial console access to the virtual machine. | |
A serial console is a text interface to a machine, you get the machine | |
output displayed in your serial console client, and what you type is | |
sent to the machine as if you had a keyboard connected to it. | |
It can be useful to read the onboarding guide before starting. | |
OpenBSD Amsterdam onboarding guide | |
## Get into the OpenBSD installer | |
Our first step is to get into the OpenBSD installer, so we can use it | |
to overwrite the disk with our VM. | |
Connect to the hypervisor, attach to your virtual machine serial | |
console by using the following command, we admit your VM name is "vm40" | |
in the example: | |
```shell | |
vmctl console vm40 | |
``` | |
You can leave the console anytime by typing "~~." to get back into your | |
ssh shell. The keys sequence "~." is used to drop ssh or a local | |
serial console, but when you need to leave a serial console from a ssh | |
shell, you need to use "~~.". | |
You shouldn't see anything because you won't get anything displayed | |
until something is showed in the machine virtual first tty, you can | |
press "enter" and you should see a login prompt. We don't need it, but | |
it confirms the serial console is working. | |
In parallel, connect to your VM using ssh, find the root password at | |
the end of ~/.ssh/authorized_keys, use "su -" to become root and run | |
"reboot". | |
You should see the shutdown sequence scrolling in the hypervisor ssh | |
session displaying the serial console, wait for the machine to reboot | |
to spot for the login prompt, in which you will type bsd.rd: | |
```text | |
Using drive 0, partition 3. | |
Loading...... | |
probing: pc0 com0 mem[638K 3838M 4352M a20=on] | |
disk: hd0+ | |
>> OpenBSD/amd64 BOOT 3.53 | |
com0: 115200 baud | |
switching console to com0 | |
>> OpenBSD/amd64 BOOT 3.53 | |
boot> bsd.rd [ENTER] # you need to type bsd.rd | |
``` | |
## Copy the NixOS VM from the installer | |
In this step, we will use the installer to fetch the NixOS VM disk and | |
overwrite the local disk with it. | |
* in the installer, type "S" to get a shell: | |
```text | |
[...] | |
Welcome to the OpenBSD/amd64 7.2 installation program. | |
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? | |
``` | |
* enable the network using DHCP with the command: | |
```shell | |
ifconfig vio0 up autoconf | |
``` | |
* create the disk device in /dev because it's missing by default: | |
```shell | |
cd /dev | |
sh MAKEDEV sd0 | |
``` | |
* fetch the NixOS disk and overwrite the local drive with it: | |
* (remove the gunzip part if you didn't compress your VM disk file) | |
``` | |
ftp -o - https://perso.pw/nixos/vm.disk.gz | gunzip -f -c | dd of=/dev/rsd0c bs… | |
``` | |
* reboot using the command "reboot" | |
## NixOS grub menu | |
At this step, in the serial console you should see a GRUB boot menu, it | |
will boot the first entry after a few seconds. Then NixOS will start | |
booting. In this menu you can access older versions of your system. | |
After the text stopped scrolling press enter. You should see a login | |
prompt, you can log in with the username "root" and the default | |
password "nixos" if you used my disk image. | |
## Configuring NixOS | |
If you used my template, your VM still doesn't have network | |
connectivity, you need to edit the file /etc/nixos/configuration.nix in | |
which I've put the most important variables you want to customize at | |
the top of the file. You need to configure your IPv4 and IPv6 | |
addresses and their gateways, and also your username with an ssh key to | |
connect to it, and the system name. | |
Once you are done, run "nixos-rebuild switch", you should have network | |
if you configured it correctly. | |
After the rebuild, run "passwd your_user" if you want to assign a | |
password to your newly declared user. | |
You should be able to connect to your VM using its public IP and your | |
ssh key with your username. | |
EXTRA: You may want to remove the profile minimal.nix which is | |
imported: it disables documentation and the use of X libraries, but | |
this may trigger packages compilation as they are not always built | |
without X support. | |
## Resizing the partition (last step) | |
Because we started with a small 2 GB raw disk to create the virtual | |
machine, the partition still has 2 GB only. We will have to resize the | |
partition /dev/vda1 to take all the disk space, and then resize the | |
ext4 file system. | |
First step is to extend the partition to 50 GB, the size of the virtual | |
disk offered at openbsd.amsterdam. | |
```shell | |
# nix-shell -p parted | |
# parted /dev/vda | |
(parted) resizepart 1 | |
Warning: The partition /dev/vda1 is currently in use. Are you sure to continue? | |
Yes/No? yes | |
End? [2147MB]? 50GB | |
(parted) quit | |
``` | |
Second step is to resize the file system to fill up the partition: | |
```shell | |
# resize2fs /dev/vda1 | |
The file system /dev/vda1 is mounted on / ; Resizing done on the fly | |
old_desc_blocks = 1, new_desc_blocks = 6 | |
The file system /dev/vda1 now has a size of 12206775 blocks (4k). | |
``` | |
Done! "df -h /" should report the new size. | |
## Congratulations | |
You have a fully functional NixOS VM! | |
# Creating the VM | |
While I provide a bootable NixOS disk image at | |
https://perso.pw/nixos/vm.disk.gz , you can generate yours with this | |
guide. | |
* create a raw disk of 2 GB to install the VM in it | |
```shell | |
qemu-img create -f raw vm.disk 2G | |
``` | |
* run qemu in a serial console to ensure it works, in the grub boot | |
menu you will need to select the 4th choice enabling serial console in | |
the installer. In this no graphics qemu mode, you can stop qemu by | |
pressing "ctrl+a" and then "c" to drop into qemu's own console, and | |
type "quit" to stop the process. | |
```shell | |
qemu-system-x86_64 \ | |
-smp 2 -m 4G \ | |
-enable-kvm \ | |
-display curses -nographic \ | |
-cdrom nixos-minimal*.iso \ | |
-drive file=vm.disk,if=virtio,format=raw | |
``` | |
* we create the partitions and prepare the chroot | |
``` | |
sudo -i | |
parted /dev/vda -- mklabel msdos | |
parted /dev/vda -- mkpart primary 1MiB 100% | |
mkfs.ext4 -L nixos /dev/vda1 | |
mount /dev/disk/by-label/nixos /mnt | |
mkdir -p /mnt/etc/nixos/ | |
``` | |
* edit the file /mnt/etc/nixos/configuration.nix , the NixOS install | |
has nano available by default, but you can have your favorite editor by | |
using "nix-shell -p vim" if you prefer vim. Here is a configuration | |
file that will work: | |
NixOS configuration.nix file for OpenBSD Amsterdam | |
* edit the file /mnt/etc/nixos/hardware-configuration.nix | |
NixOS hardware-configuration.nix file for OpenBSD Amsterdam | |
* we can run the installer, it will ask for the root password, and then | |
we can shut down the VM | |
```shell | |
nixos-install | |
systemctl poweroff | |
``` | |
Now, you have to host the disk file somewhere to make it available | |
through http or ftp protocol in order to retrieve it from the | |
openbsd.amsterdam VM. I'd recommend compressing the file by running | |
gzip on it, that will drastically reduce its size from 2GB to ~500MB. | |
# Full disk encryption | |
The ext4 file system offers a way to encrypt specific directories, it | |
can be enough for most users. | |
However, if you want to enable full disk encryption, you need to use | |
the guide above to generate your VM, but you need to create a separate | |
/boot partition and create a LUKS volume for the root partition. This | |
is explained in the NixOS manual, in the installer section. You should | |
adapt the according bits in the configuration file to match your new | |
setup. | |
Don't forget you will need to connect to the hypervisor to type your | |
password through the serial access every time you will reboot. | |
# Known issue and workaround | |
There is an issue with the OpenBSD hypervisor and Linux kernels at the | |
moment, when you reboot your Linux VM, the VM process on the OpenBSD | |
host crashes. Fortunately, it crashes after all the shutdown process | |
is done, so it doesn't let the file system in a weird state. | |
This problem is fixed in OpenBSD -current as of August 2022, and won't | |
happen in OpenBSD 7.2 hypervisors that will be available by the end of | |
the year. | |
A simple workaround is to open a tmux session in the hypervisor to run | |
an infinite loop regularly checking if your VM is running, and starting | |
it when it's stopped: | |
```shell | |
while true ; do vmctl status vm40 | grep stopped && vmctl start vm40 ; sleep 30… | |
``` | |
Mailing list archives: vmx_fault_page: uvm_fault returns 14, GPA=0xfe001818, ri… | |
Mailing list archives: vmm page fault with VM upgraded from Ubuntu 18LTS to 20L… | |
# Conclusion | |
It's great to have more choice when you need a VM. The OpenBSD | |
Amsterdam team is very kind, professional and regularly give money to | |
the OpenBSD project. | |
# Going further | |
This method should work for other hosting providers, given you can | |
access the VM disk from an live environment (installer, rescue system | |
etc..). You may need to pay attention to the disk device, and if you | |
can't obtain a serial console access to your system, you need to get | |
the network done right in the VM before copying it to the disk. | |
In the same vein, you can use this method to install any operating | |
system supported by the hypervisor. I chose NixOS because I love this | |
system, and it's easy to reproduce a result with its declarative | |
paradigm. |