Title: Creating a NixOS live USB for a full featured APU router | |
Author: Solène | |
Date: 03 August 2022 | |
Tags: networking security nixos apu | |
Description: This explains how to create a live USB image of NixOS to | |
boot on an APU router including all required features for your network. | |
# Introduction | |
At home, I'm running my own router to manage Internet, run DHCP, do | |
filter and caching etc... I'm using an APU2 running OpenBSD, it works | |
great so far, but I was curious to know if I could manage to run NixOS | |
on it without having to deal with serial console and installation. | |
It turned out it's possible! By configuring and creating a live NixOS | |
USB image, one can plug the USB memory stick into the router and have | |
an immutable NixOS. | |
NixOS wiki about creating a NixOS live CD/USB | |
# Network diagram | |
Here is a diagram of my network. It's really simple except the bridge | |
part that require an explanation. The APU router has 3 network | |
interfaces and I only need 2 of them (one for WAN and one for LAN), but | |
my switch doesn't have enough ports for all the devices, just missing | |
one, so I use the extra port of the APU to connect that device to the | |
whole LAN by bridging the two network interfaces. | |
``` | |
+----------------+ | |
| INTERNET | | |
+----------------+ | |
| | |
| | |
| | |
+----------------+ | |
| ISP ROUTER | | |
+----------------+ | |
| 192.168.1.254 | |
| | |
| | |
| 192.168.1.111 | |
+----------------+ | |
| APU ROUTER | | |
+----------------+ | |
|bridge #2 and #3| | |
| 10.42.42.42 | | |
+----------------+ | |
|port #3 | | |
| | port #2 | |
+----------+ | | |
| | | |
| +--------+ +----------+ | |
| 10.42.42.150 | switch |-----| Devices | | |
+--------+ +--------+ +----------+ | |
| NAS | | |
+--------+ | |
``` | |
Here is a list of services I need on my router, this doesn't include | |
all my filtering rules and specific tweaks. | |
- DHCP server | |
- DNS resolving caching using unbound | |
- NAT | |
- SSH | |
- UPnP | |
- Munin | |
- Bridge ethernets ports #2 and #3 to use #3 as an extra port like a | |
switch | |
# The whole configuration | |
For the curious, here is the whole configuration of the setup. In the | |
sections after, I'll explain each parts of the code. | |
```nix | |
{ config, pkgs, ... }: | |
{ | |
isoImage.squashfsCompression = "zstd -Xcompression-level 5"; | |
powerManagement.cpuFreqGovernor = "ondemand"; | |
boot.kernelPackages = pkgs.linuxPackages_xanmod_latest; | |
boot.kernelParams = [ "copytoram" ]; | |
boot.supportedFilesystems = pkgs.lib.mkForce [ "btrfs" "vfat" "xfs" "ntfs" "c… | |
services.irqbalance.enable = true; | |
networking.hostName = "kikimora"; | |
networking.dhcpcd.enable = false; | |
networking.usePredictableInterfaceNames = true; | |
networking.firewall.interfaces.eth0.allowedTCPPorts = [ 4949 ]; | |
networking.firewall.interfaces.br0.allowedTCPPorts = [ 53 ]; | |
networking.firewall.interfaces.br0.allowedUDPPorts = [ 53 ]; | |
security.sudo.wheelNeedsPassword = false; | |
services.acpid.enable = true; | |
services.openssh.enable = true; | |
services.unbound = { | |
enable = true; | |
settings = { | |
server = { | |
interface = [ "127.0.0.1" "10.42.42.42" ]; | |
access-control = [ | |
"0.0.0.0/0 refuse" | |
"127.0.0.0/8 allow" | |
"10.42.42.0/24 allow" | |
]; | |
}; | |
}; | |
}; | |
services.miniupnpd = { | |
enable = true; | |
externalInterface = "eth0"; | |
internalIPs = [ "br0" ]; | |
}; | |
services.munin-node = { | |
enable = true; | |
extraConfig = '' | |
allow ^63\.12\.23\.38$ | |
''; | |
}; | |
networking = { | |
defaultGateway = { address = "192.168.1.254"; interface = "eth0"; }; | |
interfaces.eth0 = { | |
ipv4.addresses = [ | |
{ address = "192.168.1.111"; prefixLength = 24; } | |
]; | |
}; | |
interfaces.br0 = { | |
ipv4.addresses = [ | |
{ address = "10.42.42.42"; prefixLength = 24; } | |
]; | |
}; | |
bridges.br0 = { | |
interfaces = [ "eth1" "eth2" ]; | |
}; | |
nat.enable = true; | |
nat.externalInterface = "eth0"; | |
nat.internalInterfaces = [ "br0" ]; | |
}; | |
services.dhcpd4 = { | |
enable = true; | |
extraConfig = '' | |
option subnet-mask 255.255.255.0; | |
option routers 10.42.42.42; | |
option domain-name-servers 10.42.42.42, 9.9.9.9; | |
subnet 10.42.42.0 netmask 255.255.255.0 { | |
range 10.42.42.100 10.42.42.199; | |
} | |
''; | |
interfaces = [ "br0" ]; | |
}; | |
time.timeZone = "Europe/Paris"; | |
users.mutableUsers = false; | |
users.users.solene.initialHashedPassword = "$6$ffffffffffffffff$TTTTTTTTTTTTT… | |
users.users.solene = { | |
isNormalUser = true; | |
extraGroups = [ "sudo" "wheel" ]; | |
}; | |
} | |
``` | |
# Explanations | |
This setup deserves some explanations with regard to each part of it. | |
## Live USB specific | |
I prefer to use zstd instead of xz for compressing the liveUSB image, | |
it's way faster and the compression ratio is nearly identical as xz. | |
```nix | |
isoImage.squashfsCompression = "zstd -Xcompression-level 5"; | |
``` | |
There is currently an issue when trying to use a non default kernel, | |
ZFS support is pulled in and create errors. By redefining the list of | |
supported file systems you can exclude ZFS from the list. | |
```nix | |
boot.supportedFilesystems = pkgs.lib.mkForce [ "btrfs" "vfat" "xfs" "ntfs" "c… | |
``` | |
## Kernel and system | |
The CPU frequency should stay at the minimum until the router has some | |
load to compute. | |
```nix | |
powerManagement.cpuFreqGovernor = "ondemand"; | |
services.acpid.enable = true; | |
``` | |
This makes the system to use the XanMod Linux kernel, it's a set of | |
patches reducing latency and improving performance. | |
Xanmod XanMod project website | |
```nix | |
boot.kernelPackages = pkgs.linuxPackages_xanmod_latest; | |
``` | |
In order to reduce usage of the USB memory stick, upon boot all the | |
content of the liveUSB will be loaded in memory, the USB memory stick | |
can be removed because it's not useful anymore. | |
```nix | |
boot.kernelParams = [ "copytoram" ]; | |
``` | |
The service irqbalance is useful as it assigns certain IRQ calls to | |
specific CPUs instead of letting the first CPU core to handle | |
everything. This is supposed to increase performance by hitting CPU | |
cache more often. | |
```nix | |
services.irqbalance.enable = true; | |
``` | |
## Network interfaces | |
As my APU wasn't running Linux, I couldn't know the name if the | |
interfaces without booting some Linux on it, attach to the serial | |
console and check their names. By using this setting, Ethernet | |
interfaces are named "eth0", "eth1" and "eth2". | |
```nix | |
networking.usePredictableInterfaceNames = true; | |
``` | |
Now, the most important part of the router setup, doing all the | |
following operations: | |
- assign an IP for eth0 and a default gateway | |
- create a bridge br0 with eth1 and eth2 and assign an IP to br0 | |
- enable NAT for br0 interface to reach the Internet through eth0 | |
```nix | |
networking = { | |
defaultGateway = { address = "192.168.1.254"; interface = "eth0"; }; | |
interfaces.eth0 = { | |
ipv4.addresses = [ | |
{ address = "192.168.1.111"; prefixLength = 24; } | |
]; | |
}; | |
interfaces.br0 = { | |
ipv4.addresses = [ | |
{ address = "10.42.42.42"; prefixLength = 24; } | |
]; | |
}; | |
bridges.br0 = { | |
interfaces = [ "eth1" "eth2" ]; | |
}; | |
nat.enable = true; | |
nat.externalInterface = "eth0"; | |
nat.internalInterfaces = [ "br0" ]; | |
}; | |
``` | |
This creates a user solene with a predefined password, add it to the | |
wheel and sudo groups in order to use sudo. Another setting allows | |
wheel members to run sudo without password, this is useful for testing | |
purpose but should be avoided on production systems. You could add | |
your SSH public key to ease and secure SSH access. | |
```nix | |
users.mutableUsers = false; | |
security.sudo.wheelNeedsPassword = false; | |
users.users.solene.initialHashedPassword = "$6$bVPyGA3aTEMTIGaX$FYkFnOqwk8GNf… | |
users.users.solene = { | |
isNormalUser = true; | |
extraGroups = [ "sudo" "wheel" ]; | |
}; | |
``` | |
## Networking services | |
This will run a DHCP server advertising the local DNS server and the | |
default gateway, as it defines ranges for DHCP clients in our local | |
network. | |
```nix | |
services.dhcpd4 = { | |
enable = true; | |
extraConfig = '' | |
option subnet-mask 255.255.255.0; | |
option routers 10.42.42.42; | |
option domain-name-servers 10.42.42.42, 9.9.9.9; | |
subnet 10.42.42.0 netmask 255.255.255.0 { | |
range 10.42.42.100 10.42.42.199; | |
} | |
''; | |
interfaces = [ "br0" ]; | |
}; | |
``` | |
All systems require a name in order to work, and we don't want to use | |
DHCP to get the IPs addresses. We also have to define a time zone. | |
```nix | |
networking.hostName = "kikimora"; | |
networking.dhcpcd.enable = false; | |
time.timeZone = "Europe/Paris"; | |
``` | |
This enables OpenSSH daemon listening on port 22. | |
```nix | |
services.openssh.enable = true; | |
``` | |
This enables the service unbound, a DNS resolver that is able to do | |
some caching as well. We need to allow our network 10.42.42.0/24 and | |
listen on the LAN facing interface to make it work, and not forget to | |
open the ports TCP/53 and UDP/53 in the firewall. This caching is very | |
effective on a LAN server. | |
```nix | |
services.unbound = { | |
enable = true; | |
settings = { | |
server = { | |
interface = [ "127.0.0.1" "10.42.42.42" ]; | |
access-control = [ | |
"0.0.0.0/0 refuse" | |
"127.0.0.0/8 allow" | |
"10.42.42.0/24 allow" | |
]; | |
}; | |
}; | |
}; | |
networking.firewall.interfaces.br0.allowedTCPPorts = [ 53 ]; | |
networking.firewall.interfaces.br0.allowedUDPPorts = [ 53 ]; | |
``` | |
This enables the service miniupnpd, this can be quite dangerous because | |
its purpose is to allow computer on the network to create NAT | |
forwarding rules on demand. Unfortunately, this is required to play | |
some video games and I don't really enjoy creating all the rules for | |
all the video games requiring it. | |
```nix | |
services.miniupnpd = { | |
enable = true; | |
externalInterface = "eth0"; | |
internalIPs = [ "br0" ]; | |
}; | |
``` | |
This enables the service munin-node and allow a remote server to | |
connect to it. This service is used to gather metrics of various data | |
and make graphs from them. I like it because the agent running on the | |
systems is very simple and easy to extend with plugins, and on the | |
server side, it doesn't need a lot of resources. As munin-node listens | |
on the port TCP/4949 we need to open it. | |
```nix | |
services.munin-node = { | |
enable = true; | |
extraConfig = '' | |
allow ^13\.17\.23\.28$ | |
''; | |
}; | |
networking.firewall.interfaces.eth0.allowedTCPPorts = [ 4949 ]; | |
``` | |
# Conclusion | |
By building a NixOS live image using Nix, I can easily try a new | |
configuration without modifying my router storage, but I could also use | |
it to ssh into the live system to install NixOS without having to deal | |
with the serial console. |