Title: How to make a local NixOS cache server | |
Author: Solène | |
Date: 02 June 2022 | |
Tags: nixos unix bandwidth nocloud | |
Description: This explains how to create your own local cache for NixOS | |
packages in order to save bandwidth. | |
# Introduction | |
If like me, you have multiple NixOS system behind the same router, you | |
may want to have a local shared cache to avoid downloading packages | |
multiple time. | |
This can be done simply by using nginx as a reverse proxy toward the | |
official repository and by enabling caching the result. | |
nix-binary-cache-proxy project I used as a base | |
# Server side configuration | |
We will declare a nginx service on the server, using http protocol only | |
to make setup easier. The packages are signed, so their authenticity | |
can't be faked. In this setup, using https would add anonymity which | |
is not much of a concern in a local network, for my use case. | |
In the following setup, the LAN cache server will be reachable at the | |
address 10.42.42.150, and will be using the DNS resolver 10.42.42.42 | |
every time it needs to reach the upstream server. | |
```nix code | |
services.nginx = { | |
enable = true; | |
appendHttpConfig = '' | |
proxy_cache_path /tmp/pkgcache levels=1:2 keys_zone=cachecache:100m max_s… | |
# Cache only success status codes; in particular we don't want to cache 4… | |
# See https://serverfault.com/a/690258/128321 | |
map $status $cache_header { | |
200 "public"; | |
302 "public"; | |
default "no-cache"; | |
} | |
access_log /var/log/nginx/access.log; | |
''; | |
virtualHosts."10.42.42.150" = { | |
locations."/" = { | |
root = "/var/public-nix-cache"; | |
extraConfig = '' | |
expires max; | |
add_header Cache-Control $cache_header always; | |
# Ask the upstream server if a file isn't available locally | |
error_page 404 = @fallback; | |
''; | |
}; | |
extraConfig = '' | |
# Using a variable for the upstream endpoint to ensure that it is | |
# resolved at runtime as opposed to once when the config file is loaded | |
# and then cached forever (we don't want that): | |
# see https://tenzer.dk/nginx-with-dynamic-upstreams/ | |
# This fixes errors like | |
# nginx: [emerg] host not found in upstream "upstream.example.com" | |
# when the upstream host is not reachable for a short time when | |
# nginx is started. | |
resolver 10.42.42.42; | |
set $upstream_endpoint http://cache.nixos.org; | |
''; | |
locations."@fallback" = { | |
proxyPass = "$upstream_endpoint"; | |
extraConfig = '' | |
proxy_cache cachecache; | |
proxy_cache_valid 200 302 60d; | |
expires max; | |
add_header Cache-Control $cache_header always; | |
''; | |
}; | |
# We always want to copy cache.nixos.org's nix-cache-info file, | |
# and ignore our own, because `nix-push` by default generates one | |
# without `Priority` field, and thus that file by default has priority | |
# 50 (compared to cache.nixos.org's `Priority: 40`), which will make | |
# download clients prefer `cache.nixos.org` over our binary cache. | |
locations."= /nix-cache-info" = { | |
# Note: This is duplicated with the `@fallback` above, | |
# would be nicer if we could redirect to the @fallback instead. | |
proxyPass = "$upstream_endpoint"; | |
extraConfig = '' | |
proxy_cache cachecache; | |
proxy_cache_valid 200 302 60d; | |
expires max; | |
add_header Cache-Control $cache_header always; | |
''; | |
}; | |
}; | |
}; | |
``` | |
Be careful, the default cache is located under /tmp/ but the nginx | |
systemd service is hardened and its /tmp/ is faked in a temporary | |
directory, meaning if you restart nginx you lose the cache. I'd advise | |
using a directory like /var/cache/nginx/ if you want your cache to | |
persist across restarts. | |
# Client side configuration | |
Using the cache server on a system is really easy. We will define the | |
binary cache to our new local server, the official cache is silently | |
added so we don't have to list it. | |
```nix code | |
nix.binaryCaches = [ "http://10.42.42.150/" ]; | |
``` | |
Note that you have to use this on the cache server itself if you want | |
the system to use the cache for its own needs. | |
# Conclusion | |
Using a local cache can save a lot of bandwidth when you have more than | |
one computer at home (or if you extensively use nix-shell and often run | |
the garbage collector). Due to NixOS packages names being unique, we | |
won't have any issues of a newer package version behind hidden by a | |
local copy cached, which make the setup really easy. |