Title: Mirroring sources used in nixpkgs (software preservation) | |
Author: Solène | |
Date: 03 November 2022 | |
Tags: nix life | |
Description: In this text, I will show how to download locally all the | |
sources used to build packages in nixpkgs | |
# Introduction | |
This may appear like a very niche use case, in my quest of software | |
conservancy for nixpkgs I didn't encounter many people understanding | |
why I was doing this. | |
I would like to present you a project I made to easily download all the | |
sources files required to build packages from nixpkgs, allowing to keep | |
offline copies. | |
Why would you like to keep a local copy? If upstream disappear, you | |
can't get access to the sources anymore, except maybe in Hydra, but you | |
rely on a third party to access the sources, so it's still valuable to | |
have local copies of software you care about, just to make copies. | |
It's not that absolutely useful for everyone, but it's always important | |
to have such tools available. | |
nixpkgs-mirror-tarballs project page | |
# How to use it | |
You must run it on a system with `nix` installed. | |
After cloning and 'cd-ing' into the directory, simply run `./run.sh | |
some package list | ./mirror.pl`. The command `run.sh` will generate a | |
JSON structure containing all the dependencies used by the packages | |
listed as arguments, and the script `mirror.pl` will iterate over the | |
JSON list and use nix's fetcher to gather the sources in the nix store, | |
verifying the checksum on the go. This will create a directory | |
`distfiles` containing symlinks to the sources files stored in the | |
store. | |
The symlinks are very important as they will prevent garbage collection | |
from the store, and it's also used internally to quickly check if a | |
file is already in the store. | |
To delete a file from the store, remove its symlink and run the garbage | |
collector. | |
# Limitation | |
I still need to figure how to get a full list of all the packages, I | |
currently have a work in progress relying on `nix search --json` but it | |
doesn't work on 100% of the packages for some reasons. | |
It's currently not possible to easily trim distfiles that aren't useful | |
anymore, I plan to maybe add it someday. | |
# Trivia | |
This task is natively supported in the OpenBSD tool building packages | |
(dpb), it can fetch multiples files in parallel and automatic remove | |
files that aren't used anymore. This was really complicated to figure | |
how to replicate this with nixpkgs. |