Title: Using dpb on OpenBSD for package compilation cluster | |
Author: Solène | |
Date: 30 May 2021 | |
Tags: openbsd | |
Description: | |
# Introduction | |
Today I will explain how to easily setup your own OpenBSD dpb infra. | |
dpb is a tool to manage port building and can use chroot to use a sane | |
environment for building packages. | |
This is particularly useful when you want to test packages or build | |
your own, it can parallelize package compilation in two way: multiples | |
packages at once and multiples processes for one package. | |
dpb man page | |
proot man page | |
The dpb and proot executable files are available under the bin | |
directory of the ports tree. | |
Building your packages provide absolutely NOTHING compared to using | |
binary packages except wasting CPU time, disk space and bandwidth. | |
# Setup | |
You need a ports tree and a partition that you accept to mount with | |
wxallowed,nosuid,dev options. I use /home/ for that. To simplify the | |
setup, we will create a chroot in /home/build/ and put our ports tree | |
in /home/build/usr/ports (then your /usr/ports can be a symlink). | |
Create a text file that will be used as a configuration file for proot | |
```proot configuration file | |
chroot=/home/build | |
WRKOBJDIR=/tmp/pobj | |
LOCKDIR=/tmp/locks | |
PLIST_REPOSITORY=/data/plist | |
DISTDIR=/data/distfiles | |
PACKAGE_REPOSITORY=/data/packages | |
actions=unpopulate | |
sets=base comp etc xbase xfont xshare xetc xserver | |
``` | |
This will tell proot to create a chroot in /home/build and preconfigure | |
some variables for /etc/mk.conf, use all sets listed in "sets" and | |
clean everything when run (this is what actions=unpopulate is doing). | |
Running proot is as easy as "proot -c proot_config". | |
Then, you should be able to run "dpb -B /home/build/ some/port" and it | |
will work. | |
# Ease of use | |
I wrote a script to clean locks from dpb, locks from ports system and | |
pobj directories but also taking care of adding the mount options. | |
Options -p and -j will tell dpb how many cores can be used for parallel | |
compilation, note that dpb is smart and if you tell it 3 ports in | |
parallel and 3 threads in parallel, it won't use 3x3, it will compile | |
three ports at a time and once it's stuck with only one port, it will | |
add cores to its build to make it faster. | |
```script shell | |
#!/bin/sh | |
CHROOT=/home/build/ | |
CORES=3 | |
rm -fr ${CHROOT}/usr/ports/logs/amd64/locks/* | |
rm -fr ${CHROOT}/tmp/locks/* | |
rm -fr ${CHROOT}/tmp/pobj/* | |
mount -o dev -u /home | |
mount -o nosuid -u /home | |
mount -o wxallowed -u /home | |
/usr/ports/infrastructure/bin/dpb -B $CHROOT -c -p $CORES -j $CORES $* | |
``` | |
Then I use "doas ./my_dpb.sh sysutils/p5-Rex lang/guile" to run the | |
build process. | |
It's important to use -c in dpb command line which will clear | |
compilation logs of the packages but retains the log size, this will be | |
used to estimate further builds progress by comparing current log size | |
with previous logs sizes. | |
You can harvest your packages from /home/build/data/packages/ , I even | |
use a symlink from /usr/ports/packages/ to the dpb packages directory | |
because sometimes I use make in ports and sometimes I use dpb, this | |
allow recompiling packages in both areas. I do the same for distfiles. | |
# Going further | |
dpb can spread the compilation load over remote hosts (or even manage | |
compilation for a different architecture), it's not complicated to | |
setup but it's out of scope for the current guide. This requires | |
setting up ssh keys and NFS shares, the difficulty is to think with the | |
correct paths depending on chroot/not chroot and local / nfs. | |
I extremely recommend reading dpb man pages, it supports many options | |
such as providing it a list of pkgpaths (package address such as | |
editor/vim or www/nginx) or building ports in random order. | |
Here is a simply command to generate a list of pkgpaths of outdated | |
packages on your system compared to the ports tree, the -q parameter is | |
to make it a lot quicker but less accurate for shared libraries. | |
```command line example | |
/usr/ports/infrastructure/bin/pkg_outdated -q | awk '/\// { print $1 }' | |
``` | |
# Conclusion | |
I use dpb when I want to update my packages from sources because the | |
binary packages are not yet available or if I want to build a new | |
package in a clean environment to check for missing dependencies, | |
however I use a simple "make" when I work on a port. |