Title: Using dpb on OpenBSD for package compilation cluster

	Title: Using dpb on OpenBSD for package compilation cluster
	Author: Solène
	Date: 30 May 2021
	Tags: openbsd
	Description:

	# Introduction

	Today I will explain how to easily setup your own OpenBSD dpb infra.
	dpb is a tool to manage port building and can use chroot to use a sane
	environment for building packages.

	This is particularly useful when you want to test packages or build
	your own, it can parallelize package compilation in two way: multiples
	packages at once and multiples processes for one package.

	dpb man page
	proot man page

	The dpb and proot executable files are available under the bin
	directory of the ports tree.

	Building your packages provide absolutely NOTHING compared to using
	binary packages except wasting CPU time, disk space and bandwidth.

	# Setup

	You need a ports tree and a partition that you accept to mount with
	wxallowed,nosuid,dev options. I use /home/ for that. To simplify the
	setup, we will create a chroot in /home/build/ and put our ports tree
	in /home/build/usr/ports (then your /usr/ports can be a symlink).

	Create a text file that will be used as a configuration file for proot

	```proot configuration file
	chroot=/home/build
	WRKOBJDIR=/tmp/pobj
	LOCKDIR=/tmp/locks
	PLIST_REPOSITORY=/data/plist
	DISTDIR=/data/distfiles
	PACKAGE_REPOSITORY=/data/packages
	actions=unpopulate
	sets=base comp etc xbase xfont xshare xetc xserver
	```

	This will tell proot to create a chroot in /home/build and preconfigure
	some variables for /etc/mk.conf, use all sets listed in "sets" and
	clean everything when run (this is what actions=unpopulate is doing).
	Running proot is as easy as "proot -c proot_config".

	Then, you should be able to run "dpb -B /home/build/ some/port" and it
	will work.

	# Ease of use

	I wrote a script to clean locks from dpb, locks from ports system and
	pobj directories but also taking care of adding the mount options.

	Options -p and -j will tell dpb how many cores can be used for parallel
	compilation, note that dpb is smart and if you tell it 3 ports in
	parallel and 3 threads in parallel, it won't use 3x3, it will compile
	three ports at a time and once it's stuck with only one port, it will
	add cores to its build to make it faster.

	```script shell
	#!/bin/sh

	CHROOT=/home/build/
	CORES=3

	rm -fr ${CHROOT}/usr/ports/logs/amd64/locks/*
	rm -fr ${CHROOT}/tmp/locks/*
	rm -fr ${CHROOT}/tmp/pobj/*
	mount -o dev -u /home
	mount -o nosuid -u /home
	mount -o wxallowed -u /home
	/usr/ports/infrastructure/bin/dpb -B $CHROOT -c -p $CORES -j $CORES $*
	```

	Then I use "doas ./my_dpb.sh sysutils/p5-Rex lang/guile" to run the
	build process.

	It's important to use -c in dpb command line which will clear
	compilation logs of the packages but retains the log size, this will be
	used to estimate further builds progress by comparing current log size
	with previous logs sizes.

	You can harvest your packages from /home/build/data/packages/ , I even
	use a symlink from /usr/ports/packages/ to the dpb packages directory
	because sometimes I use make in ports and sometimes I use dpb, this
	allow recompiling packages in both areas. I do the same for distfiles.

	# Going further

	dpb can spread the compilation load over remote hosts (or even manage
	compilation for a different architecture), it's not complicated to
	setup but it's out of scope for the current guide. This requires
	setting up ssh keys and NFS shares, the difficulty is to think with the
	correct paths depending on chroot/not chroot and local / nfs.

	I extremely recommend reading dpb man pages, it supports many options
	such as providing it a list of pkgpaths (package address such as
	editor/vim or www/nginx) or building ports in random order.

	Here is a simply command to generate a list of pkgpaths of outdated
	packages on your system compared to the ports tree, the -q parameter is
	to make it a lot quicker but less accurate for shared libraries.

	```command line example
	/usr/ports/infrastructure/bin/pkg_outdated -q \| awk '/\// { print $1 }'
	```

	# Conclusion

	I use dpb when I want to update my packages from sources because the
	binary packages are not yet available or if I want to build a new
	package in a clean environment to check for missing dependencies,
	however I use a simple "make" when I work on a port.