Introduction
Introduction Statistics Contact Development Disclaimer Help
Title: Toward an automated tracking of OpenBSD ports contributions
Author: Solène
Date: 15 November 2020
Tags: openbsd automation
Description:
Since my previous article about a continous integration service to
track OpenBSD ports contribution I made a simple proof of concept that
allowed me to track what works and what doesn't work.
## The continuous integration goal
A first step for the CI service would be to create a database of diffs
sent to ports. This would allow people to track what has been sent and
not yet committed and what the state of the contribution is
(build/don't built, apply/don't apply). I would proceed following this
logic:
* a mail arrive and is sent to the pipeline
* it's possible to find a pkgpath out of the file
* the diff applies
* distfiles can be fetched
* portcheck is happy
Step 1 is easy, it could be mail dumped into a directory that get
scanned every X minutes.
Step 2 is already done in my POC using a shell script. It's quite hard
and required tuning. Submitted diffs are done with diff(1), cvs diff or
git diff. The important part is to retrieve the pkgpath like
"lang/php/7.4". This allow testing the port exists.
Step 3 is important, I found three cases so far when applying a diff:
* it works, we can then register in the database it can be used to
build
* it doesn't work, human investigation required
* the diff is already applied and patch think you want to reverse it.
It's already committed!
Being able to check if a diff is applied is really useful. When
building the contributions database, a daily check of patches that are
known to apply can be done. If a reverse patch is detected, this mean
it's committed and the entry could be delete from the database. This
would be rather useful to keep the database clean automatically over
time.
Step 4 is an inexpensive extra check to be sure the distfiles can be
downloaded over the internet.
Step 5 is also an inexpensive check, running portinfo can reports easy
to fix mistakes.
All the steps only require a ports tree. Only the step 4 could be
tricked by someone malicious, using a patch to make the system download
very huge files or files with some legal concerns, but that message
would also appear on the mailing list so the risk is quite limited.
To go further in the automation, building the port is required but it
must be done in a clean virtual machine. We could then report into the
database if the diff has been producing a package correctly, if not,
provide the compilation log.
## Automatic VM creation
Automatically creating an OpenBSD-current virtual machine was tricky
but I've been able to sort this out using vmm, rsync and upobsd.
The script download the last sets using rsync, that directory is served
from a mail server. I use upobsd to create an automatic installation
with bsd.rd including my autoinstall file. Then it gets tricky :)
vmm must be started with its storage disk AND the bsd.rd, as it's an
auto install, it will reboot after the install finishes and then will
install again and again.
I found that using the parameters "-B disk" would make the vm to
shutdown after installation for some reasons. I can then wait for the
vm to stop and then start it without bsd.rd.
My vmm VM creation sequence:
```shell commands to generate an OpenBSD virtual machine
upobsd -i autoinstall-vmm-openbsd -m http://localhost:8080/pub/OpenBSD/
vmctl stop -f -w integration
vmctl start -B disk -m 1G -L -i 1 -d main.qcow2 -b autobuild_vm/bsd.rd integrat…
vmctl wait integration
vmctl start -m 1G -L -i 1 -d main.qcow2 integration
```
The whole process is long though. A derivated qcow image could be used
after creation to try each port faster until we want to update the VM
again.
Multplies vm could be used at once to make parallel testing and make
good use of host ressources.
## What's done so far
I'm currently able to deposite email as files in a directory and run a
script that will extract the pkgpath, try to apply the patch, download
distfiles, run portcheck and run the build on the host using
PORTS_PRIVSEP. If the ports compiled fine, the email file is deleted
and a proper diff is made from the port and moved into a staging
directory where I'll review the diffs known to work.
This script would stop on blocking error and write a short text report
for each port. I intended to sent this as a reply to the mailing at
first, but maintaining a parallel website for people working on ports
seems a better idea.
You are viewing proxied material from dataswamp.org. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.