Introduction
Introduction Statistics Contact Development Disclaimer Help
Title: Using GitHub Actions to maintain Gentoo packages repository
Author: Solène
Date: 04 March 2023
Tags: gentoo automation
Description: In this article, I explain how I used GitHub actions to
build a packages repository for Gentoo and keep it up to date
# Introduction
In this blog post, I'd like to share how I had fun using GitHub actions
in order to maintain a repository of generic x86-64 Gentoo packages up
to date.
Built packages are available at https://interbus.perso.pw/ and can be
used in your `binrepos.conf` for a generic x86-64 packages provider,
it's not building many packages at the moment, but I'm open to add more
packages if you want to use the repository.
GitHub Project page: Build Gentoo Packages For Me
# Why
I don't really like GitHub, but if we can use their CPU for free for
something useful, why not? The whole implementation and setup looked
fun enough that I should give it a try.
I was using a similar setup locally to build packages for my Gentoo
netbook using a more powerful computer, so it was actually achievable,
so I had to try. I don't have much use of it myself, but maybe a
reader will enjoy the setup and do something similar (maybe not for
Gentoo).
My personal infrastructure is quite light, with only an APU router plus
a small box with an Atom CPU as a NAS, I was looking for a cheap way to
keep their Gentoo systems running without having to compile locally.
# Challenges
Building a generic Gentoo packages repository isn't straighforward for
a rew reasons:
* compilation flags must match all the consumers' architecture
* default USE flags must be useful for many
* no support for remote builders
* the whole repository must be generated on a single machine with all
the files (can't be incremental)
Fortunately, there are Gentoo containers images that can be used to
start a fresh Gentoo, and from there, build packages from a clean
system every time. Packages have to be added into the container before
each change, otherwise the file `Packages` that will be generated as a
repository index won't contain all the files.
Using a `-march=x86-64` compiler flag allows targeting all the amd64
systems, at the cost of less optimized binaries.
For the USE flags, a big part of Gentoo, I chose to select a default
profile and simply stick with it. People using the repository could
still change their USE flags, and only pick the binary packages from
the repo if they still match expectations.
# Setup
We will use GitHub actions (Free plan) to build packages for a given
Gentoo profile, and then upload it to a remote server that will share
the packages over HTTPS.
The plan is to use a docker image of a stage3 Gentoo provided by the
project gentoo-docker-images, pull previously built packages from my
server, build new packages or updating existing packages, and push the
changes to my server. Meanwhile, my server is serving the packages
over https.
GitHub's actions are a feature from GitHub allowing to create
Continuous Integration easy by providing "actions" (reusable components
made by other) that you organize in steps.
For the job, I used the following steps on an Ubuntu system:
1. Deploy SSH keys (used to pull/push packages to my server) stored as
secrets in the GitHub project
2. Checkout the sources of the project
3. Make a local copy of the packages repository
4. Create a container image based on the Gentoo stage3 + instructions
to run
5. Run the image that will use emerge to build the packages
6. Copy the new repository on the remote server (using rsync to copy
the diff)
GitHub project page: Gentoo Docker Images
# Problems encountered
While the idea is simple, I faced a lot of build failures, here is a
list of problems I remember.
## Go is failing to build (problem is Docker specific)
For some reasons, Go was failing to build with a weird error, this is
due to some sandboxing done by emerge that wasn't allowed by the Docker
environment.
The solution is to loose the sandboxing with `FEATURES="-ipc-sandbox
-pid-sandbox -sandbox -usersandbox"` in `/etc/portage/make.conf`.
That's not great.
## Raw stage3 is missing pieces
The starter image is a stage3 of Gentoo, it's quite bare, one critical
package missing to build other but never pulled as dependency is kernel
sources.
You need to install `sys-kernel/gentoo-sources` if you want builds to
succeed for many packages.
## No merged-usr profile
The gentoo docker images repository isn't provided merged-usr profiles
(yet?), I had to install merged-usr and run it, to have a correct
environment matching the selected profile.
## Compilation is too long
The job time is limited to 6h00 on the free plan, I added a timeout for
the emerge doing the building job to stop a bit earlier, to let it some
time to push the packages to the remote server, this will allow saving
time for the next run. Of course, this only works until a single
package require more than the timeout time to build (but it's quite
unlikely given the CI is fast enough).
# Security
One has to trust GitHub actions, GitHub employees may have access to
jobs running there, and could potentially compromise built packages
using a rogue container image. While it's unlikely, this is a
possibility.
Also, please note that the current setup doesn't sign the packages.
This is something that could be added later, you can find documentation
on the Gentoo Wiki for this part.
Gentoo Wiki: Binary package guide
Another interesting area for security was the rsync access of the
GitHub actions to easily synchronize the packages with the builder.
It's possible to restrict an SSH key to a single command to run, like a
single rsync with no room to change a single parameter. Unfortunately,
the setup requires using rsync in two different cases: downloading and
pushing files, so I had to write a wrapper looking at the variable
`SSH_COMMAND` and allowing either the "pull" rsync, or the "push"
rsync.
Restrict rsync command over SSH
# Conclusion
The GitHub free plan allows you to run a builder 24/7 (with no parallel
execution), it's really fast enough to keep a non-desktop @world up to
date. If you have a pro account, the local cache GitHub cache may not
be limited, and you may be able to keep the built packages there,
removing the "pull packages" step.
If you really want to use this, I'd recommend using a schedule in the
GitHub action to run it every day. It's as simple as adding this in
the GitHub workflow.
```yaml
on:
schedule:
- cron: '0 2 * * *' # every day at 02h00
```
# Credits
I would like to thank Jonathan Tremesaygues who wrote most of the
GitHub actions pieces after I shared with him about my idea and how I
would implement it.
Jonathan Tremesaygues's website
# Going further
Here is a simple script I'm using to use a local Linux machine as a
Gentoo builder for the box you run it from. It's using a gentoo stage3
docker image, populated with packages from the local system and its
`/etc/portage/` directory.
Note that you have to use `app-misc/resolve-march-native` to generate
the compiler command line parameters to replace `-march=native` because
you want the remote host to build with the correct flags and not its
own `-march=native`, you should also make sure those flags are working
on the remote system. From my experience, any remote builder newer
than your machine should be compatible.
Tildegit: Example of scripts to build packages on a remote machine for the loca…
You are viewing proxied material from dataswamp.org. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.