On building an Ansible training environment on FreeBSD

In the course of the last three years I've had the pleasure of doing a
lot of teaching, and it really has mostly been a pleasure. I take
teaching very seriously and invest quite a bit of energy to bring my
love for a particular topic across. It's strenuous, nay exhausting, at
times but worth it.

For the courses I prepare system environments on which the students
get to run labs (a.k.a. practical exercises). They create and run
[1]Ansible playbooks in Ansible trainings, and they install DNS
servers during DNS trainings. In the past I used VirtualBox with which
to create these lab environments because it runs on multiple
platforms. Preconfigured VMs with specific capabilities (e.g. for
Ansible a VM contains DHCP and DNS, package repo, pip repo, git
server, etc.) are created, and a small shell script unpacks the
VirtualBox "appliance" (.ova file) into a number of VMs for each
student which they then use as their "data center" (I love saying
that). When you think about it it is a bit of a data center, if a
teeny tiny one: until now there were four VMs from and onto which
deployments are developed; nothing fancy, but many students need a bit
of effort in understanding what's going on.

I give trainings at customer sites, and there are clients who say "we
don't have Unix/Linux workstations; yes, we know Ansible requires
Unix, but we want the training on Windows PCs", and the shell script
installer (it cannot practicably be Ansible at this point) becomes
useless. The individual steps need to be typed out manually. (I could
turn it into a .bat file or whatever, but my motivation for doing that
has been low, knowing I would change the whole setup sometime.)

While VirtualBox has served us well in the past, it is slow, a bit
brittle on Linux (closing the lid on the laptop often causes breakage
the next morning; strangely that happens neither on Windows nor on
macOS as far as I've been able to ascertain), and in particular it is
cumbersome to update anything in that "data center": I have to bring
up the VMs, change whatever requires changing, package up the VMs into
an .ova and copy the 8GB onto thumb drives and/or make them available
for downloading.

It is high time for a change. What I envision is a laptop which I take
to a customer site, power it up, plug it into a switch, and off we go:
users can login when it's time to start doing labs and experience a
trouble-free Ansible environment. The laptop and its software must be
easy to update, in particular the Ansible installation must be easily
exchanged for a newer version just before a training. However, I also
require the possibility to fall back to a prior version if something
goes awry.

(I built an environment for DNS trainings using Fedora Linux and
systemd-nspawn after hearing [2]Pieter discuss systemd-nspawn at
Loadays. The system, based on [3]Btrfs has worked well (and still
does), but I recall being not amused by some of the pitfalls I ran
into (and then overcame). My plan is to redo that based on the work I
present here, and I hope to demonstrate why.)

The following diagram depicts what we're doing, and I intend to
describe what the individual bits are and how they interact.

The controller is the [4]Ansible controller; that's the machine
trainees login to in order to develop and then run playbooks. We give
them a choice of editors, from the Standard Unix editor via joe, pine,
pico, mg, through to vi and vim. Students create a "system" which
requires three machines in their data center; those are the little
boxes on the right hand side. Each student has three.

the new training environment

revisiting *BSD

Having mostly ignored all things *BSD during the last dozen years, I
[5]visited BSDCan and [6]EuroBSDCon this year, and I became enamored
once again with the clean simplicity of some of what I saw. (There's a
very balanced [7]comparative introduction to FreeBSD for Linux users I
can recommend you read if you want to get into the mood.)

Earlier this year [8]I installed OpenBSD on a laptop, and while it
hasn't been able to replace my Mac, it's a close second. I considered
using OpenBSD with VMM/VMD as experiments were promising, and
[9]Mischa runs [10]openbsd.amsterdam successfully with it, but one
idea led to the next, and I decided to give [11]FreeBSD a closer look,
in particular to jails and [12]ZFS.

Oh, and before I forget to mention it, if you want a really
sexy-looking graphical UI for FreeBSD, head over to [13]NomadBSD: I
gave it a whirl and it looks gorgeous. (As you'll read in a moment I
don't require or need a graphical workstation for this project, but if
I did I would install NomadBSD.)

a few weeks in jail

(With apologies to Michael Lucas for stealing bits from the [14]title
of a talk of his.)

I suspected [15]FreeBSD jails would be well suited for my project.
Jails are a virtualization of access to file systems, users,
processes, and networking. Students require a platform onto which they
can SSH via Ansible and install some packages, deploy templates,
create users, etc. Why not jails?

I started a bit low-level, first with [16]jail(8) and then [17]ezjail.
I then quickly found [18]BastilleBSD which I liked a lot, but because
I got stuck with networking, I then tried [19]iocage and have remained
with it. (The record should show that my "stuck" with networking was
not BastilleBSD's fault but mine.)

A combination of [20]Lucas' Jails book, the [21]iocage documentation,
and [22]a glance or two at the iocage source got me going.

The main reason for choosing [23]iocage over [24]BastilleBSD for jail
creation is iocage's [25]templates which allow me to build a base jail
containing all I need in it and then fire off creation of jails based
on that template. (BastilleBSD also has templating - a bit like
automation which is applied to the jail after it's launched. This is
practical, but for my purposes iocage templates are faster.)

The Ansible training template is created as follows, with an initial
[26]bootstrap shell script:
host# iocage create -r 12.1-RELEASE -n t-ansible0 ip4_addr="jail0|127.
0.2.1"
host# cp ansi-bootstrap.sh /zroot/iocage/jails/t-ansible0/root/tmp/
host# iocage start t-ansible0
mach% /tmp/ansi-bootstrap.sh
mach% exit
host# iocage stop t-ansible0
host# iocage set template=yes notes="template for Ansible student mach
ines" t-ansible0

That completes building the template which takes a minute or two.

In order to then build the three jails for each student I run the
mk-ansi-jails.sh shell script which is generated from an Ansible
template (see below):
iocage create -t t-ansible0 -n an102 \
               notes="ansible,xx" \
               quota=4G \
               resolver="domain XX.example;nameserver 10.53.1.1" \
               ip4_addr="jail0|10.53.1.102/32"
..

Apropos "stuck with networking", that happened to me with iocage as
well, but I found out what I was doing incorrectly: I [27]had to add
jail addresses to PF.

delivering packages

For setting up the Ansible template jail as well as for providing a
package repository for students when they install something during lab
work, we need a FreeBSD package repository, but I wanted to keep it as
simple as possible: recall this is on a laptop which will be used to
conduct training sessions, so I can afford to cut a few corners in
terms of security: at the end of a session, we reset all the jails:

[28]Dan taught me to make a jail for each "system", so I do that: the
jail with the package repository is lovingly called dhl (because it's
a service which delivers packages; ok, sorry), and it's set to boot=on
when [29]iocage starts. dhl basically just runs an nginx-lite service
with autoindex on, and I build the package repository proper with a
Makefile. (I think I recently confessed: I love [30]make - it's one of
Unix' hidden gems):
LIST = screen sudo tmux bash wget curl \
      bind-tools moreutils python36 ...

all:
       ASSUME_ALWAYS_YES=yes pkg fetch -U --output . -d $(LIST)
       pkg repo .

That's it: fetch obtains the packages with their dependencies (-d),
and repo creates a repository. All I need to do within the repository
directory is ensure there's a symbolic link from pkg.txz to the latest
version so that machines can bootstrap the package system.

The training jails built from the t-ansible0 iocage template use the
dhl repository server with this package configuration installed on
them: we disable FreeBSD's repository and add our own. I really like
the way this is implemented: no need to muck about with the FreeBSD
provided configuration from /etc/pkg; simply add a new file to
/usr/local/etc/pkg/ and therein disable the FreeBSD standard
repository:
FreeBSD: { enabled: no }

DHL: { url: http://dhl.ansible.example:80/repo,
          enabled: yes,
          signature_type: none }

Copying packages in this way is likely simplistic to the FreeBSD folk
who run [31]Poudriere for building packages, but that would be
overkill here; this lightweight copy of just the packages and their
dependencies we'll be requiring for labs is simple and sufficient.

ansible installs ansible

Each student will login to their own account on the ansible
controller, irrespective of whether it runs on the host or in a
dedicated jail, and we set up a dedicated login class for them:
ansible:\
  :nologin=/var/run/ansible-no:\
  :path=~/bin /bin /usr/bin /usr/local/bin /usr/local/ansible/latest/
bin:\
  :hushlogin:\
  :tc=standard:

This permits us to have different Ansible installations and use $PATH
and a symbolic link called latest to point students to the current
version.
$ ls -l /usr/local/ansible
drwxr-xr-x  5 root  wheel   7 Dec  3 18:41 2.7.6
drwxr-xr-x  5 root  wheel   7 Dec  3 18:41 2.9.1
drwxr-xr-x  5 root  wheel   7 Dec 11 13:29 2.9.2
lrwxr-xr-x  1 root  wheel  24 Dec 11 13:30 latest -> /usr/local/ansibl
e/2.9.2

How do those versions get there? Well, we distribute Ansible with
Ansible, of course! :-)
vars:
 adir: "/usr/local/ansible"
 aver: "2.9.2"
tasks:
- name: ansible | install ansible from pip into new virtualenv
 pip:
     name: "ansible=={{ aver }}"
     virtualenv: "{{ adir }}/{{ aver }}"
     virtualenv_command: "/usr/local/bin/python3.6 -m venv"

- name: ansible | symlink current version to latest
 file:
     src: "{{ adir }}/{{ aver }}"
     dest: "{{ adir }}/latest"
     state: link
     force: true

naming things

Students shall each have a subdomain so that when a lab instructs them
to create a playbook to "deploy this onto host web" they can all use
that short name which will point to their specific Web server jail
(one of 0, 1, or 2 in the diagram above).

We all know, don't we, that naming things is difficult so I had a bit
of fun asking on Twitter, and spent the better part of an hour
grinning about all manner of suggestions. (I have of course taken note
of [32]what certain people [33]said about me. :-) )

One of the [DEL: least useless :DEL] good ideas was elements in the
periodic table, so I took it a step further and chose those which
matched ccTLD names. (It turns out [34]Tony got a list of those years
ago.) The result became a small [35]YAML "database" (yeah, I knew
you'd like that!):
---
hostnames:
 - app
 - mosq
 - web
domains:
 - { code: "al", elem: "aluminium", cc: "ALBANIA" }
 - { code: "as", elem: "arsenic", cc: "AMERICAN SAMOA" }
 - { code: "au", elem: "aurum", cc: "AUSTRALIA" }
 ...

I use this data base within Ansible templates for creating [36]Unbound
DNS local data, rules for the PF firewall, for creating the users (al,
as, au, ...) on the Ansible controller, and for templating out the
mk-ansi-jails.sh script which creates student jails:
# {{ ansible_managed }}
# This shell script is GENERATED

{% set ns = namespace(counter = 0, n = 0) %}
{% for d in domains %}
{%    for h in hostnames %}
{%      set nnn = "%d" | format(100 + ns.counter + ns.n) %}
{%      set ip = "10.53.1.%d" | format(100 + ns.counter + ns.n) %}

iocage create -t t-ansible0 -n an{{ nnn }} \
               notes="ansible,{{ d.code|lower }}" \
               quota=4G \
               resolver="domain {{ d.code|upper }}.example;nameserver
10.53.1.1" \
               ip4_addr="jail0|{{ ip }}/32"

{%      set ns.n = ns.n + 1 %}
{%    endfor %}
{%    set ns.counter = ns.counter + 10 %}{# skip to next block of 10 #
}
{%    set ns.n = 0 %}

{% endfor %}

domain name stuff

Student jails are created with a resolver configuration which sets
domain within the jail:
host# iocage create .... resolver="domain XX.example;nameserver 10.53.
1.1"

iocage uses the resolver= setting, replacing semicolons by newlines,
to install an /etc/resolv.conf file in the jail when it's started.
Each of the student jails therewith accesses their own domain.

In the shell accounts, we template out a .profile which sets
$LOCALDOMAIN to the same value, e.g. XX.example, so when a student
uses Ansible to ssh to www, say, they get a response to the query for
www.XX.example. ($LOCALDOMAIN doesn't make a lot of sense if I set
domain in the resolver configuration, but we also use it with
[37]figlet(1), one of Unix' more important utilities, to welcome a
user on login.

figlet in operation

Both host and jails point to the same resolver on 10.53.1.1 which is
an [38]Unbound configured with local data. (The TXT records are there
because I'm bound to forget what the two-letter codes mean.) This we
template out from above domains.yml:
# Ansible managed
server:
       local-data:     "app.AL.example. IN A 10.53.1.100"
       local-data:     "app.AL.example. IN TXT 'element: aluminium'"
       local-data:     "app.AL.example. IN TXT 'cc: albania'"
       local-data-ptr: "10.53.1.100 app.AL.example."
       ...

Then come incoming PF port redirections for a lab with a Web server:
rdr pass inet proto tcp from any to any port { 8102 } -> 10.53.1.102 p
ort 80 # AL aluminium
rdr pass inet proto tcp from any to any port { 8112 } -> 10.53.1.112 p
ort 80 # AS arsenic
rdr pass inet proto tcp from any to any port { 8122 } -> 10.53.1.122 p
ort 80 # AU aurum

The domains.yml database is also used to template out the
mk-ansi-jails.sh shell script with which to create the student iocage
jails (seen earlier), and lastly, I create directories, users, etc. on
the Ansible controller jail:
- name: create .profile
 copy:
     content: |
         export LOCALDOMAIN="{{ item.code|upper }}.example."
         figlet $LOCALDOMAIN
     dest: "{{ homeprefix }}/{{ item.code|lower }}/.profile"
     owner: "{{ item.code|lower }}"
     mode: "0755"
 with_items: '{{ domains }}'

You get the picture.

enough resources

The training laptop is a Thinkpad T430 running FreeBSD 12.1 in 16GB of
RAM. Before starting I asked around whether people thought that sort
of hardware would manage 30 to 40 jails with Python in them; the
response was "yeah, probably, it depends" of course. :-) The number 30
comes from 10 participants, each with 3 machines in their "data
center".

I've not yet had the Ansible training laptop in production, but being
the [DEL: pessimistic :DEL] careful type I am, I've simulated some
load on it, putting much more pressure on the system than I expect it
to have.

I created 170 jails (over five times the amount I need) and I could
have likely done many more. The following output is an almost idle
T430 running 171 jails.
last pid: 21220;  load averages:  0.84,  0.89,  0.59  up 1+10:13:20
07:49:27
568 processes: 1 running, 567 sleeping
CPU:  0.1% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.7% idle
Mem: 645M Active, 1869M Inact, 7613M Wired, 5461M Free
ARC: 3586M Total, 1984M MFU, 980M MRU, 65M Anon, 35M Header, 522M Othe
r
    1856M Compressed, 4297M Uncompressed, 2.32:1 Ratio
Swap: 2048M Total, 2048M Free

I then whipped up [39]a very simple playbook to install a few packages
and launch nginx and a BIND name server on each of those jails, using
ten forks. The total runtime was 10m42s. During this time the host
(i.e. laptop) remained very responsive; if I hadn't known all those
processes were running I wouldn't have noticed. The instant the
playbook finished, the system looked like this:
last pid: 17626;  load averages:  7.52,  6.33,  3.71  up 1+10:30:00
08:06:07
1112 processes:1 running, 1111 sleeping
CPU:  0.1% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.7% idle
Mem: 4163M Active, 3400M Inact, 65M Laundry, 7321M Wired, 639M Free
ARC: 4407M Total, 2453M MFU, 1210M MRU, 65M Anon, 54M Header, 625M Oth
er
    2338M Compressed, 4536M Uncompressed, 1.94:1 Ratio
Swap: 2048M Total, 2048M Free

After a minute or two, with all the jails still running a copy of
[40]BIND and nginx, the load average fell to 0.75.

I decided to spice things up a bit and fired off the creation of a
cron entry which would execute once a minute to all machines via
Ansible:
#Ansible: keygenerate
* * * * * rm -f /tmp/xxx; ssh-keygen -t rsa -b 4096 -N bla -f /tmp/xxx

I ran the playbook in batches of about 50, and after the second batch,
this happened:
last pid: 31375;  load averages: 150.10, 107.97, 59.07  up 1+10:59:32
  08:35:39
1385 processes:154 running, 1226 sleeping, 3 stopped, 2 zombie
CPU: 99.4% user,  0.0% nice,  0.6% system,  0.0% interrupt,  0.0% idle
Mem: 4086M Active, 4071M Inact, 64M Laundry, 6921M Wired, 442M Free
ARC: 4035M Total, 2466M MFU, 824M MRU, 65M Anon, 48M Header, 632M Othe
r
    1933M Compressed, 3702M Uncompressed, 1.91:1 Ratio
Swap: 2048M Total, 2048M Free

The machine became a wee sluggish, to put it mildly. To be honest, it
was unusable. There was but one thing left to do: iocage destroy was
too slow, but this killed off all the jails quickly:
jail -r $(jls | grep an | awk '{print$1;}')

After that I removed all the jails:
iocage destroy -f $(iocage list -H | awk 'substr($2, 1, 2) == "an" { p
rint $2; }')

Running 100 SSH key generations, many of them in parallel, on 170
jails is not what this environment is built for, but I was curious.
Apart from that experiment, I can say that working on the machine
proper I didn't notice any performance issues, and I don't expect any
trouble at all.

getting users logged in

I originally envisioned a VNET jail onto which trainees would login,
but I think that's more trouble than it's worth. For one, I couldn't
get DHCP working with a VNET [41]iocage (here again, likely something
missing in PF), and on the other hand I cannot use a static IP because
the machine must connect to whichever network I'm visiting with the
training laptop.

Unix being Unix and a multi user system, I first decided to let users
login to the host itself: they require neither root permissions nor
any special privileges other than a text editor, being able to run
Ansible and SSH, so there's not much that students can kaputt. On the
other hand, I hear Dan saying "one jail per service!", but with jails
on shared private IP space ... ? I could configure PF to redirect a
different port into the Ansible controller jail, but then I hear the
Lucas guy yell

this leads to telling users things like "add 61000 to all your port
numbers to find the services for jail 61. Yes, as in 61022, 61443, and
so on."

I also must consider I'll have users with Windows only (running
[42]Putty, [43]MobaXterm, or similar), so for now I've got the
following options:
1. create training users on the host proper
2. create a controller jail for the users into which SSH clients jump

The users and the Ansible configuration would be identical in both
cases (and we create all that is required via Ansible from our
"domains.yml" data base), but the second case requires a bit of
additional work, and it requires special configuration on the client
side.

I easily create an SSH jump host configuration in sshd_config on the
host:
host# echo training | pw useradd -n detour -c 'SSH Jump' -s /bin/csh -
m -h 0 # -w none

host# tail -10 /etc/ssh/sshd_config
Match user detour
   AllowAgentForwarding no
   AllowTcpForwarding yes
   PermitTunnel no
   GatewayPorts no
   X11Forwarding no
   PubkeyAuthentication no
   PermitEmptyPasswords yes
   PasswordAuthentication yes
   ForceCommand echo 'This account is only for ProxyJump'

This handles both new and old versions of "jumping" via a bastion host
with the disadvantage of having to specify two passwords (one for the
jump host, the other for the target jail). But as students basically
do this once a day, I think it's just a minor inconvenience.
$ ssh -J [email protected] -l xx 10.53.1.20
$ ssh -o ProxyCommand="ssh -W %h:%p [email protected]" -l xx 10.53.
1.20

(The minor inconvenience can be overcome by creating the user with -w
none as shown above, adding nullok to the end of the line
^auth.*required.*pam_unix\.so in /etc/pam.d/sshd, and restarting sshd;
the jump host will not prompt for a password for users. But please
don't tell anybody I told you that...)

Now, while that configuration also works well for [44]MobaXterm, the
venerable [45]Putty doesn't support jump hosts; [46]SOCKS to the
rescue in form of [47]ss5:
$ grep ss5 /etc/rc.conf
ss5_enable="YES"
ss5_flags="-b 0.0.0.0:1080 -u root"

$ cat /usr/local/etc/ss5/ss5.conf
set SS5_VERBOSE

auth      0.0.0.0/0  - n
permit -  0.0.0.0/0  - 10.53.1.0/24  22 -  - - -

When I configure Putty to use the SOCKS4 proxy on port 1080, I see the
connection in the ss5 logs:
$ grep 202 /var/log/ss5/ss5.log
[11/Dec/2019:10:43:24 UTC] [15102] 192.168.1.202  "CONNECT" STARTED 0
0 0 (192.168.1.202:50157 -> 10.53.1.102:22)
[11/Dec/2019:10:53:45 UTC] [15102] 192.168.1.202  "CONNECT" TERMINATED
3070 2508 21 (192.168.1.202:50157 -> 10.53.1.102:22)

I could even use ss5 as a socks proxy to get Firefox to speak to the
users' lab jails ...

I've not decided whether students should "jump" into the Ansible
controller jail or whether I should make their lives easier and
populate them on the host proper. I tend to the former, because
resetting it all is easier at the end of a training, but the latter
would be more comfortable for students.

again and again and again

Would I do this again? Gladly. Will I do this again? Yes!

Working with [48]FreeBSD is rewarding: [49]its documentation is very
good, the software is solid, and everything makes a very polished
impression. It's not a problem that some of the packaged programs are
a bit older (get a port), and neither, for the record, is the fact
that a special console driver is not automatically detected.

I much appreciate some of what initially looks almost primitive (e.g.
rc.conf) but in fact is inspiring through its simplicity. For example,
upon reading the manual for [50]login.conf(5) I appreciate the well
thought-out integration in the system and, in particular, that what I
read actually functions that way. What I mentioned earlier about
overwriting the standard FreeBSD package repository by dropping a file
which invalidates it into a separate directory not only makes sense,
but makes a lot of things easier. That's just one example.

Most if not all of the parts in the BSD operating systems are well
thought out. As Lucas H. said recently, the software "has a long term
feel to it".

In the past years I've had to work with at least three distinct Linux
distributions at customer sites. In no particular order these are
CentOS and Debian in differing versions, and Suse Linux or SLES also
in different versions.

I've lost count on how many different locations I have to try in order
to configure networking or set up DNS resolution. Was it the
INI-looking file in /etc/networks/ was it /etc/sysconfig/something, oh
right, it's *.yaml here. Or it's NetworkManager, or this or that, and
I fear that the next version of something will again rip out and
replace yet another system. I'm sure there are plenty of you out there
who know all this by muscle memory for thirtytween distributions, but
I don't and never will, and I can add: I simply don't want to!

I've also, and this to me is much more important, lost count of the
number of times I've not been able to find what I need in
documentation bundled with the Linux distro du jour and instead
"googled" for an often incorrect answer.

The BSDs may be somewhat old-fashioned compared to the newest
whiz-bang Linux distro, but that suits me: I also am.

[51]Continued...

References

  1. https://jpmens.net/2012/06/06/configuration-management-with-ansible/
  2. https://blog.lieter.nl/presentations/plexis-LOADays-systemd-nspawn.pdf
  3. https://btrfs.wiki.kernel.org/index.php/Main_Page
  4. https://jpmens.net/2012/06/06/configuration-management-with-ansible/
  5. https://jpmens.net/2019/05/22/bsdcan-2019/
  6. https://jpmens.net/2019/09/24/eurobsdcon-2019/
  7. https://www.digitalocean.com/community/tutorials/a-comparative-introduction-to-freebsd-for-linux-users
  8. https://jpmens.net/2019/04/06/challenge-accepted-openbsd-on-a-laptop/
  9. https://www.mischapeters.com/
 10. https://openbsd.amsterdam/
 11. https://www.freebsd.org/
 12. https://en.wikipedia.org/wiki/ZFS
 13. https://nomadbsd.org/
 14. https://www.youtube.com/watch?v=DShkyYlqpmw
 15. https://www.freebsd.org/doc/handbook/jails.html
 16. https://www.freebsd.org/cgi/man.cgi?query=jail&apropos=0&sektion=8&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html
 17. https://www.freebsd.org/doc/handbook/jails-ezjail.html
 18. https://bastillebsd.org/
 19. https://iocage.io/more-on-iocage/
 20. https://mwl.io/nonfiction/os#fmjail
 21. https://iocage.readthedocs.io/en/latest/index.html
 22. https://github.com/iocage/iocage
 23. https://iocage.io/more-on-iocage/
 24. https://bastillebsd.org/
 25. https://iocage.readthedocs.io/en/latest/templates.html
 26. https://gist.github.com/jpmens/f0c0b825404849d94d7e59decb067f06
 27. https://jpmens.net/2019/12/07/add-address-of-iocage-jail-to-pf-table/
 28. https://dan.langille.org/
 29. https://iocage.io/more-on-iocage/
 30. https://en.wikipedia.org/wiki/Make_(software)
 31. https://www.freebsd.org/doc/handbook/ports-poudriere.html
 32. https://twitter.com/kolkman/status/1204352962790862849
 33. https://twitter.com/mahtin/status/1204367792482529287
 34. https://fanf.livejournal.com/138594.html
 35. http://yaml.org/
 36. http://unbound.net/
 37. https://www.freebsd.org/cgi/man.cgi?query=figlet&apropos=0&sektion=6&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html
 38. http://unbound.net/
 39. https://gist.github.com/jpmens/5659269454de6a4777e0dd3779b44a85
 40. https://www.isc.org/bind/
 41. https://iocage.io/more-on-iocage/
 42. https://www.chiark.greenend.org.uk/~sgtatham/putty/
 43. https://mobaxterm.mobatek.net/
 44. https://mobaxterm.mobatek.net/
 45. https://www.chiark.greenend.org.uk/~sgtatham/putty/
 46. https://en.wikipedia.org/wiki/SOCKS
 47. http://ss5.sourceforge.net/
 48. https://www.freebsd.org/
 49. https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/
 50. https://www.freebsd.org/cgi/man.cgi?query=login.conf&apropos=0&sektion=5&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html
 51. https://jpmens.net/2020/02/20/short-report-on-the-freebsd-training-machine/

You are viewing proxied material from serf.jpmens.net. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.