Jails with nullfs mount on FreeBSD 10 without buildworld
--------------------------------------------------------
Last edited: $Date: 2010/08/10 21:01:58 $
> FreeBSD jails are one ot the features that make FreeBSD awesome.
Jails are mostly used for enhanced security but they help whenever
working in silos offers an advantage. Very often one is adviced not to
mix packages build through the ports system with binary packages
installed by pkg. With jails it is easy to keep them separated. Also
jails can be useful in testing some application or some setup. Et
ceterea, there can be many more reasons to setup jails.
This is a setup without ezjail.
Of course working with ezjail is simple but setting up everything by
hand helps to learn a little.
The system I did this on does not have a lot of computing power so I
choose for a way to do this without building world.
The use of nullfs mounts (comparable with bind mounts on Linux) saves
diskspace and because that part is mounted readonly in the jail adds
some security.
I started with a fresh install of FreeBSD 10 i386.
Most of the stuff below I did according to the very good FreeBSD
handbook and some other recources I found on the internet.
## Install cpdup
If you have a fresh FreeBSD10 system, you may need to install cpdup.
pkg install cpdup
## General overview
The jail system to create has the following layout:
home
|-- j
| |-- mroot
| |-- skel
| |-- myjail
| `-- mysecondjail
|-- js
| |-- myjail
| `-- mysecondjail
* /home/j/mroot: this holds the base system that will be mounted
readonly over nullfs
* /home/j/skel: this is a template for the read-write part of the jail
* /home/j/myjail: this is where the base system gets mounted for the
jail 'myjail'
* /home/js/myjail: this is where the read-write part of the jail lives
The directory /home/j/mroot holds the base system. For every jail this
will be mounted readonly over nullfs. So this part will only exists
once on your harddisk.
The /home/j/skel contains the variable files that differs from jail to
jail, like /etc/rc.conf.
Setting up the directory for a new jail consists of two parts:
* creating a directory in /home/j/ in which the base system will be
mounted readonly
* copy the /home/j/skel to a directory in /home/js/, this will holds
the variable files like /etc/rc.conf of the jail and the home
directories of the users. Also it contains jail specific stuff, like
jail specific applications.
Also, a new jail requires adding two lines to /etc/fstab, adding an
instance to /etc/jail.conf and some finetuning.
## Prepare base system
The base system will be mounted read-only into the jail.
We populate the base system from the install CDROM, so we don't have
to do a buildworld.
mkdir -p /home/j/mroot
setenv DESTDIR /home/j/mroot
mount_cd9660 /dev/cd0 /mnt
tar -xf /mnt/usr/freebsd-dist/base.txz -C $DESTDIR
tar -xf /mnt/usr/freebsd-dist/doc.txz -C $DESTDIR
tar -xf /mnt/usr/freebsd-dist/ports.txz -C $DESTDIR
cd
umount /mnt
cp /etc/resolv.conf $DESTDIR/etc/
chroot $DESTDIR
tzsetup
pkg install cpdup
With this last step (pkg install cpdup) we force the system to install
the pkg utility. Later on in the process this will not be as easy as
this, because we are going to move directories like /etc out of the
base system.
## Prepare template for the read-write part of the jail
First setup the directory for the template:
mkdir /home/j/skel /home/j/skel/home /home/j/skel/usr-X11R6 /home/j/skel/distfiles
Now populate it by moving parts of the base system into it:
cd /home/j/mroot
mv etc /home/j/skel
mv usr/local /home/j/skel/usr-local
mv tmp /home/j/skel
mv var /home/j/skel
mv root /home/j/skel
And create symlinks back into the base system:
cd /home/j/mroot
mkdir s
ln -s s/etc etc
ln -s s/home home
ln -s s/root root
ln -s /s/usr-local usr/local
ln -s /s/usr-X11R6 usr/X11R6
ln -s /s/distfiles usr/ports/distfiles
ln -s s/tmp tmp
ln -s s/var var
Setup make.conf:
echo "WRKDIRPREFIX?= /s/portbuild" >> /home/j/skel/etc/make.conf
## Creating the directories for a new jail
Two directories are created for the new jail:
* /home/j/myjail
* /home/js/myjail
The /home/j/myjail will be populated by a nullfs mount, the
/home/js/myjail will get real files by copying the skel directory to
it.
### Populate /home/js/myjail
cpdup /home/j/skel /home/js/myjail
### Edit /etc/fstab
On the host add the following lines to /etc/fstab:
/home/j/mroot /home/j/myjail nullfs ro 0 0
/home/js/myjail /home/j/myjail/s nullfs rw 0 0
### Edit /etc/jail.conf
For each jail a block has to be added to /etc/jail.conf:
imjail {
host.hostname = "myjail";
path = "/home/j/myjail";
ip4.addr += "192.168.1.100/32";
devfs_ruleset = 4;
allow.raw_sockets = 0;
exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_myjail_console.log";
mount.devfs;
allow.set_hostname = 0;
allow.sysvipc = 0;
}
### devfs and FreeBSD 10
The release of FreeBSD 10 brought some challenges around devs. I have
not found a great solution for this on the internet. The best is to
manually mount devfs into each jail directory before starting the jail
services.
mount -t devfs /dev /home/j/myjail/dev
This can for example be done from /etc/rc.local
### Last steps
Mount the new directories for the jail, setup root password, add a
user, set the hostname and make sure sshd will only listen to the ip
address of the jail:
mount -a
mount
chroot /home/j/myjail
passwd
adduser
vi /etc/rc.conf
vi /etc/ssh/sshd_config
exit
## Start the jail
services jail start myjail
## Mount jail on a remote NFS share
Currently I am experimenting with mounting the read write part of the
jail from a remote NFS server. My NFS server is a small system running
Debian.
First, add some lines to /etc/rc.conf:
rpcbind_enable="YES"
rpc_statd_enable="YES"
rpc_lockd_enable="YES"
To be sure, I did a reboot of the FreeBSD system, maybe just a restart
of rpc.bind is enough. After this, I mounted a remote NFS directory to
/home/jnfs, followed by:
mkdir /home/j/nfsjail
cpdup /home/j/skel /home/jnfs/nfsjail
mount_nullfs -o ro /home/j/mroot /home/j/nfsjail/
mount_nullfs -o rw /home/jnfs/nfsjail /home/j/nfsjail/s/
mount -t devfs devfs /home/j/nfsjail/dev/
From there on, editted sshd_config in the nfsjail directory to only
listen to the ip address of the jail, set a hostname in the rc.conf
and started the jail.
Entered the jail with jexec [jailnumber] /bin/sh.
The ultimate test was to set the root password and add a user. Both
were possible without complaints about not being able to lock the
password file.
Now that this seems to work, I will try to run a few jails this way
and see if this can run stable. The idea behing this is to be able to
run a FreeBSD jail server diskless without having to run the jails
from iSCSlI.
See Building a diskless FreeBSD jail server,
gopher://box.matto.nl/0/freebsd10_disklessjailserver.txt.
## Resources
There is a lot of good and usefull information on the internet. The
following resources were very helpfull:
* Creating and Controlling Jails chapter in FreeBSD handbook
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-build.html
* Updating Multiple Jails chapter in FreeBSD handbook
http://www.freebsd.org/doc/handbook/jails-application.html
* Multiple FreeBSD Jails with nullfs from
http://www.scottro.net/
http://www.scottro.net/nullfsjail.html#FreeBSD-10
$Id: freebsd10jails.txt,v 1.3 2010/08/10 21:01:58 matto Exp $
