Building a diskless FreeBSD 10 Jail server
------------------------------------------

Last edited: $Date: 2013/12/13 14:36:38 $


## Building a new diskless FreeBSD 10.0 Jail server

> FreeBSD Jail is a great lightweight virtualisation method.
> See https://en.wikipedia.org/wiki/FreeBSD_jail

Our project of the day is to build a new diskless system.

The goal  is to  have a setup  on the PXE-server  that will  allow a
diskless motherboard  to run  as a FreeBSD  10.0 server.  The rootfs
will be  mounted over NFS. Also  the diskless server mounts  a iscsi
virtual disk on which the jails are hosted.

Our diskless server will be running on an old VIA EPIA 600 MHz mini-
ITX board. This means a low power processor and not that much RAM.

In the network there is a SAN that serves both NFS shares as well as
iscsi drives. The root filesystem of our new diskless server will be
mounted  over NFS  on this  SAN. This  makes it  easy to  access the
rootfs from outside the server.

For our jails NFS is not  a good choice because starting and running
jails requires  file locks.  Because of  this we  choose to  mount a
iscsi drive which will hold our jails.


## Steps

 * Get FreeBSD 10.0 on a virtual machine
 * Get the latest source code
 * Build server rootfs with buildworld
 * Build kernel for diskless boot with rootfs over NFS
 * Mount rootfs over NFS
 * Install world and kernel on rootfs

Administrating  FreeBSD  means  practicing   patience.  We  have  to
checkout  the source  code and  build a  new environment  so several
steps will take a  lot of time. It helps to plan  to do steps during
the night...

## Get FreeBSD 10.0 on a virtual machine

At the time of writing  this page, FreeBSD 10.0-BETA4 was available.
By using  this beta  we get all  the new goodies  FreeBSD 10  has to
offer.

FreeBSD offers  several possibilities  to get it  up and  running, I
downloaded the KVM qcow2 image and started from there.

This is how I started the virtual machine on my Debian kvm-server:


   kvm -vnc 192.168.1.25:0  \
   -net nic,model=e1000,vlan=0 \
   -net tap,ifname=tap1,script=/etc/qemu-ifup-br0\
   -m 2048 -hda FreeBSD-10.0-BETA4-i386-20131130-r258774.qcow2


This virtual machine will be used to build the rootfs and the kernel
for our new diskless server.

The  time  needed  to  create  our new  diskless  server  is  mostly
determined  by the  duration of  compiling world  and compiling  the
kernel. So the  computing power of the virtual machine  has a lot of
impact on the duration of building the new diskless system.


## Fetch the source code

> svnlite is the new cvsup

Next we  get the latest  source code to  build FreeBSD 10.0.  In the
virtual machine we fetch this code  with the new svnlite that is the
replacement for cvsup.

   svnlite checkout https://svn0.us-west.freebsd.org/base/releng/10.0/ /usr/src

Choose a mirror site close to you. see
http://www.freebsd.org/doc/handbook/svn- mirrors.html

The checkout seems  to take ages, so  do this at the end  of the day
and continue with the following steps the next morning.


### Build server rootfs with buildworld

First, set up /etc/src.conf.


   WITHOUT_ATM=yes
   WITHOUT_BIND=yes
   WITHOUT_BLUETOOTH=yes
   WITHOUT_FLOPPY=yes
   WITHOUT_GAMES=yes
   WITHOUT_INET6=yes
   WITHOUT_IPX=yes
   WITHOUT_PPP=yes
   WITHOUT_SENDMAIL=yes
   WITHOUT_SHAREDOCS=yes
   WITHOUT_WIRELESS=yes
   WITHOUT_ZFS=yes


The VIA EPIA motherboard does not  support that much RAM, so we will
not use  ZFS. I  have not yet  played with IPV6,  so we  first build
without IPV6 support. After having set up /etc/src.conf we can start
building the root filesystem.

Probably we could have disabled more  parts of the build to create a
smaller rootfs,  but this way we  have something to start  with that
can be optimized in a later phase.


   mount server:/path/to/nfsshare /mnt
   setenv NFSROOTDIR /mnt
   cd /usr/src
   make buildworld
   make installworld DESTDIR=${NFSROOTDIR}


### Build kernel for diskless boot with rootfs over NFS

Next we have to build a kernel that allows diskless booting.

Create  /usr/src/sys/i386/conf/MYKERNEL as  a  config  file for  the
configuration of the kernel.


   make buildkernel KERNCONF=MYKERNEL
   make installkernel KERNCONF=MYKERNEL DESTDIR=${NFSROOTDIR}
   make distribution DESTDIR=${NFSROOTDIR}


## Prepare new environment

Go into the newly created environment, set password and create a new
user, that can be used for ssh login.


   chroot ${NFSROOTDIR}
   passwd
   adduser
   echo 'sshd_enable="YES"' > /etc/rc.conf
   echo 'defaultrouter="192.168.1.254"' >>  /etc/rc.conf
   mkdir /iscsi
   mkdir -p conf/base
   mkdir -p conf/base/iscsi
   tar -c -v -f conf/base/etc.cpio.gz --format cpio --gzip etc
   tar -c -v -f conf/base/var.cpio.gz --format cpio --gzip var
   exit


The /conf/base/var and  /conf/base/etc directories become read-write
mounted as  /var and /etc  on a memory  file system in  our diskless
system.

The conf/base/iscsi  directory becomes read-write mounted  as /iscsi
on  our  diskless system.  We  need  this to  be  able  to mount  an
iscsi-drive on which we can install our jails.


## Prepare PXE server

On  the PXE/DHCP  server we  now have  to set  up an  entry for  our
diskless server. I use the  isc-dhcp-server on a Debian-box. This is
what is needed in /etc/dhcp/dhcpd.conf:


   host disklessfreebsdbox {
       hardware ethernet 12:34:56:78:90:ab;
       fixed-address 192.168.1.9;
       filename "freebsd10/boot/pxeboot";
       next-server 192.168.1.2;
       option domain-name "tradesystem.nl";
       option domain-name-servers 192.168.1.1;
       option subnet-mask 255.255.255.0;
       option root-path "192.168.1.2:/rootfs/freebsd10";
   }


You have to change this according to your local situation.

Here is some explanation on the entry above:

 * hardware ethernet: the mac address of your diskless system
 * fixed-address: this is the ip address to be asigned to your diskless system
 * filename "freebsd10/boot/pxeboot": the location of the file named
   "pxeboot" which was installed during the execution of the make
   commands above, the path is relative to the tftpboot-path on the
   PXE-server
 * next-server 192.168.1.2: the ip-address of the NFS-server
 * option domain-name-servers 192.168.1.1: your DNS-server (mine is local)
 * option subnet-mask 255.255.255.0: the netmask for your diskless system
 * option root-path "192.168.1.2:/rootfs/freebsd10": the location of
   the root filesystem which was installed during the execution of
   the make commands above, that is going to be mounted over NFS by
   the diskless system.


## Boot diskless

After these  preparations are  done we  can do a  test boot  of the.
system Solve any problems before going to the next part            .

Enter your fresh diskless system and have a look around. Notice that
the root filesystem is mounted read-ony.  This is done by the script
inside /etc/rc.initdiskless,  this is  part of the  standard FreeBSD
environment. This  is why  we need the  /conf/base directory  on the
NFS-  server. The  contents of  this  directory will  be mounted  in
read-write mode on a memory filesystem.


## Install ezjail

Stop  the  diskless  system.  In  the  rootfilesystem  we  create  a
/etc/iscsi.conf file and run the  tar command again to re-create the
cpio format etc.cpio.gz file.

After that chroot again into the root filesystem and install ezjail.


   cd ${NFSROOTDIR}
   cd etc
   cp /etc/resolv.conf .
   chroot ${NFSROOTDIR}
   pkg install ezjail


Now the pkg repository catalogue will  be initialised. Have a cup of
tea, it might take a while.

The  total amount  of data  on  the ${NFSROOTDIR}  for our  diskless
system will now be grown to about 520 Mb.


## Prepare ezjail

### Setup iscsi, mount iscsi drive

Go into the diskless system.

Use the created /etc/iscsi.conf file  to connect to the iscsi-server
and mount the iscsi-share on /iscsi on the diskless system.


   mkdir -p /iscsi/jails/etc/ezjail


On the virtual machine create a symlink in the NFS-share


   chroot ${NFSROOTDIR}
   cd /usr/local/etc
   rmdir ezjail
   ln -s /iscsi/jails/etc/ezjail .


In /usr/local/etc/ezjail.conf set the ezjail_jaildir to the /iscsi
directory


   ezjail_jaildir=/iscsi/jails


This way when we create jails  in the diskless server, these will be
created on the  iscsi-drive, which is the  only rewritable partition
on a harddisk.


### Initialize ezjail



   ezjail-admin install


After this, create a first jail to test



   ezjail-admin create -f example example.com 10.0.0.2


Make  sure you  have created  an  alias on  your network  interface.
This  can be  done on  the  virtual machine  in the  rc.conf on  the
${NFSROOTDIR}.


## Using the system

One  of the  quirks of  FreeBSD jails  is that  for every  jail that
is  created a  file  named /etc/fstab.<jailname>  is created,  where
<jailname>  is  the  name  of  the jail  with  periods  replaced  by
underscores.

In  our  diskless  system  the  /etc directory  lives  in  a  memory
filesystem. This means  any run-time changes are lost  on shutdow of
the system. I have  not found a way to change this  path. So we have
to conserve these files on a writable harddisk.

What i  do is copy  the /etc/fstab.* files to  /iscsi/jails/etc/ and
after a reboot and a remount of  /iscsi copy the files back to /etc.
It is ugly, but it works...


## Resources

To build this server I used a  tons of info from the internet. Below
are some very helpfull pages.


 * PXE Booting with an NFS Root File System, a chapter from the excellent
   FreeBSD handbook, http://www.freebsd.org/doc/handbook/network-pxe-nfs.html
 * [Chapter 9.4 about configuring the FreeBSD kernel (chapter from the
   excellent FreeBSD handbook,
   http://www.freebsd.org/doc/en/books/handbook/kernelconfig-config.htm


$Id: freebsd10_disklessjailserver.txt,v 1.2 2013/12/13 14:36:38 matto Exp $

You are viewing proxied material from box.matto.nl. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.