❗️

backwards compatibility

Due to problems we encountered with the approach taken in 0.9.2 it will change drastically with 0.9.3. The changes are so far-reaching FiFo 0.9.3 will not recognize jails created with 0.9.2!

Project-FiFo provides experimental support for BSD jails on FreeBSD.

At this point, FiFo depends on FreeBSD-11 +, RCTL, and vnet support.

Many of the steps below are FreeBSD centric, and we highly recommend to contact the FreeBSD community if you have problems or trouble with them as they are a lot more qualified to answer questions regarding BSD specific topics then we are.

🚧

Be safe

As always:

• make backups
• don't use experimental features on critical systems

Getting Started

compile a kernel with VIMAGE

# exact kernel version does not matter!
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/`freebsd-version -k`/src.txz -o /tmp/src.txz
tar -C / -xzf /tmp/src.txz
cd /usr/src/sys/amd64/conf
cat > FIFOKERNEL <<EOL
include GENERIC
ident FIFOKERNEL

options         VIMAGE # VNET/Vimage support
options         RACCT  # Resource containers
options         RCTL   # same as above
EOL

cd /usr/src
make -j`sysctl -n hw.ncpu` buildkernel KERNCONF=FIFOKERNEL
make -j`sysctl -n hw.ncpu` installkernel KERNCONF=FIFOKERNEL

## enable required modules and enable rcct
cat <<EOF >> /boot/loader.conf
linux64_load="YES"
linux_load="YES"
fdescfs_load="YES"
linprocfs_load="YES"
linsysfs_load="YES"
tmpfs_load="YES"
mac_portacl_load="YES"
kern.racct.enable=1
EOF

reboot

Setting up some ZFS

All FiFo components follow a simple rule: mutable data is stored in /data/<service> best practice is to set up a ZFS filesystem for each of them:

zfs create zroot/data
zfs set mountpoint=/data zroot/data
zfs create zroot/data/sniffle
zfs create zroot/data/snarl
zfs create zroot/data/howl
zfs create zroot/jails
zfs set mountpoint=/zroot/jails zroot/jails

Add package repository

We provide packages for all FiFo components to add the repository please run:

mkdir -p /usr/local/etc/pkg/repos
cat <<EOF > /usr/local/etc/pkg/repos/ProjectFiFo.conf
ProjectFiFo: {
  url: "pkg+https://freebsd.project-fifo.net/rel/amd64/11.0",
  mirror_type: "srv",
  enabled: yes
}
EOF
pkg update

Install core components

FiFo has two sets of components, the management system itself and what needs to be installed on each hypervisor, so lets install the hypervisor

pkg install vmadm chunter zlogin

cat  <<EOF >> /etc/rc.conf
zlogin_enable="YES"
chunter_enable="YES"
vmadm_enable="YES"
EOF

service zlogin start
service chunter start

Note vmadm does not have to be started, it's run as a service to automatically boot jails on system startup that have the autoboot attribute set to true.

install management suite

The management components do not have to live on every hypervisor instead it's enough to install them on one, however for production use where uptime mattes it's recommended to run a cluster for at least 5 nodes.

This component can be run both in a jail on the host.

pkg install fifo-sniffle fifo-snarl fifo-howl fifo-cerberus

cat  <<EOF >> /etc/rc.conf
sniffle_enable="YES"
snarl_enable="YES"
howl_enable="YES"
EOF

The packages will print some more required steps like allowing howl to use privileged ports like 80/443.

kldload  mac_portacl

cat  <<EOF >> /etc/sysctl.conf
security.mac.portacl.enabled=1
security.mac.portacl.rules=uid:1004:tcp:80,uid:1004:tcp:443
EOF

service sniffle start
service snarl start
service howl start

Cerberus isn't a separate service but rather a set of static files for the UI, so does not need to be started.

Networking

vmadm requires a bridge interface to allow VNET interfaces to attach to. If you have multiple networks you can have multiple bridge interfaces. The mapping of a network to its bridge is configured in the vmadm config file.

Setting up the components

📘

IP addresses

Please make sure that for all components, when editing the config file you chan change the IP address to the interface you want to use to communicate over.

vmadm

FiFo provides a vmadm clone for FreeBSD that works with jails. It is API compatible with the SmartOS vmadm in the sense that it provides a subset of its functionality and translates those to native jail equivalents

You might want to configure the config file /usr/local/etc/vmadm.toml and enable vmadm as a startup services.

Both the pool and networks section might need changing

pool = "zroot/jails"
[networks]
admin = "bridge0"

❗️

FreeBSD and bridges

When using a bridge interface it is important to disable some of the hardware features of the NIC. If this is not done FreeBSD will take them down itself for the first VNET interface is added possibly causing a connection loss for multiple seconds. To avoid this add the following flags -tso -lro -rxcsum -txcsum -rxcsum6 -txcsum6 to the rc.conf bringing up your interface for example:

ifconfig_em0="DHCP -tso -lro -rxcsum -txcsum -rxcsum6 -txcsum6"

Initial config

Otherwise please follow tine Installing FiFo guide for the setup.

LeoFS

FiFo uses LeoFS as a local dataset cache, we do not provide own FreeBSD packages for it as they provide it themselfs: http://leo-project.net/

An alternative is to disable LeoFS