General instructions

These instructions cover steps that will initiate the update. Please be aware that depending on your update path additional steps might be required. The version specific section covers the steps required to go from one version to another. While you're on release following these steps on a version update is enough. Things are a bit more complicated when living on the bleeding edge (aka dev). In general you should check here on every update since required steps for a version upgrade will slowly grow during the development process.

❗️

It is highly advised to make a snapshot of the FiFo Zone before updating. While the procedure should be safe it is always better to stay on the side of caution.

Zone

Please verify that your zone is up to date as to the Zone requirements.

When the Zone version changes you are required to update to a new zone! If you use a delegate dataset and store data there as documented you can use vmadm reprovision to update the zone, otherwise you will have to do a fresh installation.

To upgrade the components in a zone it is necessary to install the new packages and restart the services. In addition to this there are sometimes additional steps needed depending on the update.

pkgin -fy up
pkgin install fifo-snarl fifo-sniffle fifo-howl fifo-cerberus fifo-dns
svcadm restart sniffle
svcadm restart snarl
svcadm restart howl

Hypervisors

There are two ways to update the server on the hypervisor.

1. Connect to the hypervisor and run:

/opt/chunter/bin/update -u

It's always good to confirm Chunter is running:

svcs chunter
STATE          STIME    FMRI
online         21:16:28 svc:/network/chunter:default

2. Use fifoadm to update a hypervisor by running:

fifoadm hypervisors update

This will trigger all hypervisors to update.

0.9.3

❗️

Be safe

The upgrade to 0.9.3 requires to upgrade your dataset, this can be done using the reprovision mechanism. However, it is required that a delegated dataset exists and the /data partition is used.

As reprovisioning will destroy the zoon root (but not the delegate!) this is a potentially destructive operation and it is recommended to make a backup! The simplest way to backup your data is to tar up the /data partition (if you changed the settings to somewhere else backup that place).

The 0.9.3 release is built for the 17.2.0 datasets, it is necessary to reprovision your dataset. A simple way is using reprovision the same way the AIO dataset is upgraded.

echo '{"image_uuid": "2f538996-672b-11e7-af09-2331e56e26e9"}' | vmadm reprovision FIFO_ZONE_UUID

After the reprovisioning we will need to reinstall packages as described in Installing FiFo Section 3.

The following lines need to be removed from the sniffle, snarl and howl config files:

## The place the SNMP agent looks for it's configuration.
##
## Default: snmp/conf/
##
## Acceptable values:
##   - text
snmp.config_dir = snmp/conf/

## This is where the SNMP related databases and data gets
## stored, it must be writable for the process.
##
## Default: /var/db/snarl
##
## Acceptable values:
##   - text
snmp.data_dir = /var/db/snarl

With 0.9.3 dataset server management is no longer part of the UI but rather of sniffle so datasets servers need to be added as part of the upgrade. However, for new installations, a default set of servers is added as part of the leofs-init command.

sniffle-admin datasets servers add https://datasets.project-fifo.net/images
sniffle-admin datasets servers add https://bsd.project-fifo.net/images # for free bsd images
sniffle-admin datasets servers add https://datasets.at/images

0.9.0

❗️

New Dataset Version

0.9.0 moves from the 14.4-LTS to the 15.4-LTS (e1faace4-e19b-11e5-928b-83849e2fd94a) dataset having a /data delegate means the zone can simply be re-provisioned with the new dataset and the new packages installed.

0.9.0 removes rocksdb as an backend option for anti entropy, this means the following line in the sniffle, snarl and howl config need to be deleted:

## Default database destination
##
## Default: rocksdb
##
## Acceptable values:
##   - one of: leveldb, rocksdb
hashtree.backend = ...

If it was set to rocksdb then it is also required to remove the old anti entropy trees by deleting /data/<service>/db/anti_entropy and /data/<service>/db/cluster_meta/trees

❗️

LeoFS Data structure

This version changes how LeoFS is installed, adding a /data partition to decouple data and application to allow easier updates. Existing data can either be carried over by creating a new VM and joining / leaving the ring, or by creating a new machine with the same IP and manually copying the data in /var/db/ over. The Upgrading FiFo AIO Dataset section explains how to add a data partition after the fact.

📘

Multiple Datasets

This version adds the ability to have Cerberus display datasets from multiple sources. Please look at /opt/local/fifo-cerberus/config.js.example.

0.8.3

In the howl and dalmatiner frontend config the following configuration parameters have to be renamed:

0.8.2

This release features some additional updates to Tachyon, the tachyon.conf file needs to be updated to be compatible with the current .example file to contain all new values.

This will allow using dimensional metrics for all kstat data gathered by tachyon when used in combination with the Postgres Indexer.

The following lines need to removed from the howl.conf if they are present:

##
## Default: 15
##
## Acceptable values:
##   - an integer
upload.pool_max = 15

##
## Default: 15
##
## Acceptable values:
##   - an integer
download.pool_max = 15

0.8.1

The following section has to be removed form the sniffle config file (/data/sniffle/etc/sniffle.conf):

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
consensus = on

0.8.0

❗️

Default Database change

Please read the notes below on how to change your database to leveldb if you are updating an existing fifo. A migration to rocksdb can be done by migrating nodes.

Ignoring this warning might lead to dataloss and unresponsive systems!

❗️

CPU Support

Sniffle, Snarl and Howl require a CPU with AVX support, both intel and AMD introduced this in 2011, please check with your manufacturer when you are unsure about it.

❗️

Directory Structure and zone configuration!

This version changes how FiFo is installed, adding a /data partition to decouple data and application to allow easier updates. This means that a reinstallation is required. Existing data can either be carried over by creating a new VM and joining / leaving the ring, or by creating a new machine with the same IP and manually copying the data in /var/db/ over.

🚧

Deprecation Warning

0.7.3 makes the v3 api public and marks the v2 API as deprecated, the main change is in the redeploy feature for backups, other commands are simply added.

All database services will use rocksdb as a default database backend starting with this release, if you upgrade from an existing setup you have to add the following lines to the sniffle.conf, snarl.conf and howl.conf !

If you are moving an old installation you need to copy the config files from /opt/local/fifo-<service>/etc/ to /data/<service>/etc or it will use a default config.

Snifle & Snarl

## Default database destination
##
## Default: rocksdb
##
## Acceptable values:
##   - one of: leveldb, rocksdb
hashtree.backend = leveldb

## Default database destination
##
## Default: rocksdb
##
## Acceptable values:
##   - one of: leveldb, rocksdb
db.backend = leveldb

Howl & DDB

## Default database destination
##
## Default: rocksdb
##
## Acceptable values:
##   - one of: leveldb, rocksdb
hashtree.backend = leveldb

0.7.2

❗️

Depreciation Warning

This release removes the 0.1.0 API along with the x-* header fields and the old authentication used by it, please use the v2 api, the oAuth2 authentication and query parameters for full-list and full-list-fields instead.

The following sections have to be remove form the chunter.config

## Enables or disables the ARC stats for this server.
##
## Default: enabled
##
## Acceptable values:
##   - enabled or disabled
kstat.arc = enabled

0.7.1

0.7.1 removes Watchdog, so the watchdog package, if installed, can be removed.

svcadm disable watchdog
svccfg delete watchdog

In addition the following sections need to be removed from all config files:

## the log level of the watchdog log
##
## Default: error
##
## Acceptable values:
##   - one of: debug, info, warning, error
log.watchdog.level = error

## the log level of the watchdog log
##
## Default: fifo
##
## Acceptable values:
##   - text
log.watchdog.cluster = fifo

## The ip of the watchdog server
##
## Acceptable values:
##   - an IP/port pair, e.g. 192.168.1.21:10011
## log.watchdog.host.name = 192.168.1.21:4444

In addition the following sections have to be remove form the chunter.config

## cpu_type used for creating VM's. This only applies to KVM machines,
## 'default' will not set the option at all. Please see 'man vmadm' for details.
##
## Default: default
##
## Acceptable values:
##   - one of: default, qemu64, host
cpu_type = default

## The interval in which the systems ARC status is checked, this is
## purely informational and slowly changing a higher interval is usually
## not a issue.
##
## Default: 30s
##
## Acceptable values:
##   - a time duration with units, e.g. '10s' for 10 seconds
arc_interval = 30s

In the global zone a new component is added fifo-zlogin will be used to handle zlogin consoles for all zones. fifo-zlogin is a minimal component with the goal to not require frequent updates of the services.

It needs to be installed in the GZ before chunter is updated as chunter depends on it.

❗️

steps before the update

Please be sure to disable chunter and epmd in this order! before installing zfifo for the first time!

VERSION=rel
curl -O http://release.project-fifo.net/gz/${VERSION}/fifo_zlogin-latest.gz

0.7.0

❗️

Service restart

Since this version includes a all new Earlang version please make sure to shut down the services before installing the new packages and start them afterwards again.

❗️

SmartOS changes

Newer SmartOS versions have changed the required fields in a image manifest, if you update a old hypervisor to get LX compatibility you should run imgadm delete to fetch the required files.

FiFo 0.7.0 combines the functionality that was formally spread out between wiggle howl and nginx into howl, this means neither nginx nor wiggle are any longer needed. Before updating stop and remove those services.

In addition Jingles is now replaced by Cerberus as a UI, to update you will need to delete Jingles and install the new package.

svcadm disable wiggle
svcadm disable nginx
pkgin remove fifo-wiggle fifo-jingles nginx
pkgin install fifo-cerberus

Chunter

The following lines need to be removed form the config file /opt/chunter/etc/chunter.conf:

## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = chunter

## ...
kstat.metrics = true

🚧

The location of the package has changed. You can get a new update script by running the following command on your hypervisors:
curl http://release.project-fifo.net/gz/rel/chunter-update > /opt/chunter/bin/update
or by changing line 58 of /opt/chunter/bin/update to read BASE=http://release.project-fifo.net/gz/${BRANCH}

Howl

If you already have howl installed you will need to grant the user additional permissions to be able to open reserved ports (http: 80, and https: 443) this can be done with the following command. New installations handle this grant as part of the installation routine.

The setting accpetors was misspelled in the former version and was corrected with this release, so it needs to be corrected to acceptors in the config file.

/usr/sbin/usermod -K defaultpriv=basic,net_privaddr howl

With how now being the API server, the howl.port setting in the config file should be changed to 80, and 443 for the ssl port. In addition the following lines need to be removed form the config file /opt/local/fifo-howl/etc/howl.conf:

## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = howl

Snarl

One more change is that with 0.7.0 we introduce OAuth2 as a authentication method. One change required for this is adding a defualt scope this can be done by the following commands:

snarl-admin scope add default Everything "This allows everything"
snarl-admin scope grant default Everything ...
snarl-admin scope toggle default Everything

The following lines need to be removed from /opt/local/fifo-snarl/etc/snarl.conf

## This section deals with Yubikey support a API key and
## client ID can be obtained from:
## https://upgrade.yubico.com/getapikey/
##
## Acceptable values:
##   - text
yubico.client_id = ...

## The Secret key related that was issued along with
## the Yubico cleint ID.
##
## Acceptable values:
##   - text
yubico.secret_key = ...

## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = snarl

## Default database destination
##
## Default: leveldb
##
## Acceptable values:
##   - one of: leveldb, hanoidb, bitcask
db.backend = leveldb

Sniffle

The following lines need to be removed from /opt/local/fifo-sniffle/etc/sniffle.conf

## The Backend to store large data objects in, this applies to snapshots
## and images. Objects stored in one backend will not be migrated to another if
## this is changed.
##
## Default: internal
##
## Acceptable values:
##   - one of: internal, s3
large_data_backend = internal

##
## Default: fifo-images
##
## Acceptable values:
##   - text
s3.bucket.image = fifo-images

##
## Default: fifo-snapshots
##
## Acceptable values:
##   - text
s3.bucket.snapshot = fifo-snapshots

##
## Default: fifo
##
## Acceptable values:
##   - text
s3.bucket.general = fifo

##
## Acceptable values:
##   - text
## s3.access_key = access_key

##
## Acceptable values:
##   - text
## s3.secret_key = secret_key

##
## Default: 192.168.1.42:8443
##
## Acceptable values:
##   - an IP/port pair, e.g. 192.168.1.42:10011
s3.host = 192.168.1.42:8443


## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = sniffle

## Default database destination
##
## Default: leveldb
##
## Acceptable values:
##   - one of: leveldb, hanoidb, bitcask
db.backend = leveldb

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
dataset.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
hypervisor.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
vm.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
iprange.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
network.aae = on

##
## Default: off
##
## Acceptable values:
##   - one of: on, off
img.aae = off

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
dtrace.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
package.aae = on

0.6.1

When upgrading, the Snarl config file (/opt/local/fifo-snarl/etc/snarl.conf) will always includes the line folsom_ddb.ip = .... If your DDB has not been previously configured to collect FiFo metrics, then this line needs to be commented out or else the "snarl" service will go into maintenance. This will be the case for the majority of users and does not affect any other FiFo services.

0.6.0

Version 0.6.0 of FiFo introduces a feature that allows for multiple parallel authentication realms inside of Snarl <../snarl.html>_. To support this the former global information into a realm can be archived by running the DB update command:

❗️

It is critical that ALL services are running and connected during this update otherwise data loss can occur!

snarl-admin db update default

This will place all users, roles and organizations into the default realm. Another realm can be chosen but it will require configuration of the remaining FiFo services.

0.4.4

With 0.4.4 there is a considerable update to the database. Therefore additional steps need to be taken. Once all services have been updated the following commands need to be run:

❗️

It is critical that ALL services are running and connected during this update otherwise data loss can occur!

sniffle-admin db update
snarl-admin db update

These changes also affect the AAE code. Therefore when AAE is enabled the old AAE data needs to be deleted. This has no impact on the system itself.

rm -r /var/db/sniffle/anti_entropy
rm -r /var/db/snarl/anti_entropy