General instructions

These instructions cover steps that will initiate the update. Please be aware that depending on your update path additional steps might be required. The version specific section covers the steps required to go from one version to another. While you're on release following these steps on a version update is enough. Things are a bit more complicated when living on the bleeding edge (aka dev). In general you should check here on every update since required steps for a version upgrade will slowly grow during the development process.

❗️

It is highly advised to make a snapshot of the FiFo Zone before updating. While the procedure should be safe it is always better to stay on the side of caution.

Zone

To upgrade the components in a zone it is necessary to install the new packages and restart the services. In addition to this there are sometimes additional steps needed depending on the update.

pkgin -fy up
pkgin install fifo-snarl fifo-sniffle fifo-howl fifo-cerberus fifo-watchdog fifo-dns
svcadm restart sniffle
svcadm restart snarl
svcadm restart howl

Hypervisors

There are two ways to update the server on the hypervisor.

1. Connect to the hypervisor and run:

/opt/chunter/bin/update -u

It's always good to confirm Chunter is running:

svcs chunter
STATE          STIME    FMRI
online         21:16:28 svc:/network/chunter:default

2. Use fifoadm to update a hypervisor by running:

fifoadm hypervisors update

This will trigger all hypervisors to update.

0.8.3

In the howl and dalmatiner frontend config the following configuration parameters have to be renamed:

0.8.2

This release features some additional updates to Tachyon, the tachyon.conf file needs to be updated to be compatible with the current .example file to contain all new values.

This will allow using dimensional metrics for all kstat data gathered by tachyon when used in combination with the Postgres Indexer.

The following lines need to removed from the howl.conf if they are present:

##
## Default: 15
##
## Acceptable values:
##   - an integer
upload.pool_max = 15

##
## Default: 15
##
## Acceptable values:
##   - an integer
download.pool_max = 15

0.8.1

The following section has to be removed form the sniffle config file (/data/sniffle/etc/sniffle.conf):

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
consensus = on

0.8.0

❗️

Default Database change

Please read the notes below on how to change your database to leveldb if you are updating an existing fifo. A migration to rocksdb can be done by migrating nodes.

Ignoring this warning might lead to dataloss and unresponsive systems!

❗️

CPU Support

Sniffle, Snarl and Howl require a CPU with AVX support, both intel and AMD introduced this in 2011, please check with your manufacturer when you are unsure about it.

❗️

Directory Structure and zone configuration!

This version changes how FiFo is installed, adding a /data partition to decouple data and application to allow easier updates. This means that a reinstallation is required. Existing data can either be carried over by creating a new VM and joining / leaving the ring, or by creating a new machine with the same IP and manually copying the data in /var/db/ over.

🚧

Deprecation Warning

0.7.3 makes the v3 api public and marks the v2 API as deprecated, the main change is in the redeploy feature for backups, other commands are simply added.

All database services will use rocksdb as a default database backend starting with this release, if you upgrade from an existing setup you have to add the following lines to the sniffle.conf, snarl.conf and howl.conf !

If you are moving an old installation you need to copy the config files from /opt/local/fifo-<service>/etc/ to /data/<service>/etc or it will use a default config.

Snifle & Snarl

## Default database destination
##
## Default: rocksdb
##
## Acceptable values:
##   - one of: leveldb, rocksdb
hashtree.backend = leveldb

## Default database destination
##
## Default: rocksdb
##
## Acceptable values:
##   - one of: leveldb, rocksdb
db.backend = leveldb

Howl & DDB

## Default database destination
##
## Default: rocksdb
##
## Acceptable values:
##   - one of: leveldb, rocksdb
hashtree.backend = leveldb

0.7.2

❗️

Depreciation Warning

This release removes the 0.1.0 API along with the x-* header fields and the old authentication used by it, please use the v2 api, the oAuth2 authentication and query parameters for full-list and full-list-fields instead.

The following sections have to be remove form the chunter.config

## Enables or disables the ARC stats for this server.
##
## Default: enabled
##
## Acceptable values:
##   - enabled or disabled
kstat.arc = enabled

0.7.1

0.7.1 removes Watchdog, so the watchdog package, if installed, can be removed.

svcadm disable watchdog
svccfg delete watchdog

In addition the following sections need to be removed from all config files:

## the log level of the watchdog log
##
## Default: error
##
## Acceptable values:
##   - one of: debug, info, warning, error
log.watchdog.level = error

## the log level of the watchdog log
##
## Default: fifo
##
## Acceptable values:
##   - text
log.watchdog.cluster = fifo

## The ip of the watchdog server
##
## Acceptable values:
##   - an IP/port pair, e.g. 192.168.1.21:10011
## log.watchdog.host.name = 192.168.1.21:4444

In addition the following sections have to be remove form the chunter.config

## cpu_type used for creating VM's. This only applies to KVM machines,
## 'default' will not set the option at all. Please see 'man vmadm' for details.
##
## Default: default
##
## Acceptable values:
##   - one of: default, qemu64, host
cpu_type = default

## The interval in which the systems ARC status is checked, this is
## purely informational and slowly changing a higher interval is usually
## not a issue.
##
## Default: 30s
##
## Acceptable values:
##   - a time duration with units, e.g. '10s' for 10 seconds
arc_interval = 30s

In the global zone a new component is added fifo-zlogin will be used to handle zlogin consoles for all zones. fifo-zlogin is a minimal component with the goal to not require frequent updates of the services.

It needs to be installed in the GZ before chunter is updated as chunter depends on it.

❗️

steps before the update

Please be sure to disable chunter and epmd in this order! before installing zfifo for the first time!

VERSION=rel
curl -O http://release.project-fifo.net/gz/${VERSION}/fifo_zlogin-latest.gz

0.7.0

❗️

Service restart

Since this version includes a all new Earlang version please make sure to shut down the services before installing the new packages and start them afterwards again.

❗️

SmartOS changes

Newer SmartOS versions have changed the required fields in a image manifest, if you update a old hypervisor to get LX compatibility you should run imgadm delete to fetch the required files.

FiFo 0.7.0 combines the functionality that was formally spread out between wiggle howl and nginx into howl, this means neither nginx nor wiggle are any longer needed. Before updating stop and remove those services.

In addition Jingles is now replaced by Cerberus as a UI, to update you will need to delete Jingles and install the new package.

svcadm disable wiggle
svcadm disable nginx
pkgin remove fifo-wiggle fifo-jingles nginx
pkgin install fifo-cerberus

Chunter

The following lines need to be removed form the config file /opt/chunter/etc/chunter.conf:

## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = chunter

## ...
kstat.metrics = true

🚧

The location of the package has changed. You can get a new update script by running the following command on your hypervisors:
curl http://release.project-fifo.net/gz/rel/chunter-update > /opt/chunter/bin/update
or by changing line 58 of /opt/chunter/bin/update to read BASE=http://release.project-fifo.net/gz/${BRANCH}

Howl

If you already have howl installed you will need to grant the user additional permissions to be able to open reserved ports (http: 80, and https: 443) this can be done with the following command. New installations handle this grant as part of the installation routine.

The setting accpetors was misspelled in the former version and was corrected with this release, so it needs to be corrected to acceptors in the config file.

/usr/sbin/usermod -K defaultpriv=basic,net_privaddr howl

With how now being the API server, the howl.port setting in the config file should be changed to 80, and 443 for the ssl port. In addition the following lines need to be removed form the config file /opt/local/fifo-howl/etc/howl.conf:

## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = howl

Snarl

One more change is that with 0.7.0 we introduce OAuth2 as a authentication method. One change required for this is adding a defualt scope this can be done by the following commands:

snarl-admin scope add default Everything "This allows everything"
snarl-admin scope grant default Everything ...
snarl-admin scope toggle default Everything

The following lines need to be removed from /opt/local/fifo-snarl/etc/snarl.conf

## This section deals with Yubikey support a API key and
## client ID can be obtained from:
## https://upgrade.yubico.com/getapikey/
##
## Acceptable values:
##   - text
yubico.client_id = ...

## The Secret key related that was issued along with
## the Yubico cleint ID.
##
## Acceptable values:
##   - text
yubico.secret_key = ...

## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = snarl

## Default database destination
##
## Default: leveldb
##
## Acceptable values:
##   - one of: leveldb, hanoidb, bitcask
db.backend = leveldb

Sniffle

The following lines need to be removed from /opt/local/fifo-sniffle/etc/sniffle.conf

## The Backend to store large data objects in, this applies to snapshots
## and images. Objects stored in one backend will not be migrated to another if
## this is changed.
##
## Default: internal
##
## Acceptable values:
##   - one of: internal, s3
large_data_backend = internal

##
## Default: fifo-images
##
## Acceptable values:
##   - text
s3.bucket.image = fifo-images

##
## Default: fifo-snapshots
##
## Acceptable values:
##   - text
s3.bucket.snapshot = fifo-snapshots

##
## Default: fifo
##
## Acceptable values:
##   - text
s3.bucket.general = fifo

##
## Acceptable values:
##   - text
## s3.access_key = access_key

##
## Acceptable values:
##   - text
## s3.secret_key = secret_key

##
## Default: 192.168.1.42:8443
##
## Acceptable values:
##   - an IP/port pair, e.g. 192.168.1.42:10011
s3.host = 192.168.1.42:8443


## the log level of the watchdog log
##
## Default:
##
## Acceptable values:
##   - text
log.watchdog.service = sniffle

## Default database destination
##
## Default: leveldb
##
## Acceptable values:
##   - one of: leveldb, hanoidb, bitcask
db.backend = leveldb

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
dataset.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
hypervisor.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
vm.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
iprange.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
network.aae = on

##
## Default: off
##
## Acceptable values:
##   - one of: on, off
img.aae = off

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
dtrace.aae = on

##
## Default: on
##
## Acceptable values:
##   - one of: on, off
package.aae = on

0.6.1

When upgrading, the Snarl config file (/opt/local/fifo-snarl/etc/snarl.conf) will always includes the line folsom_ddb.ip = .... If your DDB has not been previously configured to collect FiFo metrics, then this line needs to be commented out or else the "snarl" service will go into maintenance. This will be the case for the majority of users and does not affect any other FiFo services.

0.6.0

Version 0.6.0 of FiFo introduces a feature that allows for multiple parallel authentication realms inside of Snarl <../snarl.html>_. To support this the former global information into a realm can be archived by running the DB update command:

❗️

It is critical that ALL services are running and connected during this update otherwise data loss can occur!

snarl-admin db update default

This will place all users, roles and organizations into the default realm. Another realm can be chosen but it will require configuration of the remaining FiFo services.

0.4.4

With 0.4.4 there is a considerable update to the database. Therefore additional steps need to be taken. Once all services have been updated the following commands need to be run:

❗️

It is critical that ALL services are running and connected during this update otherwise data loss can occur!

sniffle-admin db update
snarl-admin db update

These changes also affect the AAE code. Therefore when AAE is enabled the old AAE data needs to be deleted. This has no impact on the system itself.

rm -r /var/db/sniffle/anti_entropy
rm -r /var/db/snarl/anti_entropy