Project-FiFo

The Project-FiFo Developer Hub

Welcome to the Project-FiFo developer hub. You'll find comprehensive guides and documentation to help you start working with Project-FiFo as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    

Sniffle is Project FiFo's central decision making and datastore service. Sniffle stores data about hypervisors, vms, networks, IP ranges, and packages. Besides being the central datastore, Sniffle also makes system wide decisions such as vm placement.

Sniffle has the following properties:

  • No SPOF when using multiple nodes.
  • Data is stored using LevelDB.
  • Orders actions that take place on Chunter (Project FiFo's hypervisor interface).
  • Communicates changes to Howl (Project FiFo's live communication service) for realtime updates (ie. vm boots, reboots).
  • Commands Snarl (Project FiFo's RBAC service) to grant/revoke rights, and checks for permissions before taking action.

Configuration

Configuration file

Sniffle's configuration file is located in /data/sniffle/etc/sniffle.conf. It is automatically generated on the first install and not overwritten on updates.

Nonetheless the newest version of the file is always located in /opt/local/fifo-sniffle/etc/sniffle.conf.example.

The configuration file is documented in-line but we'll go over go over some more interesting settings here.

Active Anti Entropy (AAE)

AAE is riak's mechanism of background synchronization of systems to ensure a higher data consistency. It was ported to FiFo in order to offer increased stability in multi node setups. AAE increases the required resources considerably and does not have much use with a single system so it is disabled by default.

📘

When having more then one system it is strongly recommended to enable AAE! It is possible to selectively enable and disable for different subsystems.

# enable active anti-entropy subsystem
anti_entropy = on
dataset.aae = on
hypervisor.aae = on
vm.aae = on
iprange.aae = on
network.aae = on
dtrace.aae = on
package.aae = on

Database

FiFo uses LevelDB or RocksDB as its back-end database. LevelDB has many different tune-ables some of the more important ones are.

ring_size

The number of VNodes used by FiFo. This setting can only be changed before the system is booted the first time so choose it carefully. It defaults to 64 but may be set lower for single node installations . Generally >10 vnodes per physical node are a good rule of thumb. The ring_size needs to be a a power of two (2, 4, 8 ... 64 ...).

leveldb.mmap_size

The chunk size of each mmaped file. This has a huge impact of the memory requirements. Since FiFo does not store lots of data a setting of 1MB is a valid value for small and medium installations. The settings can be increased as long as it is ensured that enough memory is present.

Global configuration

In addition to the config files that apply on a per node level there are global configurations that can be changed from one system and are applied globally. Unless otherwise noted all these settings can be changed during runtime.

Sniffle-admin config show


sniffle-admin config show

Shows a list of all settings in the global configuration.

sniffle-admin config set <key> <value>

Sets a global configuration value. The following keys are used.

S3 connection related options

keyexampledescription
storage.s3.hostleo.project-fifo.netFQDN (hostname) of the S3 gateway.
storage.s3.port443HTTPS port of the S3 gateway.
storage.s3.access_keyblablalbaAccess key for the S3 user.
storage.s3.secret_keyblablablaSecret key for the S3 user.
storage.s3.backup_hostoptional will default to normal host
storage.s3.backup_portoptional will default to normal port
storage.s3.backup_access_keyoptional will default to normal key
storage.s3.backup_secret_keyoptional will default to normal key

S3 bucket related options, the backup_ prefixed versions are optional and can be used if different s3 endpoints are used for backups and images.

keyexamplemeaning
storage.s3.general_bucketfifoGeneral S3 bucket to use
storage.s3.snapshot_bucketfifo-backupsS3 Bucket to store snapshots/backups in.
storage.s3.image_bucketfifo-imagesS3 bucket to store images/datasets in.

Other configuration parameters

network.http.proxyhttp://some.proxyProxy used when downloading datasets from the internet.

Administration

The Sniffle admin command is sniffle-admin but many commands can also be accessed via fifoadm command. Please keep in mind that fifoadm is not designed as an every day command but only as a last fall-back when commands are not available through the API.

General management

Sniffle uses the SMF to manage it's running state so it is restarted in the case of crashes and booted accordingly on system start. Sniffle can be enabled, disabled and restarted via: svcadm enable sniffle, svcadm disable sniffle and svcadm restart sniffle.

Cluster management

sniffle-admin cluster join <nodename>@<ip>

Joins a sniffle cluster, please note that all data on the joining (not the joined) node is deleted.

sniffle-admin cluster leave

Cleanly removes a node from the ring. This is helpful when nodes get moved and the cluster downsized

sniffle-admin cluster leave <nodename>@<ip>

Forcefully removes a node from the ring. This can be used after fatal node crash.

sniffle-admin cluster plan

Shows the current plan on how to change the cluster.

sniffle-admin cluster commit

Executes the current cluster plan.

sniffle-admin member-status

Lists the status of each node and the distribution of data over the ring nodes.

+-----------------------------------------------------------------+
|                           Membership                            |
+========+==========+=========+===================================+
| Status | Ring     | Pending | Nodes                             |
+--------+----------+---------+-----------------------------------+
| valid  | 100.0%   |   --    | 'sniffle@192.168.0.254'           |
+--------+----------+---------+-----------+-----------------------+
|Valid:1 | Leaving:0|Exiting:0| Joining:0 | Down:0                |
+--------+----------+---------+-----------+-----------------------+

sniffle-admin ring-status
Gives a extended report on the ring, including hand-offs and downed nodes.

+-------------------------------------------------------------------+
|Claimant                                                           |
+===========+=======================================================+
|Claimant   |'sniffle@192.168.0.254'                                |
+-----------+--------------------+----------------------------------+    
|Status     | up                 |                                  |
+-----------+--------------------+----------------------------------+
|Ring Ready | true               |                                  |
+-----------+--------------------+----------------------------------+ 

+-------------------------------------------------------------------+
| Ownership handoff                                                 |
+===================================================================+
| No pending changes.                                               |
+-------------------------------------------------------------------+

+-------------------------------------------------------------------+
| Unreachable Nodes                                                 |
+===================================================================+
| All nodes are up and reachable                                    |
+-------------------------------------------------------------------+

sniffle-admin status
A simple command that returns the overall cluster status, it returns a proper return code and is useful for scripted rolling updates.

sniffle-admin aae-status
Gives a detailed status on the AAE status of the system.

sniffle-admin ring
Shows the current ring.

Hash                                               Node
-------------------------------------------------- ---------------
                                                 0 sniffle@192.168
   91343852333181432387730302044767688728495783936 sniffle@192.168
  182687704666362864775460604089535377456991567872 sniffle@192.168
  274031556999544297163190906134303066185487351808 sniffle@192.168
  365375409332725729550921208179070754913983135744 sniffle@192.168
  456719261665907161938651510223838443642478919680 sniffle@192.168
  548063113999088594326381812268606132370974703616 sniffle@192.168
  639406966332270026714112114313373821099470487552 sniffle@192.168
  730750818665451459101842416358141509827966271488 sniffle@192.168
  822094670998632891489572718402909198556462055424 sniffle@192.168
  913438523331814323877303020447676887284957839360 sniffle@192.168
 1004782375664995756265033322492444576013453623296 sniffle@192.168
 1096126227998177188652763624537212264741949407232 sniffle@192.168
 1187470080331358621040493926581979953470445191168 sniffle@192.168
 1278813932664540053428224228626747642198940975104 sniffle@192.168
 1370157784997721485815954530671515330927436759040 sniffle@192.168

sniffle-admin init-leofs [<manager-ip>] <gateway-FQDN>
Initializes a LeoFS system for FiFo, sets up users, buckets, permissions and configures sniffle to use this data.

General tasks

sniffle-admin hypervisors
list - lists all available hypervisors
delete <uuid> - removes a hypervisor

sniffle-admin vms
list - lists all VM's
delete <uuid> - deletes a VM

sniffle-admin packages
list - lists all Packages
delete <uuid> - deletes a Package

sniffle-admin datasets
list - lists all Datasets
delete <uuid> - deletes a Dataset

sniffle-admin datasets servers
list - lists all Datasets Servers
add <url> - add Dataset Server
* remove <url> - lists all Datasets Servers

DB debugging and recovery commands

❗️

Use with caution

The following commands are for experienced users only, they allow direct manipulation of the database for debugging and recovery proposes. DO NOT USE THIS UNLESS YOU ARE WILLING TO LOSE YOUR DATA!

sniffle-admin db list [-p <prefix>]
raw listing of DB keys with partitions on the local node

[root@fifo01 ~]# sniffle-admin db list -p vm

                                                 0 sniffle@192.168
------------------------------------------------------------------
vm56cb73b6-297d-45e3-b8d0-211ebd1f65c9
vm4fdfbd27-c51b-459f-9d9f-623ece60a4ce

   91343852333181432387730302044767688728495783936 sniffle@192.168
------------------------------------------------------------------

  182687704666362864775460604089535377456991567872 sniffle@192.168
------------------------------------------------------------------

  274031556999544297163190906134303066185487351808 sniffle@192.168
------------------------------------------------------------------
vmd2d00f36-d2f8-40fa-b12c-4ff1f2d4e65b

  365375409332725729550921208179070754913983135744 sniffle@192.168
------------------------------------------------------------------
vmd2d00f36-d2f8-40fa-b12c-4ff1f2d4e65b

  456719261665907161938651510223838443642478919680 sniffle@192.168
------------------------------------------------------------------

  548063113999088594326381812268606132370974703616 sniffle@192.168
------------------------------------------------------------------

  639406966332270026714112114313373821099470487552 sniffle@192.168
------------------------------------------------------------------

  730750818665451459101842416358141509827966271488 sniffle@192.168
------------------------------------------------------------------

  822094670998632891489572718402909198556462055424 sniffle@192.168
------------------------------------------------------------------

  913438523331814323877303020447676887284957839360 sniffle@192.168
------------------------------------------------------------------

 1004782375664995756265033322492444576013453623296 sniffle@192.168
------------------------------------------------------------------
vmb605fdc8-2296-4580-9b05-2692c1b6abee

 1096126227998177188652763624537212264741949407232 sniffle@192.168
------------------------------------------------------------------
vmb605fdc8-2296-4580-9b05-2692c1b6abee

 1187470080331358621040493926581979953470445191168 sniffle@192.168
------------------------------------------------------------------

 1278813932664540053428224228626747642198940975104 sniffle@192.168
------------------------------------------------------------------

 1370157784997721485815954530671515330927436759040 sniffle@192.168
------------------------------------------------------------------
vm5ab7bd92-f67c-4a85-bdfd-ff1e96fa17d6
vm56cb73b6-297d-45e3-b8d0-211ebd1f65c9
vm4fdfbd27-c51b-459f-9d9f-623ece60a4ce
vm2e081394-7d1b-436a-bde1-08cdb28636f7

sniffle-admin db list <hash> [<prefix>]
Lists all keys in a given partition/hash with a given prefix.

[root@fifo01 ~]# sniffle-admin db list 1370157784997721485815954530671515330927436759040 package
f1bdd450-5c5a-405e-b796-953767453f54
5a39fd08-df9c-489f-ace7-ad5c425ef35f

sniffle-admin db get <hash> <prefix> <key>
Prints the database object for a given partition prefix and key

[root@fifo01 ~]# sniffle-admin db get 1370157784997721485815954530671515330927436759040 package f1bdd450-5c5a-405e-b796-953767453f54
{ft_obj,{package_1,{<<"f1bdd450-5c5a-405e-b796-953767453f54">>,
                    1442239955512645701},
                   {<<"test2">>,1442239955512645701},
                   {undefined,1},
                   {<<"off">>,1},
                   {100,1442239955509109216},
                   {undefined,1},
                   {undefined,1},
                   {10,1442239955514756482},
                   {1024,1442239955516002893},
                   {undefined,1},
                   {[],
                    {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                    {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                            []}}}},
                   {[],
                    {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                    {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                            []}}}}},
        [{'sniffle@192.168.1.42',{3,63609459155}},
         {'sniffle@192.168.1.41',{1,63609459155}}]}

sniffle-admin db delete <hash> <prefix> <key>
Deletes a key from a partition, please be aware this is NOT primarily to delete objects, it will still retain any other copies, this is purely for direct manipulation!

Log Files

With Version 0.8.0 the log file path changes:
FiFo uses extensive logging to make debugging issue and understanding behavior. The log files are located in /data/sniffle/log/. There are multiple log files with various severities.

Deprecated: (Version 0.7.x and before)
FiFo uses extensive logging to make debugging issue and understanding behavior. The log files are located in /var/log/sniffle/. There are multiple log files with various severities.

debug.log
By default the debug log is disabled, it is very verbose and should not be enabled in production systems. To enable it uncomment the following line in the sniffle.conf

log.debug.file = /var/log/sniffle/debug.log

console.log
This file contains logs of the level info and above, usually all interesting logs can be found here.

error.log
This files contains errors, it usually should be mostly empty but please keep in mind that failing is not a uncommon practice to deal with unexpected behavior so sporadic entries might just be fine.

Updated 2 years ago

Sniffle


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.