Sniffle
Sniffle is Project FiFo's central decision making and datastore service. Sniffle stores data about hypervisors, vms, networks, IP ranges, and packages. Besides being the central datastore, Sniffle also makes system wide decisions such as vm placement.
Sniffle has the following properties:
- No SPOF when using multiple nodes.
- Data is stored using LevelDB.
- Orders actions that take place on Chunter (Project FiFo's hypervisor interface).
- Communicates changes to Howl (Project FiFo's live communication service) for realtime updates (ie. vm boots, reboots).
- Commands Snarl (Project FiFo's RBAC service) to grant/revoke rights, and checks for permissions before taking action.
Configuration
Configuration file
Sniffle's configuration file is located in /data/sniffle/etc/sniffle.conf. It is automatically generated on the first install and not overwritten on updates.
Nonetheless the newest version of the file is always located in /opt/local/fifo-sniffle/etc/sniffle.conf.example.
The configuration file is documented in-line but we'll go over go over some more interesting settings here.
Active Anti Entropy (AAE)
AAE is riak's mechanism of background synchronization of systems to ensure a higher data consistency. It was ported to FiFo in order to offer increased stability in multi node setups. AAE increases the required resources considerably and does not have much use with a single system so it is disabled by default.
When having more then one system it is strongly recommended to enable AAE! It is possible to selectively enable and disable for different subsystems.
# enable active anti-entropy subsystem
anti_entropy = on
dataset.aae = on
hypervisor.aae = on
vm.aae = on
iprange.aae = on
network.aae = on
dtrace.aae = on
package.aae = on
Database
FiFo uses LevelDB or RocksDB as its back-end database. LevelDB has many different tune-ables some of the more important ones are.
ring_size
The number of VNodes used by FiFo. This setting can only be changed before the system is booted the first time so choose it carefully. It defaults to 64 but may be set lower for single node installations . Generally >10 vnodes per physical node are a good rule of thumb. The ring_size needs to be a a power of two (2, 4, 8 ... 64 ...).
leveldb.mmap_size
The chunk size of each mmaped file. This has a huge impact of the memory requirements. Since FiFo does not store lots of data a setting of 1MB is a valid value for small and medium installations. The settings can be increased as long as it is ensured that enough memory is present.
Global configuration
In addition to the config files that apply on a per node level there are global configurations that can be changed from one system and are applied globally. Unless otherwise noted all these settings can be changed during runtime.
Sniffle-admin config show
sniffle-admin config show
Shows a list of all settings in the global configuration.
sniffle-admin config set <key> <value>
Sets a global configuration value. The following keys are used.
S3 connection related options
| key | example | description |
|---|---|---|
| storage.s3.host | leo.project-fifo.net | FQDN (hostname) of the S3 gateway. |
| storage.s3.port | 443 | HTTPS port of the S3 gateway. |
| storage.s3.access_key | blablalba | Access key for the S3 user. |
| storage.s3.secret_key | blablabla | Secret key for the S3 user. |
| storage.s3.backup_host | optional will default to normal host | |
| storage.s3.backup_port | optional will default to normal port | |
| storage.s3.backup_access_key | optional will default to normal key | |
| storage.s3.backup_secret_key | optional will default to normal key |
S3 bucket related options, the backup_ prefixed versions are optional and can be used if different s3 endpoints are used for backups and images.
| key | example | meaning |
|---|---|---|
| storage.s3.general_bucket | fifo | General S3 bucket to use |
| storage.s3.snapshot_bucket | fifo-backups | S3 Bucket to store snapshots/backups in. |
| storage.s3.image_bucket | fifo-images | S3 bucket to store images/datasets in. |
Other configuration parameters
| network.http.proxy | http://some.proxy | Proxy used when downloading datasets from the internet. |
Administration
The Sniffle admin command is sniffle-admin but many commands can also be accessed via fifoadm command. Please keep in mind that fifoadm is not designed as an every day command but only as a last fall-back when commands are not available through the API.
General management
Sniffle uses the SMF to manage it's running state so it is restarted in the case of crashes and booted accordingly on system start. Sniffle can be enabled, disabled and restarted via: svcadm enable sniffle, svcadm disable sniffle and svcadm restart sniffle.
Cluster management
sniffle-admin cluster join <nodename>@<ip>
sniffle-admin cluster join <nodename>@<ip>Joins a sniffle cluster, please note that all data on the joining (not the joined) node is deleted.
sniffle-admin cluster leave
sniffle-admin cluster leaveCleanly removes a node from the ring. This is helpful when nodes get moved and the cluster downsized
sniffle-admin cluster leave <nodename>@<ip>
sniffle-admin cluster leave <nodename>@<ip>Forcefully removes a node from the ring. This can be used after fatal node crash.
sniffle-admin cluster plan
sniffle-admin cluster planShows the current plan on how to change the cluster.
sniffle-admin cluster commit
sniffle-admin cluster commitExecutes the current cluster plan.
sniffle-admin member-status
sniffle-admin member-statusLists the status of each node and the distribution of data over the ring nodes.
+-----------------------------------------------------------------+
| Membership |
+========+==========+=========+===================================+
| Status | Ring | Pending | Nodes |
+--------+----------+---------+-----------------------------------+
| valid | 100.0% | -- | 'sniffle@192.168.0.254' |
+--------+----------+---------+-----------+-----------------------+
|Valid:1 | Leaving:0|Exiting:0| Joining:0 | Down:0 |
+--------+----------+---------+-----------+-----------------------+
sniffle-admin ring-status
Gives a extended report on the ring, including hand-offs and downed nodes.
+-------------------------------------------------------------------+
|Claimant |
+===========+=======================================================+
|Claimant |'sniffle@192.168.0.254' |
+-----------+--------------------+----------------------------------+
|Status | up | |
+-----------+--------------------+----------------------------------+
|Ring Ready | true | |
+-----------+--------------------+----------------------------------+
+-------------------------------------------------------------------+
| Ownership handoff |
+===================================================================+
| No pending changes. |
+-------------------------------------------------------------------+
+-------------------------------------------------------------------+
| Unreachable Nodes |
+===================================================================+
| All nodes are up and reachable |
+-------------------------------------------------------------------+
sniffle-admin status
A simple command that returns the overall cluster status, it returns a proper return code and is useful for scripted rolling updates.
sniffle-admin aae-status
Gives a detailed status on the AAE status of the system.
sniffle-admin ring
Shows the current ring.
Hash Node
-------------------------------------------------- ---------------
0 sniffle@192.168
91343852333181432387730302044767688728495783936 sniffle@192.168
182687704666362864775460604089535377456991567872 sniffle@192.168
274031556999544297163190906134303066185487351808 sniffle@192.168
365375409332725729550921208179070754913983135744 sniffle@192.168
456719261665907161938651510223838443642478919680 sniffle@192.168
548063113999088594326381812268606132370974703616 sniffle@192.168
639406966332270026714112114313373821099470487552 sniffle@192.168
730750818665451459101842416358141509827966271488 sniffle@192.168
822094670998632891489572718402909198556462055424 sniffle@192.168
913438523331814323877303020447676887284957839360 sniffle@192.168
1004782375664995756265033322492444576013453623296 sniffle@192.168
1096126227998177188652763624537212264741949407232 sniffle@192.168
1187470080331358621040493926581979953470445191168 sniffle@192.168
1278813932664540053428224228626747642198940975104 sniffle@192.168
1370157784997721485815954530671515330927436759040 sniffle@192.168
sniffle-admin init-leofs [<manager-ip>] <gateway-FQDN>
Initializes a LeoFS system for FiFo, sets up users, buckets, permissions and configures sniffle to use this data.
General tasks
sniffle-admin hypervisors
list - lists all available hypervisors
delete <uuid> - removes a hypervisor
sniffle-admin vms
list - lists all VM's
delete <uuid> - deletes a VM
sniffle-admin packages
list - lists all Packages
delete <uuid> - deletes a Package
sniffle-admin datasets
list - lists all Datasets
delete <uuid> - deletes a Dataset
sniffle-admin datasets servers
list - lists all Datasets Servers
add <url> - add Dataset Server
* remove <url> - lists all Datasets Servers
DB debugging and recovery commands
Use with caution
The following commands are for experienced users only, they allow direct manipulation of the database for debugging and recovery proposes. DO NOT USE THIS UNLESS YOU ARE WILLING TO LOSE YOUR DATA!
sniffle-admin db list [-p <prefix>]
raw listing of DB keys with partitions on the local node
[root@fifo01 ~]# sniffle-admin db list -p vm
0 sniffle@192.168
------------------------------------------------------------------
vm56cb73b6-297d-45e3-b8d0-211ebd1f65c9
vm4fdfbd27-c51b-459f-9d9f-623ece60a4ce
91343852333181432387730302044767688728495783936 sniffle@192.168
------------------------------------------------------------------
182687704666362864775460604089535377456991567872 sniffle@192.168
------------------------------------------------------------------
274031556999544297163190906134303066185487351808 sniffle@192.168
------------------------------------------------------------------
vmd2d00f36-d2f8-40fa-b12c-4ff1f2d4e65b
365375409332725729550921208179070754913983135744 sniffle@192.168
------------------------------------------------------------------
vmd2d00f36-d2f8-40fa-b12c-4ff1f2d4e65b
456719261665907161938651510223838443642478919680 sniffle@192.168
------------------------------------------------------------------
548063113999088594326381812268606132370974703616 sniffle@192.168
------------------------------------------------------------------
639406966332270026714112114313373821099470487552 sniffle@192.168
------------------------------------------------------------------
730750818665451459101842416358141509827966271488 sniffle@192.168
------------------------------------------------------------------
822094670998632891489572718402909198556462055424 sniffle@192.168
------------------------------------------------------------------
913438523331814323877303020447676887284957839360 sniffle@192.168
------------------------------------------------------------------
1004782375664995756265033322492444576013453623296 sniffle@192.168
------------------------------------------------------------------
vmb605fdc8-2296-4580-9b05-2692c1b6abee
1096126227998177188652763624537212264741949407232 sniffle@192.168
------------------------------------------------------------------
vmb605fdc8-2296-4580-9b05-2692c1b6abee
1187470080331358621040493926581979953470445191168 sniffle@192.168
------------------------------------------------------------------
1278813932664540053428224228626747642198940975104 sniffle@192.168
------------------------------------------------------------------
1370157784997721485815954530671515330927436759040 sniffle@192.168
------------------------------------------------------------------
vm5ab7bd92-f67c-4a85-bdfd-ff1e96fa17d6
vm56cb73b6-297d-45e3-b8d0-211ebd1f65c9
vm4fdfbd27-c51b-459f-9d9f-623ece60a4ce
vm2e081394-7d1b-436a-bde1-08cdb28636f7
sniffle-admin db list <hash> [<prefix>]
Lists all keys in a given partition/hash with a given prefix.
[root@fifo01 ~]# sniffle-admin db list 1370157784997721485815954530671515330927436759040 package
f1bdd450-5c5a-405e-b796-953767453f54
5a39fd08-df9c-489f-ace7-ad5c425ef35f
sniffle-admin db get <hash> <prefix> <key>
Prints the database object for a given partition prefix and key
[root@fifo01 ~]# sniffle-admin db get 1370157784997721485815954530671515330927436759040 package f1bdd450-5c5a-405e-b796-953767453f54
{ft_obj,{package_1,{<<"f1bdd450-5c5a-405e-b796-953767453f54">>,
1442239955512645701},
{<<"test2">>,1442239955512645701},
{undefined,1},
{<<"off">>,1},
{100,1442239955509109216},
{undefined,1},
{undefined,1},
{10,1442239955514756482},
{1024,1442239955516002893},
{undefined,1},
{[],
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}}},
{[],
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}}}},
[{'sniffle@192.168.1.42',{3,63609459155}},
{'sniffle@192.168.1.41',{1,63609459155}}]}
sniffle-admin db delete <hash> <prefix> <key>
Deletes a key from a partition, please be aware this is NOT primarily to delete objects, it will still retain any other copies, this is purely for direct manipulation!
Log Files
With Version 0.8.0 the log file path changes:
FiFo uses extensive logging to make debugging issue and understanding behavior. The log files are located in /data/sniffle/log/. There are multiple log files with various severities.
Deprecated: (Version 0.7.x and before)
FiFo uses extensive logging to make debugging issue and understanding behavior. The log files are located in /var/log/sniffle/. There are multiple log files with various severities.
debug.log
By default the debug log is disabled, it is very verbose and should not be enabled in production systems. To enable it uncomment the following line in the sniffle.conf
log.debug.file = /var/log/sniffle/debug.log
console.log
This file contains logs of the level info and above, usually all interesting logs can be found here.
error.log
This files contains errors, it usually should be mostly empty but please keep in mind that failing is not a uncommon practice to deal with unexpected behavior so sporadic entries might just be fine.
Updated almost 7 years ago