Howl is Project FiFo's live communications service. Howl live updates the user interface (Cerberus)
when changes occur in the back-end. Howl checks with snarl (Project FiFo's RBAC service) for which
events a user has permission to see.

Howl is a riak_core and web-socket based message delivery system and web server for the API and UI with the following properties:

• No SPOF when using multiple nodes.
• The front end is a cowboy web server that accepts web socket connections.
• The back-end is a simple TPC/BERT based protocol.
• Each riak_core node offers both a front end and a back-end, those front and back ends are equal, as in sending messages to any back-end will reach any front end.
• Messages are send to channels, and clients can subscribe to channels.
• Provides the complete API for Project FiFo.
• Speaks to Snarl (Project FiFo's RBAC service) to authenticate users and check permissions.
• Speaks to Sniffle (Project FiFo's central control service) to get information and data.

Configuration

Configuration file

As with version 0.8.0 and the delegated dataset the configuration file of howl is located in /data/howl/etc/howl.conf

Depreciated:

Howl's configuration file is located in /opt/local/fifo-howl/etc/howl.conf it is automatically generated on the first install and not overwritten on updates.

Nonetheless the newest version of the file is always located in /opt/local/fifo-howl/etc/howl.conf.example.

The configuration file is documented in-line but we'll go over go over some more interesting settings here.

listening_ip
The TCP IP that mdns messages arrive to, this needs to be set where wiggle can reach sniffle, snarl and howl.

HTTP(S)

compression
Enable or disable compression for http, either on or off.

port
The port that wiggle listens for HTTP traffic.

ssl
Enable or disable SSL, must be either on or off.

ssl.port
The port that wiggle listens for HTTPS traffic.

ssl.cacertfile
The SSL CA certificate, this is auto-generated on the installation.

ssl.certfile
The SSL Server certificate, this is auto-generated on the installation.

ssl.keyfile
The SSL Key file, this is auto-generated on the installation.

acceptors
Number of acceptor processes that are kept ready.

DalmatinerDB / Tachyon

backend_server

This needs to be set to a DalmatinerDB database server. If set the /metrics endpoints for both hypervisors and vms will return the related performance metrics.

Caching

❗️

Caching done by Howl does not affect the other services, requests in Sniffle / Snarl are always handled with live data! Requests that modify data will never be cached.

Howl has a build in cache. This cache will cache requests that otherwise need to be handled by Sniffle or Snarl - adding latency and computation time to the request.

However as with most things caching has a trade-off: in exchange for much faster responses the liveliness of data suffers. This means that it is possible that a request will provide stale data. Howl itself tries to invalidate cached elements whenever possible but when more then one Howl service is running it is no longer possible to do this over multiple nodes.

On the other hand most of FiFo's data is immutable making it easy to cache since invalidation is only necessary on delete. To take optimal advantage of this fact Howl lets the user configure the cache lifetime on a per datafile level.

When used the cache works in a two tiered model where each tier is defined by a TTL. If the level 1 TTL is hit the cached object is returned without questioning its correctness. If the level 2 TTL (which should be larger then the level 1 TTL) is hit the object is considered suspect but not necessarily wrong - Howl will return the object for this request but then invalidate it and asynchronously update its cache so that further requests get a fresh element.

The two tier model allows short L1 TTLs and longer L2 TTLs making for a good trade-off between liveliness and performance. Nonetheless by default the cache is disabled to guarantee no unexpected behavior!

caching
Enables or disables the cache in Howl. When turned off only tokens (to authenticate) are cached with a very short TTL to speed up quick successive requests as they are a very common pattern.

ttl.element.*.l1
Level 1 time to live for an element. When the element is requested again before the L1 TTL is met it is directly served from the cache. The defaults for the L1 TTL are rather short and reflect the chance of which the object might change (i.e. packages have a much higher TTL then vms).

ttl.element.*.l2
Level 2 time to live for an element. When this element is requested again before the L2 TTL is met the cached value is returned but then invalidated and asynchronously updated. The defaults for the L2 TTL are somewhat high since even when they provide stale data this data is only send once and a simple recheck will provide new data.

ttl.list.*.l1
Analog to ttl.element.*.l1 but for list requests. Since they are much harder to cache and the chance of changes for 'all' elements is much higher then for a single one this TTL is shorter then it's element counterpart.

ttl.list.*.l2
Analog to ttl.element.*.l2 but for list requests. Since they are much harder to cache and the chance of changes for 'all' elements is much higher then for a single one this TTL is shorter then it's element counterpart.

Administration

The Howl admin command is howl-admin.

General management

Howl uses the SMF to manage it's running state so it is restarted in the case of crashes and booted accordingly on system start. Howl can be enabled, disabled and restarted via: svcadm enable howl, svcadm disable howl and svcadm restart howl

Updating

Howl can be updated by three simple steps.

1. Installing the new package

pkgin -fy update
pkgin -fy install fifo-howl

2. Updating the config

After the newest package is installed the config file should be checked for changes and edited if needed. The .example file will always contain the newest version of the config diff is a handy tool to see if some settings need to be added to the existing file.

diff /opt/local/fifo-howl/etc/howl.conf /opt/local/fifo-howl/etc/howl.conf.example
vi /opt/local/fifo-howl/etc/howl.conf

3. Restarting the service

After the config is updated the service needs to be restarted. If Howl is running in a clustered configuration and has more then N nodes it is often possible to do a rolling update by restarting one by one.

Cluster management

howl-admin cluster join <nodename>@<ip>

Joins a Howl cluster, please note that all data on the joining (not the joined) node is deleted.

howl-admin cluster leave

Cleanly removes a node from the ring. This is helpful when nodes get moved and the cluster downsized

howl-admin cluster leave <nodename>@<ip>

Removes a node from the ring. This can be used after fatal node cluster.

howl-admin cluster plan

Shows the current plan on how to change the cluster.

howl-admin cluster commit

Executes the current cluster plan.

howl-admin member-status

Lists the status of each node and the distribution of data over the ring nodes.

================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid     100.0%      --      'howl@172.16.0.254'
-------------------------------------------------------------------------------
Valid:1 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

howl-admin ring-status
Gives a extended report on the ring. Including hand-offs and downed nodes.

================================== Claimant ===================================
Claimant:  'howl@172.16.0.254'
Status:     up
Ring Ready: true

============================== Ownership Hand-off ==============================
No pending changes.

============================== Unreachable Nodes ==============================
All nodes are up and reachable

howl-admin status
A simple command that returns the overall cluster status. It returns a proper return code and is useful for scripted rolling updates.

howl-admin aae-status
Gives a detailed status on the AAE status of the system.

Log Files

With Version 0.8.0 the path to the logfiles changed to:
FiFo uses extensive logging to make debugging issue and understanding behavior. The log files are located in /data/howl/log/. There are multiple log files with various severities.

Version 0.7.x
FiFo uses extensive logging to make debugging issue and understanding behavior. The log files are located in /var/log/howl/. There are multiple log files with various severities.

debug.log
By default the debug log is disabled. It is very verbose and should not be enabled in production systems. To enable it uncomment the following line in the howl.conf

log.debug.file = /var/log/howl/debug.log

console.log
This file contains logs of the level info and above. Usually all interesting logs can be found here.

error.log
This files contains errors. It usually should be mostly empty but please keep in mind that failing is not a uncommon practice to deal with unexpected behavior so sporadic entries might just be fine.