---
agentTools:
  projectIndex: https://docs.project-fifo.net/llms.txt
---

# Troubleshooting

Problem Checklist

We often encounter users reporting similar kinds of problems. Below is a checklist to help you track them down more easily.

First some general information you’ll need:

[block:parameters]
{
  "data": {
    "h-1": "Command",
    "0-0": "Hypervisor IP",
    "1-0": "Zone IP",
    "2-0": "Chunter Version",
    "3-0": "Sniffle Version",
    "4-0": "Howl Version",
    "5-0": "Snarl Version",
    "0-1": "`ifconfig`",
    "1-1": "`ifconfig`",
    "2-1": "`cat /opt/chunter/etc/chunter.version`",
    "3-1": "`pkgin list | grep sniffle`",
    "4-1": "`pkgin list | grep howl`",
    "5-1": "`pkgin list | grep snarl`"
  },
  "cols": 2,
  "rows": 6
}
[/block]

All troubleshooting items below will be marked with a **Legend/abbreviation** which tells you in which environment or zone you should run the troubleshooting commands from:

[block:parameters]
{
  "data": {
    "0-0": "**GZ**",
    "1-0": "**FZ**",
    "2-0": "**LZ**",
    "3-0": "**DZ**",
    "0-1": "Global Zone (Hypervisor / Server itself)",
    "1-1": "FiFo Zone",
    "2-1": "LeoFS Zone",
    "3-1": "DalmatinerDB Zone",
    "4-0": "**TZ**",
    "4-1": "Tachyon Zone",
    "h-1": "Run location",
    "h-0": "Abbreviation",
    "5-0": "**CC**",
    "5-1": "Client Command (any client)"
  },
  "cols": 2,
  "rows": 6
}
[/block]

[block:callout]
{
  "type": "info",
  "title": "Zone Abbreviations",
  "body": "In certain circumstances you may be running multiple services in the same zone - if that is the case the abbreviation simply means run this command in the Zone where the service/s are running."
}
[/block]

[block:callout]
{
  "type": "warning",
  "title": "User Interface - problems",
  "body": "If you have UI related problems please **first** attempt to clear your web browser cookies. This resolves a lot of UI problems."
}
[/block]

[block:api-header]
{
  "type": "basic",
  "title": "General Checks"
}
[/block]

(GZ) **Is your FiFo Zone running?**

[block:code]
{
  "codes": [
    {
      "code": "$ vmadm list | grep fifo\n\ncae70926-cdc8-430f-9138-e06ac596c3ad  OS    512      running           fifo\n",
      "language": "shell"
    }
  ]
}
[/block]

(FZ) **Are all your processes running?**

[block:code]
{
  "codes": [
    {
      "code": "$ ps -afe | grep run_erl | grep -v grep\n\n0000104 99374 1 0 08:39:13 ? 0:00 /opt/local/sniffle/erts-5.9.1/bin/run_erl -daemon /tmp/sniffle/ /var/log/sniffl\n0000103 42238 1 0 06:55:02 ? 0:00 /opt/local/snarl/erts-5.9.1/bin/run_erl -daemon /tmp/snarl/ /var/log/snarl exec\n0000106 56504 1 0 07:20:50 ? 0:00 /opt/local/howl/erts-5.9.1/bin/run_erl -daemon /tmp/howl /var/log/howl exec /op\n",
      "language": "shell"
    }
  ]
}
[/block]

(FZ) **Services are going to maintenance**
If you are using the release build make sure your CPU supports **AVX**, consult the manufacturers spec sheet for this.

To get additional information it is also possible to run the service manually using `<service> console` for example `sniffle console` and look at the output that will print out possible configuration errors and other debug information to the shell.

(GZ) **Are all your processes running?**

[block:code]
{
  "codes": [
    {
      "code": "$ ps -afe | grep run_erl | grep -v grep\n\nroot 48037 1 0 Dec 27 ? 0:00 /opt/chunter/erts-5.9.1/bin/run_erl -daemon /tmp/chunter /var/log/chunter exec\nroot 48033 1 0 Dec 27 ? 0:00 /opt/chunter/erts-5.9.1/bin/run_erl -daemon /tmp/fifo_zlogin /var/log/fifo_zlogin exec\n",
      "language": "shell"
    }
  ]
}
[/block]

(GZ) **Is your Chunter services running?**

[block:code]
{
  "codes": [
    {
      "code": "$ svcs chunter fifo/zlogin\n\nSTATE  STIME  FMRI\nonline Jan_27 svc:/network/chunter:default\nonline Jan_27 svc:/fifo/zlogin:default\n",
      "language": "shell"
    }
  ]
}
[/block]

(GZ) **Is there communication between your Global Zone and FiFo?**

[block:code]
{
  "codes": [
    {
      "code": "$ ping $FIFOZONEIP\n\n192.168.0.126 is alive",
      "language": "shell"
    }
  ]
}
[/block]

(GZ) **Does DNS reach your hypervisor?**

*might take \~2 minutes*

[block:code]
{
  "codes": [
    {
      "code": "$ snoop inet 224.0.0.251\n\n172.16.234.10 -> 224.0.0.251 MDNS R _snarl._tcp.local. Internet PTR build._snarl._tcp.local.\n172.16.234.10 -> 224.0.0.251 MDNS R _howl._tcp.local. Internet PTR build._howl._tcp.local.\n172.16.234.10 -> 224.0.0.251 MDNS R _sniffle._tcp.local. Internet PTR build._sniffle._tcp.local.\n",
      "language": "shell"
    }
  ]
}
[/block]

(GZ) **Is Chunter listening on the correct IP Address?**

[block:code]
{
  "codes": [
    {
      "code": "$ grep ip /opt/chunter/etc/chunter.conf\n\nip = <ip in the same network as the fifo zone>:4200\n",
      "language": "shell"
    }
  ]
}
[/block]

*should be the ip address of your hypervisor within the same network as your fifo zone*

(FZ) **Does your FiFo zone have network connectivity?**

[block:code]
{
  "codes": [
    {
      "code": "$ ping project-fifo.net\n\nproject-fifo.net is alive",
      "language": "text"
    }
  ]
}
[/block]

(FZ) **Are all your FiFo Services running?**

[block:code]
{
  "codes": [
    {
      "code": "$ svcs sniffle snarl howl\n\nSTATE  STIME  FMRI\nonline Jan_25 svc:/network/snarl:default\nonline Jan_25 svc:/network/howl:default\nonline Jan_25 svc:/network/sniffle:default\n",
      "language": "shell"
    }
  ]
}
[/block]

(GZ) **Is your Memory scaled correctly?**

[block:code]
{
  "codes": [
    {
      "code": "$ prstat\n\n PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP\n 27833 howl      109M   54M sleep   56    0   2:06:24 0.3% beam.smp/164\n 27916 wiggle     91M   54M sleep   54    0   1:38:29 0.2% beam.smp/77\n 27846 snarl     358M   61M sleep   57    0   0:29:30 0.1% beam.smp/172\n 27841 sniffle  1112M  771M sleep   56    0   1:39:02 0.1% beam.smp/184\n",
      "language": "shell"
    }
  ]
}
[/block]

*Note that SIZE is much bigger then RSS, this is caused by mmaped files for the database and can cause problems if it grows too big!*

(CC) **Is the API working, can you manually login via curl?**

[block:code]
{
  "codes": [
    {
      "code": "curl -v \"http://<IP>/api/0.1.0/sessions\" -H \"Content-Type: application/json\" -H \"Accept: application/json\" --data-binary '{\"user\":\"admin\",\"password\":\"admin\"}'\n\n* About to connect() to 192.168.0.204 port 80 (#0)\n* Trying 192.168.0.204...\n* connected\n* Connected to 192.168.0.204 (192.168.0.204) port 80 (#0)\n> POST /api/0.1.0/sessions HTTP/1.1\n> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5\n> Host: 192.168.0.204\n> Content-Type: application/json\n> Accept: application/json\n> Content-Length: 33\n>\n* upload completely sent off: 33 out of 33 bytes\n< HTTP/1.1 303 See Other\n< Server: nginx/1.3.13\n< Date: Wed, 29 May 2013 10:37:59 GMT\n< Content-Type: application/json\n< Content-Length: 0\n< Location: http://<IP>/api/0.1.0/sessions/64bbb7cb-7505-4b01-adf7-c7daf5b5186a\n< Connection: keep-alive\n< access-control-allow-origin: *\n< access-control-allow-headers: content-type, x-snarl-token\n< access-control-expose-headers: x-snarl-token\n< allow-access-control-credentials: true\n< vary: accept\n< set-cookie: x-snarl-token=64bbb7cb-7505-4b01-adf7-c7daf5b5186a; Version=1; Expires=Wed, 28-May-2014 10:37:59 GMT; Max-Age=31449600\n< x-snarl-token: 64bbb7cb-7505-4b01-adf7-c7daf5b5186a\n<\n* Connection #0 to host 192.168.0.204 left intact\n* Closing connection #0",
      "language": "curl"
    }
  ]
}
[/block]

(FZ) **Can Howl connect to Sniffle and Snarl?**

[block:code]
{
  "codes": [
    {
      "code": "$ howl-admin connections\nSnarl endpoints.\nHostname                                 Node                Errors\n------------------------------ -------------------- ---------------\n                  fifo01.local 192.168.1.41:4200                0\nSnarl endpoints.\nHostname                                 Node                Errors\n------------------------------ -------------------- ---------------\n                  fifo01.local 192.168.1.41:4210                0",
      "language": "shell"
    }
  ]
}
[/block]

(FZ) **Where do I find log files for a particular service?**

[block:code]
{
  "codes": [
    {
      "code": "$ ls -Fd /data/*/log\n/data/howl/log/  /data/snarl/log/  /data/sniffle/log/\n$",
      "language": "shell"
    }
  ]
}
[/block]

* Note: You may see mentions of crashes in the log files - this can happen even on a healthy system that is running fine. Once you found a logfile you want to examine, cat or tail it accordingly\*

[block:api-header]
{
  "type": "basic",
  "title": "Hypervisor & VM Checks"
}
[/block]

(FZ) **Can you see your hypervisors being listed?**

[block:code]
{
  "codes": [
    {
      "code": "$ fifoadm hypervisors list\n\nHypervisor         IP               Memory          State\n------------------ ---------------- --------------- -------------\n00-15-17-b8-16-fc  172.16.0.4       25064/32699     ok\n",
      "language": "shell"
    }
  ]
}
[/block]

(FZ) **Can you see your machines being listed?**

[block:code]
{
  "codes": [
    {
      "code": "$ fifoadm vms list\n\nList of VMs     zone    fifoadm vms list\nUUID                                 Hypervisor        Name            State\n------------------------------------ ----------------- --------------- ----------\n7df22c41-bade-4b26-b20e-ee2b45e81bf8 00-15-17-b8-16-fc fifo            running\n21e0bc5d-af4e-4a44-8137-c7d50870dcbd 00-15-17-b8-16-fc ngnix           running\nbf57045f-42ee-42b5-8dc5-201250b7b6f4 00-15-17-b8-16-fc confluence      running\n39cd0a98-5087-472c-b89c-e75aef378a22 00-15-17-b8-16-fc dev             stopped\n49314fda-fef0-42fa-b974-77d27b097aa1 00-15-17-b8-16-fc korny           running\n2362ebf6-4988-4cfd-89ec-004dcc61a63b 00-15-17-b8-16-fc zotonic         stopped\n87cc64b1-3990-4cf6-a54d-dbc2e66adddc 00-15-17-b8-16-fc -               installing\n1df09840-f2bb-48fb-a3b3-5fe679849baf 00-15-17-b8-16-fc mail            running\n6d4a35a6-41d8-4a44-9977-e010b3ed307a 00-15-17-b8-16-fc test            running\n",
      "language": "shell"
    }
  ]
}
[/block]

(FZ) **How do I fetch information for a misbehaving VM?**

[block:code]
{
  "codes": [
    {
      "code": "$ fifoadm vms get -j <uuid>\n\n{\n \"hypervisor\": \"00-15-17-b8-16-fc\",\n \"state\": \"installing_dataset\"\n}",
      "language": "shell"
    }
  ]
}
[/block]

[block:callout]
{
  "type": "info",
  "body": "There are a lot more calls for fifoadm that can help depending on where things lead.",
  "title": "Additional fifoadm commands"
}
[/block]

[block:api-header]
{
  "type": "basic",
  "title": "Other errors"
}
[/block]

## DNS

When using a custom DNS server or host files instead of `xip.io` many people tend to get the configuration of their systems wrong this problems are easily detectable by looking for `nxdomain` errors in the log files. i.e. `grep nxdomain /var/log/chunter/error.log`.

To verify the DNS configuration the tool `dig` can be used in both the fifo zone and the global zone.

[block:code]
{
  "codes": [
    {
      "code": "$ dig 10.0.0.100.xip.io\n\n; <<>> DiG 9.8.3-P1 <<>> 10.0.0.100.xip.io\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47415\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;10.0.0.100.xip.io.\t\tIN\tA\n\n;; ANSWER SECTION:\n10.0.0.100.xip.io.\t300\tIN\tA\t10.0.0.100\n\n;; Query time: 279 msec\n;; SERVER: 192.168.1.1#53(192.168.1.1)\n;; WHEN: Thu Jul  7 01:44:49 2016\n;; MSG SIZE  rcvd: 51\n\n$ dig all-subdomains.10.0.0.100.xip.io\n\n; <<>> DiG 9.8.3-P1 <<>> all-subdomains.10.0.0.100.xip.io\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32813\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;all-subdomains.10.0.0.100.xip.io. IN\tA\n\n;; ANSWER SECTION:\nall-subdomains.10.0.0.100.xip.io. 300 IN A\t10.0.0.100\n\n;; Query time: 147 msec\n;; SERVER: 192.168.1.1#53(192.168.1.1)\n;; WHEN: Thu Jul  7 01:45:20 2016\n;; MSG SIZE  rcvd: 66",
      "language": "text"
    }
  ]
}
[/block]

To clarify, `all-subdomains` in this example means that whatever valid dns hostname is put there needs to resolve to the same IP.

[block:api-header]
{
  "type": "basic",
  "title": "Reporting an issue"
}
[/block]

[JIRA](https://project-fifo.atlassian.net/) is the best place to file a report. If you do so it is often helpful to attach some logs. They can be found in the `/data/{sniffle,snarl,howl}/logs` and `/var/log/chunter` (in the FiFo Zone or GZ respectively).

In addition there is a built in command `fifoadm diag` that will prepare all your log files and put them in a `/var/tmp/[$fifoservice]-diag` directory. This is to aid you in your log collection and submission process and to encourage folks to **always** attach logs when filing bug reports.

In the global zone run `/opt/chunter/share/chunter-diag` to grab diagnostics data.