Troubleshooting

We often encounter users reporting similar kinds of problems. Below is a checklist to help you track them down more easily.

First some general information you’ll need:

All troubleshooting items below will be marked with a Legend/abbreviation which tells you in which environment or zone you should run the troubleshooting commands from:

📘

Zone Abbreviations

In certain circumstances you may be running multiple services in the same zone - if that is the case the abbreviation simply means run this command in the Zone where the service/s are running.

🚧

User Interface - problems

If you have UI related problems please first attempt to clear your web browser cookies. This resolves a lot of UI problems.

General Checks

(GZ) Is your FiFo Zone running?

vmadm list | grep fifo

cae70926-cdc8-430f-9138-e06ac596c3ad  OS    512      running           fifo

(FZ) Are all your processes running?

ps -afe | grep run_erl | grep -v grep

0000105 65606 1 0 10:38:57 ? 0:00 /opt/local/wiggle/erts-5.9.1/bin/run_erl -daemon /tmp/wiggle/ /var/log/wiggle e
0000104 99374 1 0 08:39:13 ? 0:00 /opt/local/sniffle/erts-5.9.1/bin/run_erl -daemon /tmp/sniffle/ /var/log/sniffl
0000103 42238 1 0 06:55:02 ? 0:00 /opt/local/snarl/erts-5.9.1/bin/run_erl -daemon /tmp/snarl/ /var/log/snarl exec
0000106 56504 1 0 07:20:50 ? 0:00 /opt/local/howl/erts-5.9.1/bin/run_erl -daemon /tmp/howl /var/log/howl exec /op

(GZ) Are all your processes running?

ps -afe | grep run_erl | grep -v grep

root 48037 1 0 Dec 27 ? 0:00 /opt/chunter/erts-5.9.1/bin/run_erl -daemon /tmp/chunter /var/log/chunter exec

(GZ) Is your Chunter service running?

svcs chunter

STATE  STIME  FMRI
online Jan_27 svc:/network/chunter:default

(GZ) Is there communication between your Global Zone and FiFo?

ping $FIFOZONEIP

ping 192.168.0.126
192.168.0.126 is alive

(GZ) Does DNS reach your hypervisor?

snoop inet 224.0.0.251

might take ~2 minutes

172.16.234.10 -> 224.0.0.251 MDNS R _snarl._tcp.local. Internet PTR build._snarl._tcp.local.
172.16.234.10 -> 224.0.0.251 MDNS R _howl._tcp.local. Internet PTR build._howl._tcp.local.
172.16.234.10 -> 224.0.0.251 MDNS R _sniffle._tcp.local. Internet PTR build._sniffle._tcp.local.

(GZ) Is Chunter listening on the correct IP Address?

grep ip /opt/chunter/etc/chunter.conf

should be the ip address of your hypervisor within the same network as your fifo zone

ip = <ip in the same network as the fifo zone>:4200

(FZ) Does your FiFo zone have network connectivity?

ping project-fifo.net

project-fifo.net is alive

(FZ) Are all your FiFo Services running?

svcs wiggle sniffle snarl howl

STATE  STIME  FMRI
online Jan_25 svc:/network/snarl:default
online Jan_25 svc:/network/howl:default
online Jan_25 svc:/network/sniffle:default
online Jan_27 svc:/network/wiggle:default

(GZ) Is your Memory scaled correctly?

prstat

Note that SIZE is much bigger then RSS, this is caused by mmaped files for the database and can cause problems if it grows too big!

 PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 27833 howl      109M   54M sleep   56    0   2:06:24 0.3% beam.smp/164
 27916 wiggle     91M   54M sleep   54    0   1:38:29 0.2% beam.smp/77
 27846 snarl     358M   61M sleep   57    0   0:29:30 0.1% beam.smp/172
 27841 sniffle  1112M  771M sleep   56    0   1:39:02 0.1% beam.smp/184

(CC) Is the API working, can you manually login via curl?

curl -v "http://<IP>/api/0.1.0/sessions" -H "Content-Type: application/json" -H "Accept: application/json" --data-binary '{"user":"admin","password":"admin"}'

* About to connect() to 192.168.0.204 port 80 (#0)
* Trying 192.168.0.204...
* connected
* Connected to 192.168.0.204 (192.168.0.204) port 80 (#0)
> POST /api/0.1.0/sessions HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: 192.168.0.204
> Content-Type: application/json
> Accept: application/json
> Content-Length: 33
>
* upload completely sent off: 33 out of 33 bytes
< HTTP/1.1 303 See Other
< Server: nginx/1.3.13
< Date: Wed, 29 May 2013 10:37:59 GMT
< Content-Type: application/json
< Content-Length: 0
< Location: http://<IP>/api/0.1.0/sessions/64bbb7cb-7505-4b01-adf7-c7daf5b5186a
< Connection: keep-alive
< access-control-allow-origin: *
< access-control-allow-headers: content-type, x-snarl-token
< access-control-expose-headers: x-snarl-token
< allow-access-control-credentials: true
< vary: accept
< set-cookie: x-snarl-token=64bbb7cb-7505-4b01-adf7-c7daf5b5186a; Version=1; Expires=Wed, 28-May-2014 10:37:59 GMT; Max-Age=31449600
< x-snarl-token: 64bbb7cb-7505-4b01-adf7-c7daf5b5186a
<
* Connection #0 to host 192.168.0.204 left intact
* Closing connection #0

(FZ) Can Howl connect to Sniffle?
Note: libsniffle:servers(). and libsniffle:version(). need to be entered after attaching to wiggle

/opt/local/sbin/howl attach

 Attaching to /tmp/howl/erlang.pipe.1 (^D to exit)
^R
libsniffle:servers().
[{{"fifo.local",[{port,<<"4210">>},{ip,<<"192.168.0.123">>}]},"192.168.0.123",4210}]
(wiggle@192.168.0.123)2> libsniffle:version().
<<"test-19442d0, Sun Dec 30 08:49:03 2012 +0100">>
(wiggle@192.168.0.123)3> ^D [Quit]

(FZ) Can Howl connect to Snarl?
Note: libsnarl:servers(). and libsnarl:version(). need to be entered after attaching to wiggle

/opt/local/sbin/howl attach

Attaching to /tmp/howl/erlang.pipe.1 (^D to exit)
^R
libsnarl:servers().
[{{"build.local",[{port,<<"4200">>},{ip,<<"192.168.0.123">>}]},"192.168.0.123",4200}]
(wiggle@192.168.0.123)2> libsnarl:version().
<<"test-26da855, Sun Dec 30 07:34:20 2012 +0100">>
(wiggle@192.168.0.123)3> ^D [Quit]

(FZ) Where do I find log files for a particular service?

ls -l /var/logs/howl

• Note: You may see mentions of crashes in the log files - this can happen even on a healthy system that is running fine. Once you found a logfile you want to examine, cat or tail it accordingly*

-rw-r--r-- 1 howl howl   3225 Jan 27 22:38 console.log
-rw-r--r-- 1 howl howl         0 Jan 27 22:38 crash.log.1
-rw-r--r-- 1 howl howl 105462 Jan 27 22:40 debug.log
-rw-r--r-- 1 howl howl     84890 Jan 29 18:13 erlang.log.1
-rw-r--r-- 1 howl howl   1874 Jan 25 18:42 error.log
-rw-r--r-- 1 howl howl    270 Jan 27 22:38 run_erl.log
drwxr-xr-x 2 howl howl      7 Jan 27 22:38 sasl

Hypervisor & VM Checks

(FZ) Can you see your hypervisors being listed?

fifoadm hypervisors list

Hypervisor         IP               Memory          State
------------------ ---------------- --------------- -------------
00-15-17-b8-16-fc  172.16.0.4       25064/32699     ok

(FZ) Can you see your machines being listed?

fifoadm vms list

List of VMs     zone    fifoadm vms list
UUID                                 Hypervisor        Name            State
------------------------------------ ----------------- --------------- ----------
7df22c41-bade-4b26-b20e-ee2b45e81bf8 00-15-17-b8-16-fc fifo            running
21e0bc5d-af4e-4a44-8137-c7d50870dcbd 00-15-17-b8-16-fc ngnix           running
bf57045f-42ee-42b5-8dc5-201250b7b6f4 00-15-17-b8-16-fc confluence      running
39cd0a98-5087-472c-b89c-e75aef378a22 00-15-17-b8-16-fc dev             stopped
49314fda-fef0-42fa-b974-77d27b097aa1 00-15-17-b8-16-fc korny           running
2362ebf6-4988-4cfd-89ec-004dcc61a63b 00-15-17-b8-16-fc zotonic         stopped
87cc64b1-3990-4cf6-a54d-dbc2e66adddc 00-15-17-b8-16-fc -               installing
1df09840-f2bb-48fb-a3b3-5fe679849baf 00-15-17-b8-16-fc mail            running
6d4a35a6-41d8-4a44-9977-e010b3ed307a 00-15-17-b8-16-fc test            running

(FZ) How do I fetch information for a misbehaving VM?

fifoadm vms get -j <uuid>

{
 "hypervisor": "00-15-17-b8-16-fc",
 "state": "installing_dataset"
}

📘

Additional fifoadm commands

There are a lot more calls for fifoadm that can help depending on where things lead.

Reporting an issue

JIRA is the best place to file a report. If you do so it is often helpful to attach some logs. They can be found in the /var/logs/{sniffle,snarl,howl,wiggle} and /var/log/chunter (in the FiFo Zone or GZ respectively).

In addition there is a built in command fifoadm diag that will prepare all your log files and put them in a /var/tmp/[$fifoservice]-diag directory. This is to aid you in your log collection and submission process and to encourage folks to always attach logs when filing bug reports.