Troubleshooting
Problem Checklist
We often encounter users reporting similar kinds of problems. Below is a checklist to help you track them down more easily.
First some general information you’ll need:
| Command | |
|---|---|
| Hypervisor IP | ifconfig |
| Zone IP | ifconfig |
| Chunter Version | cat /opt/chunter/etc/chunter.version |
| Sniffle Version | pkgin list | grep sniffle |
| Howl Version | pkgin list | grep howl |
| Snarl Version | pkgin list | grep snarl |
All troubleshooting items below will be marked with a Legend/abbreviation which tells you in which environment or zone you should run the troubleshooting commands from:
| Abbreviation | Run location |
|---|---|
| GZ | Global Zone (Hypervisor / Server itself) |
| FZ | FiFo Zone |
| LZ | LeoFS Zone |
| DZ | DalmatinerDB Zone |
| TZ | Tachyon Zone |
| CC | Client Command (any client) |
Zone Abbreviations
In certain circumstances you may be running multiple services in the same zone - if that is the case the abbreviation simply means run this command in the Zone where the service/s are running.
User Interface - problems
If you have UI related problems please first attempt to clear your web browser cookies. This resolves a lot of UI problems.
General Checks
(GZ) Is your FiFo Zone running?
$ vmadm list | grep fifo
cae70926-cdc8-430f-9138-e06ac596c3ad OS 512 running fifo
(FZ) Are all your processes running?
$ ps -afe | grep run_erl | grep -v grep
0000104 99374 1 0 08:39:13 ? 0:00 /opt/local/sniffle/erts-5.9.1/bin/run_erl -daemon /tmp/sniffle/ /var/log/sniffl
0000103 42238 1 0 06:55:02 ? 0:00 /opt/local/snarl/erts-5.9.1/bin/run_erl -daemon /tmp/snarl/ /var/log/snarl exec
0000106 56504 1 0 07:20:50 ? 0:00 /opt/local/howl/erts-5.9.1/bin/run_erl -daemon /tmp/howl /var/log/howl exec /op
(FZ) Services are going to maintenance
If you are using the release build make sure your CPU supports AVX, consult the manufacturers spec sheet for this.
To get additional information it is also possible to run the service manually using <service> console for example sniffle console and look at the output that will print out possible configuration errors and other debug information to the shell.
(GZ) Are all your processes running?
$ ps -afe | grep run_erl | grep -v grep
root 48037 1 0 Dec 27 ? 0:00 /opt/chunter/erts-5.9.1/bin/run_erl -daemon /tmp/chunter /var/log/chunter exec
root 48033 1 0 Dec 27 ? 0:00 /opt/chunter/erts-5.9.1/bin/run_erl -daemon /tmp/fifo_zlogin /var/log/fifo_zlogin exec
(GZ) Is your Chunter services running?
$ svcs chunter fifo/zlogin
STATE STIME FMRI
online Jan_27 svc:/network/chunter:default
online Jan_27 svc:/fifo/zlogin:default
(GZ) Is there communication between your Global Zone and FiFo?
$ ping $FIFOZONEIP
192.168.0.126 is alive
(GZ) Does DNS reach your hypervisor?
might take ~2 minutes
$ snoop inet 224.0.0.251
172.16.234.10 -> 224.0.0.251 MDNS R _snarl._tcp.local. Internet PTR build._snarl._tcp.local.
172.16.234.10 -> 224.0.0.251 MDNS R _howl._tcp.local. Internet PTR build._howl._tcp.local.
172.16.234.10 -> 224.0.0.251 MDNS R _sniffle._tcp.local. Internet PTR build._sniffle._tcp.local.
(GZ) Is Chunter listening on the correct IP Address?
$ grep ip /opt/chunter/etc/chunter.conf
ip = <ip in the same network as the fifo zone>:4200
should be the ip address of your hypervisor within the same network as your fifo zone
(FZ) Does your FiFo zone have network connectivity?
$ ping project-fifo.net
project-fifo.net is alive
(FZ) Are all your FiFo Services running?
$ svcs sniffle snarl howl
STATE STIME FMRI
online Jan_25 svc:/network/snarl:default
online Jan_25 svc:/network/howl:default
online Jan_25 svc:/network/sniffle:default
(GZ) Is your Memory scaled correctly?
$ prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
27833 howl 109M 54M sleep 56 0 2:06:24 0.3% beam.smp/164
27916 wiggle 91M 54M sleep 54 0 1:38:29 0.2% beam.smp/77
27846 snarl 358M 61M sleep 57 0 0:29:30 0.1% beam.smp/172
27841 sniffle 1112M 771M sleep 56 0 1:39:02 0.1% beam.smp/184
Note that SIZE is much bigger then RSS, this is caused by mmaped files for the database and can cause problems if it grows too big!
(CC) Is the API working, can you manually login via curl?
curl -v "http://<IP>/api/0.1.0/sessions" -H "Content-Type: application/json" -H "Accept: application/json" --data-binary '{"user":"admin","password":"admin"}'
* About to connect() to 192.168.0.204 port 80 (#0)
* Trying 192.168.0.204...
* connected
* Connected to 192.168.0.204 (192.168.0.204) port 80 (#0)
> POST /api/0.1.0/sessions HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: 192.168.0.204
> Content-Type: application/json
> Accept: application/json
> Content-Length: 33
>
* upload completely sent off: 33 out of 33 bytes
< HTTP/1.1 303 See Other
< Server: nginx/1.3.13
< Date: Wed, 29 May 2013 10:37:59 GMT
< Content-Type: application/json
< Content-Length: 0
< Location: http://<IP>/api/0.1.0/sessions/64bbb7cb-7505-4b01-adf7-c7daf5b5186a
< Connection: keep-alive
< access-control-allow-origin: *
< access-control-allow-headers: content-type, x-snarl-token
< access-control-expose-headers: x-snarl-token
< allow-access-control-credentials: true
< vary: accept
< set-cookie: x-snarl-token=64bbb7cb-7505-4b01-adf7-c7daf5b5186a; Version=1; Expires=Wed, 28-May-2014 10:37:59 GMT; Max-Age=31449600
< x-snarl-token: 64bbb7cb-7505-4b01-adf7-c7daf5b5186a
<
* Connection #0 to host 192.168.0.204 left intact
* Closing connection #0
(FZ) Can Howl connect to Sniffle and Snarl?
$ howl-admin connections
Snarl endpoints.
Hostname Node Errors
------------------------------ -------------------- ---------------
fifo01.local 192.168.1.41:4200 0
Snarl endpoints.
Hostname Node Errors
------------------------------ -------------------- ---------------
fifo01.local 192.168.1.41:4210 0
(FZ) Where do I find log files for a particular service?
$ ls -Fd /data/*/log
/data/howl/log/ /data/snarl/log/ /data/sniffle/log/
$
- Note: You may see mentions of crashes in the log files - this can happen even on a healthy system that is running fine. Once you found a logfile you want to examine, cat or tail it accordingly*
Hypervisor & VM Checks
(FZ) Can you see your hypervisors being listed?
$ fifoadm hypervisors list
Hypervisor IP Memory State
------------------ ---------------- --------------- -------------
00-15-17-b8-16-fc 172.16.0.4 25064/32699 ok
(FZ) Can you see your machines being listed?
$ fifoadm vms list
List of VMs zone fifoadm vms list
UUID Hypervisor Name State
------------------------------------ ----------------- --------------- ----------
7df22c41-bade-4b26-b20e-ee2b45e81bf8 00-15-17-b8-16-fc fifo running
21e0bc5d-af4e-4a44-8137-c7d50870dcbd 00-15-17-b8-16-fc ngnix running
bf57045f-42ee-42b5-8dc5-201250b7b6f4 00-15-17-b8-16-fc confluence running
39cd0a98-5087-472c-b89c-e75aef378a22 00-15-17-b8-16-fc dev stopped
49314fda-fef0-42fa-b974-77d27b097aa1 00-15-17-b8-16-fc korny running
2362ebf6-4988-4cfd-89ec-004dcc61a63b 00-15-17-b8-16-fc zotonic stopped
87cc64b1-3990-4cf6-a54d-dbc2e66adddc 00-15-17-b8-16-fc - installing
1df09840-f2bb-48fb-a3b3-5fe679849baf 00-15-17-b8-16-fc mail running
6d4a35a6-41d8-4a44-9977-e010b3ed307a 00-15-17-b8-16-fc test running
(FZ) How do I fetch information for a misbehaving VM?
$ fifoadm vms get -j <uuid>
{
"hypervisor": "00-15-17-b8-16-fc",
"state": "installing_dataset"
}
Additional fifoadm commands
There are a lot more calls for fifoadm that can help depending on where things lead.
Other errors
DNS
When using a custom DNS server or host files instead of xip.io many people tend to get the configuration of their systems wrong this problems are easily detectable by looking for nxdomain errors in the log files. i.e. grep nxdomain /var/log/chunter/error.log.
To verify the DNS configuration the tool dig can be used in both the fifo zone and the global zone.
$ dig 10.0.0.100.xip.io
; <<>> DiG 9.8.3-P1 <<>> 10.0.0.100.xip.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47415
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;10.0.0.100.xip.io. IN A
;; ANSWER SECTION:
10.0.0.100.xip.io. 300 IN A 10.0.0.100
;; Query time: 279 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Thu Jul 7 01:44:49 2016
;; MSG SIZE rcvd: 51
$ dig all-subdomains.10.0.0.100.xip.io
; <<>> DiG 9.8.3-P1 <<>> all-subdomains.10.0.0.100.xip.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32813
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;all-subdomains.10.0.0.100.xip.io. IN A
;; ANSWER SECTION:
all-subdomains.10.0.0.100.xip.io. 300 IN A 10.0.0.100
;; Query time: 147 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Thu Jul 7 01:45:20 2016
;; MSG SIZE rcvd: 66
To clarify, all-subdomains in this example means that whatever valid dns hostname is put there needs to resolve to the same IP.
Reporting an issue
JIRA is the best place to file a report. If you do so it is often helpful to attach some logs. They can be found in the /data/{sniffle,snarl,howl}/logs and /var/log/chunter (in the FiFo Zone or GZ respectively).
In addition there is a built in command fifoadm diag that will prepare all your log files and put them in a /var/tmp/[$fifoservice]-diag directory. This is to aid you in your log collection and submission process and to encourage folks to always attach logs when filing bug reports.
In the global zone run /opt/chunter/share/chunter-diag to grab diagnostics data.
Updated less than a minute ago