The problem with checking Unix-socket - Monit

Hello. In service a Monit we was detect the problem. Between version 5.8 and 5.8.1 in this commit ( https://bitbucket.org/tildeslash/monit/commits/5aa93749973e632371cb5632c6c435f246331dcc ) when Monit checks the uptime of the process before perform the test socket availability. The source code in src/validate.c it looks like this:

 if (s->portlist)
                 /* skip further tests during startup timeout */
                 if (s->start)
                         if (s->inf->priv.process.uptime < s->start->timeout) return TRUE;
                 for (pp = s->portlist; pp; pp = pp->next)
                         check_connection(s, pp);

The logic of this test is that a timeout has not yet occurred in the full launch of the process and check sockets early. At the same time, for unknown reasons, within the LXC container (presumably only 2.x) monit can not get the uptime of the process. The output monit status, it looks like this:

root@debian-8:~/monit-5.9/src# monit status
The Monit daemon 5.9 uptime: 20m 

Process 'dockerd'
 status                            Running
 monitoring status                 Monitored
 pid                               224
 parent pid                        1
 uid                               0
 effective uid                     0
 gid                               0
 uptime                               <<< No uptime info
 children                          1
 memory kilobytes                  15.9 MB
 memory kilobytes total            20.7 MB
 memory percent                    1.5%
 memory percent total              2.0%
 cpu percent                       0.0%
 cpu percent total                 0.0%
 unix socket response time         0.000s to /var/run/docker.sock [HTTP]
 data collected                    Mon, 27 Feb 2017 22:43:07

We used a test configuration:

check process dockerd with pidfile /var/run/docker.pid
start program = "/bin/systemctl start docker" 
stop program = "/bin/systemctl stop docker" 
if failed unixsocket /var/run/docker.sock protocol HTTP request "/version" then alert

And with «monit -c / etc / monit / monitrc -vv -I» can be seen as a monit parse the config:

Process Name          = dockerd
Pid file             = /var/run/docker.pid
Monitoring mode      = active
Start program        = '/bin/systemctl start docker' timeout 30 second(s)
Stop program         = '/bin/systemctl stop docker' timeout 30 second(s)
Existence            = if does not exist then restart
Pid                  = if changed then alert
PPid                 = if changed then alert
Unix Socket          = if failed [/var/run/docker.sock [protocol HTTP] with timeout 5000 seconds] then restart

That is, by default monit waits 30 seconds per process run. But because uptime it does not parse, then 30 seconds is always more and call check_connection () is never executed.

The problem found in the following systems:

This issue is not detect to Ubuntu 14.04 with monit 5.6 and Debian 7 monit 5.4. At the same CentOS 6 has a monit 5.14 and then the bug is already too. brute force managed to find that 5.18 monit already able to get the uptime (in this case 5.17 are not going), but we discovered another problem. Here is:

if failed unixsocket /var/run/docker.sock protocol HTTP request "/version" then alert

In version 5.18 performs a HEAD request (instead of the usual GET), and API docker this demon does not know (but understands GET / version) and provides 404.

Comments (4)