I'm facing a similar issue, wherein I'm running my services in a docker container which has monit as pid 1 and is monitoring mongodb, vault and nginx. When I run my docker container for the first time, everything comes up properly , but when I do a docker stop "my container" followed by docker start "mycontainer", due to a residual vault pid file from the old startup, monit behaved weirdly and showed the status of vault as "Running" even though the process didn't exist. This is the Monit status for vault
Process 'vault' status Running monitoring status Monitored pid - parent pid - uid - effective uid - gid - uptime - threads - children - memory - memory total - memory percent - memory percent total - cpu percent - cpu percent total - data collected Mon, 26 Sep 2016 01:06:29
This is a snippet from monit log file. Seems like it didn't do the test check for vault at all whether pid in the vault.pid file is same as the process running.
[PDT Sep 26 01:15:12] debug : 'mongodb' process test failed [pid=147] -- No such process [PDT Sep 26 01:15:12] info : 'mongodb' start: /bin/bash [PDT Sep 26 01:15:12] debug : 'mongodb' started [PDT Sep 26 01:15:12] info : 'mongodb' process is running with pid 15 [PDT Sep 26 01:15:12] debug : 'mongodb' zombie check succeeded [PDT Sep 26 01:15:19] debug : 'nginx' process test failed [pid=252] -- No such process [PDT Sep 26 01:15:19] info : 'nginx' start: /sbin/start-stop-daemon [PDT Sep 26 01:15:19] debug : 'nginx' started [PDT Sep 26 01:15:40] debug : 'vault' process is running with pid 225 [PDT Sep 26 01:15:40] debug : 'vault' zombie check succeeded
And this issue, sometimes occurs with vault or sometimes with some other process. But I'm unable to resolve the issue.