monit is doing a bad job at stopping processes

[Former user]

I have a HA environment with 2 identical servers in Master/Slave configuration. When Master goes down, the Slave becomes master and takes over.

When a master becomes a slave we issue a "monit -g master --stop" command to stop all processes that should not run on a slave.

Monit is not reliable in terms of turning the services off and making sure that they are really killed.

[MSK Feb 20 17:08:33] info     : 'dhcp-server' stop on user request
[MSK Feb 20 17:08:33] info     : 'knockd' stop on user request
[MSK Feb 20 17:08:33] info     : 'zabbix-agent-main' stop on user request
[MSK Feb 20 17:08:33] info     : 'zabbix-server' stop on user request
[MSK Feb 20 17:08:33] info     : Monit daemon with PID 11131 awakened
[MSK Feb 20 17:08:33] info     : Awakened by User defined signal 1
[MSK Feb 20 17:08:33] info     : 'zabbix-server' stop action done
[MSK Feb 20 17:08:33] info     : 'zabbix-agent-main' stop: '/bin/bash -c pkill -f zabbix_agentd_main'
[MSK Feb 20 17:08:33] info     : 'zabbix-agent-main' stop action done
[MSK Feb 20 17:08:33] info     : 'knockd' stop: '/bin/bash -c /usr/bin/pkill -f knockd'
[MSK Feb 20 17:08:34] info     : 'knockd' stop action done
[MSK Feb 20 17:08:34] info     : 'dhcp-server' stop: '/bin/bash -c /etc/init.d/dhcpd stop'
[MSK Feb 20 17:08:34] info     : 'dhcp-server' stop action done

Everything was stopped except "zabbix-server". It kept running.

I'm on monit 5.20.0

[Tildeslash]

It's a configuration issue - not bug. The stop method allows one to set custom script to stop the service.

If it's important to kill the process with SIGKILL if graceful stop failed, the kill command should be either part of the stop method (for example add "pkill -9 <process>" to the end of the script) or you can combine them like this:

stop method = "/bin/bash -c '/etc/init.d/myprocess stop; pkill -9 myprocess'"

It may be probably useful to add new stop method option, which will instruct monit to kill the process if graceful stop failed, but that is feature request - not bug.

[Tildeslash]

created a feature request for the same: https://bitbucket.org/tildeslash/monit/issues/554/stop-method-sigkill-option