Monit and custom script (rq-dashboard), zombie process created

[Muhamed Huseinbašić]

My idea is to have monit control rq-dashboard execution. I have this script:

#!/bin/bash
PID_DIR=/opt/custom_user/run

case $1 in
   start)
      shift
      PROCESSNAME=$1
      echo $$ > $PID_DIR/$1.pid;
      shift
      username=$1
      shift
      sudo -u $username $* 2>&1
      ;;
    stop)
       pkill -TERM -P `cat $PID_DIR/$2.pid` ;;
    *)
      echo "usage: startstop.sh {start|stop} <process_name> <username> <process_command>" ;;
esac
exit 0

My monit-related lines are:

check process rq-dashboard with pidfile /opt/custom_user/run/rq-dashboard.pid
    start program = "/opt/custom_user/scripts/startup/startstop-rq-dashboard.sh start rq-dashboard custom_user python /usr/local/bin/rq-dashboard -p 8080"
    stop program = "/opt/custom_user/scripts/startup/startstop-rq-dashboard.sh stop rq-dashboard"

This was working perfectly with Monit 5.3.2 (on Ubuntu 12.04). Now, I have set up everything in the same manner with Monit 5.16 (on Ubuntu 16.04) and immediately after I start Monit I end up with this:

custom_user@monit:~$ ps aux | grep rq-da
root            16962  0.0  0.0      0     0 ?        Zs   11:21   0:00 [startstop-rq-da] <defunct>
root            16963  0.0  0.1  54824  3880 ?        S    11:21   0:00 sudo -u custom_user python /usr/local/bin/rq-dashboard -p 8080
custom_user     16964  0.0  1.0  62620 22476 ?        S    11:21   0:00 python /usr/local/bin/rq-dashboard -p 8080
custom_user     17983  0.0  0.0  14228   972 pts/1    S+   11:27   0:00 grep --color=auto rq-da

So basically immediately after I start everything, zombie process is created and never cleaned (with rq-dashboard running normally, but monit not recognizing it and thus making me unable to stop it).

Please, advise. What has changed? What should I change in this scenario of custom script?

[Tildeslash]

Hello,

it is design limitation of the current test scheduler vs. asynchronous program status, documented in Monit manual (https://mmonit.com/monit/documentation/monit.html#PROGRAM-STATUS-TEST):

The asynchronous nature of the program check allows for non-blocking behaviour in the current Monit
design, but it comes with a side-effect: when the program has finished executing and is waiting for
Monit to collect the result, it becomes a so-called "zombie" process. A zombie process does not
consume any system resources (only the PID remains in use) and it is under Monit's control and the
zombie process is removed from the system as soon as Monit collects the exit status. This means that
every "check program" will be associated with either a running process or a temporary zombie. This
unwanted zombie side-effect will be removed in a later release of Monit.

[Tildeslash]

Duplicate of #242.