monit crashes

Issue #255 resolved
Former user created an issue

H. I use monit for my ruby app and sidekiq

Ubuntu 14.04 x64 Monit version 5.14

monit config:

set daemon 30 with start delay 30 
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set mailserver x.x.x.x
set eventqueue
  basedir /var/lib/monit/events
    slots 100                     
set mail-format {
  from: notification@example.com
  subject: MONIT: $SERVICE $EVENT
  message:
Service:        $SERVICE
Date:           $DATE
Action:         $ACTION
Host:           $HOST
Description:    $DESCRIPTION

Your faithful employee,
Monit
}
set alert catcher@example.com with reminder on 1 cycle
set httpd port 2812 and
  use address localhost
  allow localhost

include /etc/monit/conf.d/*

included file is:

check system master
  if memory is greater than 70% then exec "/usr/bin/pkill --signal 9 -f ruby;/etc/init.d/nginx restart; /bin/su -l user -c 'cd core/current && bundle exec sidekiqctl stop /home/user/app/current/tmp/pids/sidekiq.pid'"

Sometime, when ruby processes consume memory too fast, monit crashes. In log file I can see only:

[MSK Sep 25 16:59:52] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:00:22] error    : 'master' mem usage of 88.6% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:00:22] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:00:54] error    : 'master' mem usage of 92.4% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:00:55] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:01:30] error    : 'master' mem usage of 92.9% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:01:32] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:02:24] error    : 'master' mem usage of 93.4% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:02:49] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:02:58] error    : Cannot fork a new process -- Cannot allocate memory

Then I need to restart monit process. I can understant that monit can't execute "stop" command due to lack of memory. But why it crashes

Comments (10)

  1. ilia

    Sorry for poor decoration at start. This issue is reproduced every day on my app server. If you need any info, please ask.

  2. Tildeslash repo owner

    Hi,

    problem is fixed, monit will continue if fork() failed, please can you test the patched version?: https://mmonit.com/tmp/monit-5.14-p5.tar.gz

    To compile:

    tar -xzf monit-5.14-p5.tar.gz
    cd monit-5.14-p5
    ./configure
    make
    

    Note that if the system is completely out of memory, monit may still exit during other operation which requires memory allocation.

  3. ilia

    Also, as I understand, when monit exit due to lack of memory - exit code is 0, because upstart doesn't respawn it. Am I right?

  4. Tildeslash repo owner

    Yet one thing ... the exec action is executed directly by execv(), if you need to use script, you have to wrap it using shell for example like this:

    ... then exec "/bin/bash -c '/usr/bin/pkill --signal 9 -f ruby;/etc/init.d/nginx restart;'"
    

    as your script is little bit more complicated (sidekiq part uses also apostrophe), so i'd recommend to save the script into some file and call that script from the exec action.

  5. ilia

    It seems works fine now, I see fork errors during memory usage problems, but main process works. Thank you.

  6. Log in to comment