monit crashes

Issue #255 resolved
Anonymous created an issue

H. I use monit for my ruby app and sidekiq

Ubuntu 14.04 x64 Monit version 5.14

monit config:

set daemon 30 with start delay 30 
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set mailserver x.x.x.x
set eventqueue
  basedir /var/lib/monit/events
    slots 100                     
set mail-format {
Service:        $SERVICE
Date:           $DATE
Action:         $ACTION
Host:           $HOST
Description:    $DESCRIPTION

Your faithful employee,
set alert with reminder on 1 cycle
set httpd port 2812 and
  use address localhost
  allow localhost

include /etc/monit/conf.d/*

included file is:

check system master
  if memory is greater than 70% then exec "/usr/bin/pkill --signal 9 -f ruby;/etc/init.d/nginx restart; /bin/su -l user -c 'cd core/current && bundle exec sidekiqctl stop /home/user/app/current/tmp/pids/'"

Sometime, when ruby processes consume memory too fast, monit crashes. In log file I can see only:

[MSK Sep 25 16:59:52] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:00:22] error    : 'master' mem usage of 88.6% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:00:22] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:00:54] error    : 'master' mem usage of 92.4% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:00:55] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:01:30] error    : 'master' mem usage of 92.9% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:01:32] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:02:24] error    : 'master' mem usage of 93.4% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:02:49] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:02:58] error    : Cannot fork a new process -- Cannot allocate memory

Then I need to restart monit process. I can understant that monit can't execute "stop" command due to lack of memory. But why it crashes

Comments (10)

  1. Tildeslash repo owner

    Yet one thing ... the exec action is executed directly by execv(), if you need to use script, you have to wrap it using shell for example like this:

    ... then exec "/bin/bash -c '/usr/bin/pkill --signal 9 -f ruby;/etc/init.d/nginx restart;'"

    as your script is little bit more complicated (sidekiq part uses also apostrophe), so i'd recommend to save the script into some file and call that script from the exec action.

  2. Log in to comment