monit crashes

[Former user]

H. I use monit for my ruby app and sidekiq

Ubuntu 14.04 x64 Monit version 5.14

monit config:

set daemon 30 with start delay 30 
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set mailserver x.x.x.x
set eventqueue
  basedir /var/lib/monit/events
    slots 100                     
set mail-format {
  from: [e-mail]
  subject: MONIT: $SERVICE $EVENT
  message:
Service:        $SERVICE
Date:           $DATE
Action:         $ACTION
Host:           $HOST
Description:    $DESCRIPTION

Your faithful employee,
Monit
}
set alert [e-mail] with reminder on 1 cycle
set httpd port 2812 and
  use address localhost
  allow localhost

include /etc/monit/conf.d/*

included file is:

check system master
  if memory is greater than 70% then exec "/usr/bin/pkill --signal 9 -f ruby;/etc/init.d/nginx restart; /bin/su -l user -c 'cd core/current && bundle exec sidekiqctl stop /home/user/app/current/tmp/pids/sidekiq.pid'"

Sometime, when ruby processes consume memory too fast, monit crashes. In log file I can see only:

[MSK Sep 25 16:59:52] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:00:22] error    : 'master' mem usage of 88.6% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:00:22] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:00:54] error    : 'master' mem usage of 92.4% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:00:55] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:01:30] error    : 'master' mem usage of 92.9% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:01:32] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:02:24] error    : 'master' mem usage of 93.4% matches resource limit [mem usage<70.0%]
[MSK Sep 25 17:02:49] info     : 'master' exec: /usr/bin/pkill
[MSK Sep 25 17:02:58] error    : Cannot fork a new process -- Cannot allocate memory

Then I need to restart monit process. I can understant that monit can't execute "stop" command due to lack of memory. But why it crashes

[Ilia Zenkevich]

Sorry for poor decoration at start. This issue is reproduced every day on my app server. If you need any info, please ask.

[Tildeslash]

Fixed Issue #255 : monit exited when program execution failed.

→ <<cset 7c3f512dea21>>

[Tildeslash]

Hi,

problem is fixed, monit will continue if fork() failed, please can you test the patched version?: https://mmonit.com/tmp/monit-5.14-p5.tar.gz

To compile:

tar -xzf monit-5.14-p5.tar.gz
cd monit-5.14-p5
./configure
make

Note that if the system is completely out of memory, monit may still exit during other operation which requires memory allocation.

[Ilia Zenkevich]

Thanks, will try and give feedback after some days

[Ilia Zenkevich]

Also, as I understand, when monit exit due to lack of memory - exit code is 0, because upstart doesn't respawn it. Am I right?

[Tildeslash]

The exit code was 1.

[Tildeslash]

Yet one thing ... the exec action is executed directly by execv(), if you need to use script, you have to wrap it using shell for example like this:

... then exec "/bin/bash -c '/usr/bin/pkill --signal 9 -f ruby;/etc/init.d/nginx restart;'"

as your script is little bit more complicated (sidekiq part uses also apostrophe), so i'd recommend to save the script into some file and call that script from the exec action.

[Ilia Zenkevich]

It seems works fine now, I see fork errors during memory usage problems, but main process works. Thank you.

[Tildeslash]

Removing version: 5.14 (automated comment)