I have monit checking several hundred tomcat processes, we rebooted the server for patching and several of the JVM's didn't start, because the old defunct catalina.pid file was present on reboot. Here is a sample output of the issue:
Process 'tomcat-inigral-sso' status Running monitoring status Monitored pid 6022 parent pid 0 uid -1 effective uid -1 gid -1 uptime 0m children 0 memory 0.0 B memory total 0.0 B memory percent 0.0% memory percent total 0.0% cpu percent 0.0% cpu percent total 0.0% port response time 0.000s to localhost:58097 [HTTP via TCP] data collected Thu, 05 Feb 2015 10:30:09
If I remove the catalina.pid file, monit will pickup that it isn't running and restart the service as intended.
[MST Feb 5 10:37:13] error : 'tomcat-inigral-sso' process is not running [MST Feb 5 10:37:13] info : 'tomcat-inigral-sso' trying to restart [MST Feb 5 10:37:13] info : 'tomcat-inigral-sso' restart: /usr/local/tomcat/inigral-sso/tomcat-inigral-sso
My check looks like below:
# *** ANSIBLE MANAGED - DO NOT EDIT BY HAND *** # check process tomcat-inigral-sso with pidfile /usr/local/tomcat/inigral-sso/catalina.pid not every "* 3-6 * * *" start program = "/usr/local/tomcat/inigral-sso/tomcat-inigral-sso start" stop program = "/usr/local/tomcat/inigral-sso/tomcat-inigral-sso stop" restart program = "/usr/local/tomcat/inigral-sso/tomcat-inigral-sso restart" if failed port 58097 and protocol http retry 6 then restart group tomcat check file tomcat-inigral-sso-logs with path /usr/local/tomcat/inigral-sso/logs/catalina.out not every "* 3-6 * * *" restart program = "/usr/local/tomcat/inigral-sso/tomcat-inigral-sso restart" if match "^.*OutOfMemoryError.*$" then restart group tomcat depends on tomcat-inigral-sso