adding depends statement in the environment

[Former user]

I am planning to add the depends statement for a set of applications which depend on mysql. My intention is to make sure that the applications should not start until mysql is started successfully. To accomplish this I planned to add the statement under each application but I did not find .monitrc file under each application. What would be another way to accomplish this feat.

[Lutz Mader]

Hello,
have you read the following samples, see
https://mmonit.com/wiki/Monit/ConfigurationExamples#mysqld

With regards,
Lutz

p.s.
General information how to use the depends on statement
https://mmonit.com/monit/documentation/monit.html#SERVICE-DEPENDENCIES

[Anirudh Pasalapudi]

Thank you very much for the reply. However in my scenario

1. I have about 44 applications running in a single server and among them there are front end apps as well as system apps
2. My aim is to make sure that monit starts system apps first and then it will start the rest of front end apps
3. Each front end application depends on more than one system application
4. To accomplish this I believe I need to use depends statement, but my concern is at what level I should use the statement whether at monit level which is by adding the statement in ~/etc/.monitrc or should I add the statement in the .conf files of each and every application
5. Also by adding depends statement the latency will be increased or not ?

‌

[Lutz Mader]

Hello Anirudh Pasalapudi,
there is no problem to do this, as long as your system capacity fit to the applications.

In general I use the host check to define implicit dependencies.
Unfortunately, services are OK for the first monitor cycle, this is the reason I use a greater monitor cycle for the services I define dependencies to.

check host Manager_Host with address myhost.local
# Prevent status "Online with all services" immediate after startup.
  every 5 cycles
  if failed port 5001 with timeout 30 seconds for 3 cycles then alert
  group Apm

check process Webview with pidfile "/usr/local/apm/wv.pid"
  start program "/usr/local/etc/monit/scripts/wv.sh start" with timeout 120 seconds
  stop program "/usr/local/etc/monit/scripts/wv.sh stop" with timeout 120 seconds
  if failed host myhost.local port 8080 protocol http for 30 cycles then exec "/usr/local/etc/monit/scripts/wv.sh restart"
     repeat every 50 cycles
  if not exist for 5 cycles then start
  if 3 restarts within 50 cycles then unmonitor
  depends on Manager_Host
  group Apm

The host check wait until the Manager is up and running and the Webview will be started by the depends on relationship.

I use config files for each application and each config file contain some definitions depends to the application.

The latency will be increased by each depends on level. This is normal for dependency definitions, but not a problem in general I think. But some of my applications spend 20 to 40 minutes to be up and running, therefore a minute less or more is not a problem.

With regards,
Lutz

[Lutz Mader]

BTW, some more complex samples are available too, see https://mmonit.com/wiki/Monit/HowTo

For some more details let us/me know some details about the used applications. Have a look to the application logs to get some useful information about your applications also.

The ports used by some applicatios are good points to define dependencies in general.

With regards,
Lutz

[Anirudh Pasalapudi]

Hello lutz,

Thank you very much for being very helpful. My scenario is explained below, please guide if possible

I have about 44 applications running in every instance and among them there are about 7 system/ infrastructure applications like mysql, rabbitmq, redis, solr, mongodb etc. So the task is to make sure that the non system applications depend on these system applications.

If the system application goes down due to some reason the apps which depend on it should go down and when that system app is brought back up all those dependent apps should come back up as well.

I tried to add the “depends on” statement in the .cfg files of the applications which are actually located in “/etc/monit.d” location.

On the other hand I did not find any control file “.monitrc” in ~/ location so I went ahead and created one and added below statements in it

include /etc/monit/monit.d/*

depends on on rabbitmq[, mongodb [, redis [, newsolr [, mysql ]]]]

So as I added the depends on mysql statement in one of the applications example app-a and when I stopped mysql the app-a is going down as well. But when I bring mysql up the app-a is not coming backup automatically rather I have to run the command “monit monitor app-a” to bring it back up

In another scenario, I added the same depends on mysql statement in another app-b cfg file as well. I repeated the process of stopping mysql. Now app-a went down as expected but nothing happened to app-b which is weird.

In another case I tried to add another system app to the depends statement like depends on mysql[, mongodb]. When I try to restart app-b it throws an error stating that mongodb is not available in control file, which is very confusing. Any help on this issue is really appreciated.

[Lutz Mader]

Hallo,
some answers only.

I tried to add the “depends on” statement in the .cfg files of the applications which are actually located in “/etc/monit.d” location.

On the other hand I did not find any control file ".monitrc" in ~/ location so I went ahead and created one and added below statements in it

include /etc/monit/monit.d/*

You should use a more specific mask like this
include /etc/monit/monit.d/*.cfg

If you start monit via systemd, the used monitrc file is specified with the start command, see https://mmonit.com/wiki/Monit/Starting

The used files are described in https://mmonit.com/monit/documentation/monit.html#FILES
My configuration file is "/usr/local/etc/monitrc" on MacOS, and start monit with root. On Linux and AIX I use "~/.monitrc", in the home folder, and start monit in the user context only.

So as I added the depends on mysql statement in one of the applications example app-a and when I stopped mysql the app-a is going down as well. But when I bring mysql up the app-a is not coming backup automatically rather I have to run the command "monit monitor app-a" to bring it back up.

Yes, this is the way how Monit handle dependencies and described in the Wiki. If you stop the root, all leafs are stopped. If you start a leaf, the root will started also. If you start the root, the leaf wil not started. Monit use a simple dependency only.

Someone else described some dependencies problems,
see https://bitbucket.org/tildeslash/monit/issues/728/after-update-of-monit-version-from-15121

Some use samples for your applications are available also,
see https://mmonit.com/wiki/Monit/ConfigurationExamples

My way to handle applications is little different.

check process Manager with pidfile "/usr/local/apm/em.pid"
  start program "/usr/local/etc/monit/scripts/em.sh start" with timeout 240 seconds
  stop program "/usr/local/etc/monit/scripts/em.sh stop" with timeout 300 seconds
  if failed host myhost.local port 8081 protocol http for 10 cycles then alert
  if failed host myhost.local port 5001 for 30 cycles then exec "/usr/local/etc/monit/scripts/em.sh restart"
     repeat every 50 cycles
  if not exist for 5 cycles then start
  if 3 restarts within 90 cycles then unmonitor
  group Apm

I start the DB/main application first and wait until the used ports are open.

check host Manager_Host with address myhost.local
# Prevent status "Online with all services" immediate after startup.
  every 5 cycles
  if failed port 5001 with timeout 30 seconds for 3 cycles then alert
  group Apm

If the port is open, I start the dependent application.

check process Webview with pidfile "/usr/local/apm/wv.pid"
  start program "/usr/local/etc/monit/scripts/wv.sh start" with timeout 120 seconds
  stop program "/usr/local/etc/monit/scripts/wv.sh stop" with timeout 120 seconds
  if failed host myhost.local port 8080 protocol http for 30 cycles then exec "/usr/local/etc/monit/scripts/wv.sh restart"
     repeat every 50 cycles
  if not exist for 5 cycles then start
  if 3 restarts within 50 cycles then unmonitor
  depends on Manager_Host
  group Apm

The frontend application wait until the port is available.
There is no direct dependency to the DB/main application, but if the port is not available the application will stopped.

You can use stop to stop the application, but to start the application in the right order, it is necessary to use monitor and wait, Monit will handle the application start.

You are right, Monit support simple depends on rules only.
I use loosely coupled dependencies based on an additional service (a port check) to handle the dependencies.
There is no way to define tightly dependencies with start and/or stop rules. Have a look to IBM Tivoli System Automation for Multiplatforms to get such a function.

With regards,
Lutz

p.s.
For more information about IBM Tivoli System Automation for Multiplatforms have a look to https://www.ibm.com/support/knowledgecenter/SSRM2X_4.1.0/com.ibm.samp.doc_4.1/welcome_samp.html

‌

[Lutz Mader]

Some samples for MySql and Redis (the ports) and two dummy applications.

check process MySql with pidfile /opt/mysql/data/myserver.mydomain.pid
   start program "/etc/init.d/mysql start"
   stop program "/etc/init.d/mysql stop"
   if failed host 192.168.1.1 port 3306 protocol mysql for 3 cycles then restart

check host MySql_Host with address myhost.local
# Prevent status "Online with all services" immediate after startup.
  every 2 cycles
  if failed port 3306 with timeout 30 seconds for 3 cycles then alert

check process Appl1 with pidfile "/var/run/appl1.pid"
  start program "/usr/local/etc/monit/scripts/appl1.sh start" with timeout 120 seconds
  stop program "/usr/local/etc/monit/scripts/appl1.sh stop" with timeout 120 seconds
  if not exist for 5 cycles then start
  if 3 restarts within 50 cycles then unmonitor
  depends on MySql_Host

check host SomePorts_Host with address myhost.local
# Prevent status "Online with all services" immediate after startup.
  every 2 cycles
  if failed port 3306 with timeout 30 seconds for 3 cycles then alert
  if failed port 6379 with timeout 30 seconds for 3 cycles then alert

check process Appl2 with pidfile "/var/run/appl2.pid"
  start program "/usr/local/etc/monit/scripts/appl2.sh start" with timeout 120 seconds
  stop program "/usr/local/etc/monit/scripts/appl2.sh stop" with timeout 120 seconds
  if not exist for 5 cycles then start
  if 3 restarts within 50 cycles then unmonitor
  depends on SomePorts_Host

A suggestion only,
Lutz

[Lutz Mader]

Hello,
sorry I can not agree, I get no error.
The following sample (see below) works well, from my point of view.

I do some testing base on monit 5.26.0 on a MacOS and AIX system.

With regards,
Lutz

The dependency sample.

#
## The branches of the roots.
#
check process branch_0 matching "sleep 600"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 600"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 600"
  depends on root_1,root_2
#
check process branch_1 matching "sleep 610"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 610"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 610"
  depends on root_1
#
check process branch_2 matching "sleep 620"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 620"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 620"
  depends on root_2
#
check process branch_3 matching "sleep 630"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 630"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 630"
  depends on root_3,branch_2
#
check process branch_4 matching "sleep 640"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 640"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 640"
  depends on root_4,branch_3
#
check process branch_5 matching "sleep 650"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 650"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 650"
  depends on root_4
#
## The roots.
#
check process root_0 matching "sleep 700"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 700"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 700"
#
check process root_1 matching "sleep 710"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 710"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 710"
#
check process root_2 matching "sleep 720"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 720"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 720"
#
check process root_3 matching "sleep 730"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 730"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 730"
  depends on root_0
#
check process root_4 matching "sleep 740"
  start program "/usr/local/etc/monit/scripts/sleep.sh start 740"
  stop program "/usr/local/etc/monit/scripts/sleep.sh stop 740"
  depends on root_3
#

‌