Monit doesn't let Django run

[Ing. Mario J. Schwaiger]

I'm having a Docker with testing-environment (Fedora) with a Django and an Apache. The Apache is for production, but the Django will live on for testing and development.
Both services are started with monit.
As both are using different ports it works fine if they are regularly started. Just monit doesn't seem to start the Django correctly:

monitrc

check process django-admin with pidfile /var/run/django-admin/django-admin.pid
         start program = "/usr/local/bin/django-admin runserver 0.0.0.0:9900 --pythonpath=/home/djangohome --settings=DjangoMoP.settings"


check process httpd with pidfile /var/run/httpd/httpd.pid
        start program = "/usr/sbin/httpd -D --FOREGROUND"

I'm starting monit -I -vv

and get these logs:

web_1  | Adding 'allow localhost' -- host resolved to [::ffff:127.0.0.1]
web_1  | Adding 'allow localhost' -- host resolved to [::1]
web_1  | Adding credentials for user 'admin'
web_1  | Adding 'allow 1xx.xxx.1.xxx' -- host resolved to [::ffff:1xx.xxx.1.xxx]
web_1  | Runtime constants:
web_1  |  Control file       = /etc/monitrc
web_1  |  Log file           = syslog
web_1  |  Pid file           = /run/monit.pid
web_1  |  Id file            = /root/.monit.id
web_1  |  State file         = /root/.monit.state
web_1  |  Debug              = True
web_1  |  Log                = True
web_1  |  Use syslog         = True
web_1  |  Is Daemon          = True
web_1  |  Use process engine = True
web_1  |  Limits             = {
web_1  |                     =   programOutput:     512 B
web_1  |                     =   sendExpectBuffer:  256 B
web_1  |                     =   fileContentBuffer: 512 B
web_1  |                     =   httpContentBuffer: 1 MB
web_1  |                     =   networkTimeout:    5 s
web_1  |                     =   programTimeout:    5 m
web_1  |                     =   stopTimeout:       30 s
web_1  |                     =   startTimeout:      30 s
web_1  |                     =   restartTimeout:    30 s
web_1  |                     = }
web_1  |  On reboot          = start
web_1  |  Poll time          = 30 seconds with start delay 0 seconds
web_1  |  Start monit httpd  = True
web_1  |  httpd bind address = Any/All
web_1  |  httpd portnumber   = 2812
web_1  |  httpd signature    = Enabled
web_1  |  httpd auth. style  = Basic Authentication and Host/Net allow list
web_1  | 
web_1  | The service list contains the following entries:
web_1  | 
web_1  | Process Name          = django-admin
web_1  |  Pid file             = /var/run/django-admin/django-admin.pid
web_1  |  Monitoring mode      = active
web_1  |  On reboot            = start
web_1  |  Start program        = '/usr/local/bin/django-admin runserver 0.0.0.0:9900 --pythonpath=/home/djangohome --settings=DjangoMoP.settings' timeout 30 s
web_1  |  Existence            = if does not exist then restart
web_1  | 
web_1  | Process Name          = httpd
web_1  |  Pid file             = /var/run/httpd/httpd.pid
web_1  |  Monitoring mode      = active
web_1  |  On reboot            = start
web_1  |  Start program        = '/usr/sbin/httpd -D --FOREGROUND' timeout 30 s
web_1  |  Existence            = if does not exist then restart
web_1  | 
web_1  | System Name           = 78b52ae2b042
web_1  |  Monitoring mode      = active
web_1  |  On reboot            = start
web_1  | 
web_1  | -------------------------------------------------------------------------------
web_1  | pidfile '/run/monit.pid' does not exist
web_1  | Starting Monit 5.25.1 daemon with http interface at [*]:2812
web_1  | Starting Monit HTTP server at [*]:2812
web_1  | Monit HTTP server started
web_1  | '78b52ae2b042' Monit 5.25.1 started
web_1  | pidfile '/var/run/django-admin/django-admin.pid' does not exist
web_1  | 'django-admin' process is not running
web_1  | -------------------------------------------------------------------------------
web_1  |     monit(+0x25c40) [0x55f7ccd36c40]
web_1  |     monit(LogError+0xba) [0x55f7ccd3794a]
web_1  |     monit(Event_post+0x271) [0x55f7ccd330b1]
web_1  |     monit(check_process+0x57b) [0x55f7ccd4868b]
web_1  |     monit(validate+0x1c3) [0x55f7ccd480d3]
web_1  |     monit(main+0x44a) [0x55f7ccd234fa]
web_1  |     /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f1df5c6e413]
web_1  |     monit(_start+0x2a) [0x55f7ccd23c4a]
web_1  | -------------------------------------------------------------------------------
web_1  | 'django-admin' trying to restart
web_1  | 'django-admin' stop skipped -- method not defined
web_1  | pidfile '/var/run/django-admin/django-admin.pid' does not exist
web_1  | 'django-admin' start: '/usr/local/bin/django-admin runserver 0.0.0.0:9900 --pythonpath=/home/djangohome --settings=DjangoMoP.settings'
web_1  | Performing system checks...
web_1  | 
web_1  | System check identified no issues (0 silenced).
web_1  | February 05, 2020 - 10:42:47
web_1  | Django version 2.2.4, using settings 'DjangoMoP.settings'
web_1  | Starting development server at <http://0.0.0.0:9900/>
web_1  | Quit the server with CONTROL-C.
web_1  | 'django-admin' failed to start (exit status -1) -- '/usr/local/bin/django-admin runserver 0.0.0.0:9900 --pythonpath=/home/djangohome --settings=DjangoMoP.settings': Program timed out -- Performing system checks...
web_1  | 
web_1  | System check identified no issues (0 silenced).
web_1  | February 05, 2020 - 10:42:47
web_1  | Django version
web_1  | -------------------------------------------------------------------------------
web_1  |     monit(+0x25c40) [0x55f7ccd36c40]
web_1  |     monit(LogError+0xba) [0x55f7ccd3794a]
web_1  |     monit(Event_post+0x271) [0x55f7ccd330b1]
web_1  |     monit(+0x20de4) [0x55f7ccd31de4]
web_1  |     monit(control_service+0x19e) [0x55f7ccd322fe]
web_1  |     monit(+0x21d33) [0x55f7ccd32d33]
web_1  |     monit(Event_post+0x2b0) [0x55f7ccd330f0]
web_1  |     monit(check_process+0x57b) [0x55f7ccd4868b]
web_1  |     monit(validate+0x1c3) [0x55f7ccd480d3]
web_1  |     monit(main+0x44a) [0x55f7ccd234fa]
web_1  |     /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f1df5c6e413]
web_1  |     monit(_start+0x2a) [0x55f7ccd23c4a]
web_1  | -------------------------------------------------------------------------------
web_1  | 'httpd' process test failed [pid=16] -- No such process
web_1  | 'httpd' process is not running
web_1  | -------------------------------------------------------------------------------
web_1  |     monit(+0x25c40) [0x55f7ccd36c40]
web_1  |     monit(LogError+0xba) [0x55f7ccd3794a]
web_1  |     monit(Event_post+0x271) [0x55f7ccd330b1]
web_1  |     monit(check_process+0x57b) [0x55f7ccd4868b]
web_1  |     monit(validate+0x1c3) [0x55f7ccd480d3]
web_1  |     monit(main+0x44a) [0x55f7ccd234fa]
web_1  |     /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f1df5c6e413]
web_1  |     monit(_start+0x2a) [0x55f7ccd23c4a]
web_1  | -------------------------------------------------------------------------------
web_1  | 'httpd' trying to restart
web_1  | 'httpd' stop skipped -- method not defined
web_1  | 'httpd' process test failed [pid=16] -- No such process
web_1  | 'httpd' start: '/usr/sbin/httpd -D --FOREGROUND'
web_1  | Cannot read proc file '/proc/18/io' -- Permission denied
web_1  | Cannot read proc file '/proc/19/io' -- Permission denied
web_1  | Cannot read proc file '/proc/20/io' -- Permission denied
web_1  | Cannot read proc file '/proc/21/io' -- Permission denied
web_1  | Cannot read proc file '/proc/22/io' -- Permission denied
web_1  | 'httpd' started
web_1  | Cannot read proc file '/proc/18/io' -- Permission denied
web_1  | Cannot read proc file '/proc/19/io' -- Permission denied
web_1  | Cannot read proc file '/proc/20/io' -- Permission denied
web_1  | Cannot read proc file '/proc/21/io' -- Permission denied
web_1  | Cannot read proc file '/proc/22/io' -- Permission denied
web_1  | pidfile '/var/run/django-admin/django-admin.pid' does not exist
web_1  | 'django-admin' process is not running
web_1  | -------------------------------------------------------------------------------
web_1  |     monit(+0x25c40) [0x55f7ccd36c40]
web_1  |     monit(LogError+0xba) [0x55f7ccd3794a]
web_1  |     monit(Event_post+0x271) [0x55f7ccd330b1]
web_1  |     monit(check_process+0x57b) [0x55f7ccd4868b]
web_1  |     monit(validate+0x1c3) [0x55f7ccd480d3]
web_1  |     monit(main+0x44a) [0x55f7ccd234fa]
web_1  |     /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f1df5c6e413]
web_1  |     monit(_start+0x2a) [0x55f7ccd23c4a]
web_1  | -------------------------------------------------------------------------------
web_1  | 'django-admin' trying to restart
web_1  | 'django-admin' stop skipped -- method not defined
web_1  | pidfile '/var/run/django-admin/django-admin.pid' does not exist
web_1  | 'django-admin' start: '/usr/local/bin/django-admin runserver 0.0.0.0:9900 --pythonpath=/home/djangohome --settings=DjangoMoP.settings'
web_1  | Error: That port is already in use.
web_1  | Performing system checks..

The port 9900 is not really taken. Eventually these are the ports to be listened:

[root@78b52ae2b042 djangohome]# lsof -i -P -n | grep LISTEN 
monit     1 root   10u  IPv4 4848274      0t0  TCP *:2812 (LISTEN)
monit     1 root   11u  IPv6 4848275      0t0  TCP *:2812 (LISTEN)
python3  10 root    5u  IPv4 4852752      0t0  TCP *:9900 (LISTEN)

If the django-admin process is removed from the monitrc file, the python3-listener will not come up too. So somehow it appears the Django-process is not started properly or somehow interrupted.

What could be the case and how is is supposed to be started?

In the time which is set for the interval-checks Django runs fine.
Then the process is interrupted (I assume because Django has its own output while monit expects some kind of success?) And can't be started properly any more.
How to solve this issue?

[Henning Bopp]

The thing is that monit is not a process manager but more a service watcher...

You need to provide a service starter to start program, like a systemctl call

check process django-admin with pidfile /var/run/django-admin/django-admin.pid
         start program = "/bin/systemctl start django-admin"

The program call has a timeout in which it has to start the service. Monit cannot handle a foreground-task like you try to run...

You could also try to background the service yourself with something like

#!/usr/bin/env bash
/usr/local/bin/django-admin runserver 0.0.0.0:9900 --pythonpath=/home/djangohome --settings=DjangoMoP.settings & 2>&1 > /dev/null
curl -s --connect-timeout 30 localhost:9900 > /dev/null

For your Apache you might want to try httpd -k start. You could also provide a stop program with

stop program = "/usr/sbin/httpd -k stop"

In any case I can also only recommend to read the documentation page... It's full of clever examples and useful information...