monit status gives incorrect resource utilization

[nshubhy]

We are monitoring a script which is starting the mysql process. Every 30 seconds, Monit keeps checking if this script exists in the process table. If the script is present in the process table, that means the mysql process is running. On doing "monit status mysql", we expect the output to be overall resource utilization from mysql process but the output reported is incorrect, almost 0% for the resources:

Process 'mysql' 
status Running 
monitoring status Monitored 
pid 2355 
parent pid 1986 
uid 0 
effective uid 0 
gid 0 
uptime 31m 
children 2 
memory 1.6 MB 
memory total 180.0 MB 
memory percent 0.0% 
memory percent total 2.3% 
cpu percent 0.0% 
cpu percent total 0.1% 
data collected Mon, 28 Mar 2016 23:25:36

[nshubhy]

@harivrdhn

[Tildeslash]

The process check uses either a pidfile or process name pattern. In this case your check matches a process with PID 2355.

Please send output of following command:

ps -ef | grep 2355  #note: modify the PID according to "monit status mysql" output if it has changed in the meantime

Please check if the monitored PID matches the PID of the process that you expect to be monitored - if not, fix the mysql process check configuration.

If the process is correct and you think the resource usage is higher, please provide method that you used to measure the utilization.

[Tildeslash]

Issue #338 was marked as a duplicate of this issue.

[Tildeslash]

Thanks for data.

The "run.sh" script just starts the mysql daemon, in your case via "/usr/bin/mysqld_safe", which in turn will also just start the main mysqld process (not visible in the provided "ps" output, as it is child of mysld_safe). The real work is done by that mysqld process, not run.sh script nor mysqld_safe.

The CPU usage of "run.sh" will be zero, monit report is correct (the "cpu percent 0.0%" statistics shows the CPU usage of the monitored process itself - not its children).

You can see the total CPU usage of the whole children subtree (including the main "mysqld" process), in the "cpu percent total" statistics. Note however, that until monit 5.16, the CPU usage was divided by number of available CPUs. Monit 5.16 has modified CPU usage calculation formula, changelog excerpt (https://mmonit.com/monit/changes/):

Fixed: Issue #230:

In short:
#######
You can now check a process' CPU usage in a more natural way as a percentage between 0-100 on a multi-core system. For instance, to check if a    
single-threaded application like node.js has gotten stuck on 100% CPU,
if cpu usage = 100% for 2 cycles then alert

In detail:
#######
Calculating a process CPU usage has been normalised to take into account the number of threads in use by the process. Previously Monit 
calculated process CPU usage as a fraction over available CPU cores. For instance, if you wanted to check if a single-threaded application used 
100% CPU you had to check for 25% CPU utilization on a 4 core machine (100/4). Likewise, in top terminology, a multi-threaded application could 
use up to 400% CPU on the same machine. Monit now calculates CPU usage based on number of threads vs. available CPU cores. If a process has 
one thread, 100% CPU usage is the same as 100% utilization of one CPU core. If it has 2 threads, 100% CPU usage is reported when it uses 2 
CPU cores 100%, etc. If a process has more threads than the machine's available CPU cores then 100% CPU usage corresponds to the utilization 
of all available CPU cores.

Action plan:

1.) upgrade monit (5.17.1 recommended) ... will calculate the CPU usage with respect to number of process' threads

2.) if you want to see the CPU usage in the "cpu percent" statistics instead of "cpu percent total", you need to either reconfigure the check to test the main mysqld process (not the start script), or if the run.sh itself needs to be monitored, you can add second "check process" just for the mysql process itself.

3.) there may be difference in comparison with docker stats, as docker reports CPU usage of the whole container, whereas monit's "check process" reports the CPU usage of the process itself (or including children in the "cpu percent total"). The difference can be also caused if docker will present different number of system CPUs inside the container (/proc/cpuinfo inside the container vs. /proc/cpuinfo on the parent system).

[nshubhy]

Thanks for the quick response. I understand that in monit 5.17, CPU usage is being calculated with respect to number of process threads. Does the same apply to memory usage because we observe the same in memory percentage reported by monit?

[Tildeslash]

The resident memory usage is related to the whole process image - regardless of number of threads.

When you use memory usage reported by docker statistics, it reports the memory used by the whole system image (all processes in the container), whereas Monit reports the memory used by the monitored process.

[nshubhy]

Resolved by tildeslash