Monit Shows Apache CPU usage 100%

[Former user]

For some reason Monit web page shows that apache is using 100% of the CPU Total, even though the real value is in range 15-25%. No alarms are generated so the problem is somewhere in the web page code, I think.

I'm using pretty basic httpd configuration:

check process Apache with pidfile /var/run/httpd.pid
    group webserver
    start program "/etc/init.d/httpd start" with timeout 60 seconds
    stop program  "/etc/init.d/httpd stop"
    if failed host <server-name> port 80 protocol http and request "/monit/token" then alert
    if cpu         > 30%      for 2 cycles then alert
    if totalmem > 4.5 GB for 5 cycles then exec "/usr/sbin/apachectl graceful"

Version information:

• Apache 2.2.15
• CentOS 6.8
• PHP 5.4.45
• Monit 5.20

My previous Monit version that worked was 5.14. It was installed from CentOS 6 repository using yum, whereas I installed this new 5.20 version from this binary packeage: monit-5.20.0-linux-x64.tar.gz

[Tildeslash]

The "total cpu" is sum of cpu usage of apache main/parent process and all its children. The alert is not triggered, because the is no "total cpu" test, just "cpu" which tests the apache main process only (it has usually low CPU usage).

Please can you attach output of the following command?:

ps -eo pcpu,args | egrep "(apache|httpd)" | grep -v grep

[Tildeslash]

waiting for data

[Former user]

Sorry it took so long, but here it is:

 2.4 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.6 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.2 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 0.0 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.6 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.6 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.6 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.0 /usr/sbin/httpd -k start
 2.1 /usr/sbin/httpd -k start
 2.1 /usr/sbin/httpd -k start
 2.2 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.3 /usr/sbin/httpd -k start
 2.7 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 2.4 /usr/sbin/httpd -k start
 1.7 /usr/sbin/httpd -k start
 1.9 /usr/sbin/httpd -k start
 2.5 /usr/sbin/httpd -k start
 2.0 /usr/sbin/httpd -k start

Summing up, that makes it over 100%... How come then version 5.14 of Monit showed different values and the server load average, as reported by top, is under 1.5 with six core processor?

[Former user]

Reopened after providing data

[Tildeslash]

The total cpu usage is 112.7 so monit report is correct.

[Former user]

How come Monit report is corrent?

My math says 100% differs a little bit from 112.7%...

[Former user]

Thanks for bearing with me.

Any other usefull information that I could provide?

[Tildeslash]

I think the problem could be related to how the CPU usage is calculated since monit 5.16, changelog excerpt:

Fixed: Issue #230: The process CPU usage calculation now reflects the number of
process threads. Originally monit showed process' CPU usage as its fraction of
all available CPU resources utilization (number of CPU cores). For single-threaded
applications that was however tricky, as such process may utilize one CPU core only
and if it was working on its limits, on 8-CPU-cores machine monit showed 12.5% CPU
utilization (100/8). If you wanted to check that the process is not stuck on 100%,
you had to adjust the limit reflecting the CPU cores on the machine. Monit now
calculates the CPU usage based on number of threads vs. available CPU cores. If the
process has one thread, the 100% CPU usage equals to 100% utilization of one CPU core.
If it has 2 threads, 100% CPU usage is reported when it uses 2 CPU cores on 100%, etc.
If the process has more threads then the machine's available CPU cores, then the 100%
CPU usage corresponds to utilization of all available CPU cores.

Whereas the formula is correct for the process CPU usage, in the case of "total cpu usage" where each child is also single-threaded (as in apache case), the child cpu usage is percent of single CPU core usage (equal to "ps" output) ... on 6-core machine it may then sum up to 600% CPU usage (monit has cap on 100%). The 112.7% total where the base is single CPU core may roughly correspond to 1.5 load average.

[Former user]

Yes, I tough also that something has to be had changed since Monit 5.14.

Any way to fix this issue without messing up the fix above?

By the way, it is good that Monit shows 0-100% whether there is one or several cores in use.

[Former user]

Any progress to solve this issue?

I could e.g test pre-release if it becomes available with fix.

[Tildeslash]

not yet, working on disk IO monitoring

[Tildeslash]

Duplicate of #657.

[Tildeslash]

The problem was fixed in the development version (the issue #657 describes the same problem). The Monit 5.24.0 that will contain the fix should be available ca. within 1 week.