Monit fails with message APACHE-STATUS: error — no scoreboard found

[Former user]

We have mod_status setup and we have the weblink monitor link working real well. both the http://www.example.com/server-status?refresh=5 and http://www.example.com/server-status?auto.

We just setup Monit with the following in /etc/monit.conf:

check process apache with pidfile /usr/local/apache/logs/httpd.pid
    start "/etc/init.d/httpd start"
    stop  "/etc/init.d/httpd stop"
    if failed port 80
        protocol apache-status loglimit > 60%
    then restart

Unfortunately Monit fails with the following message in /var/log/monit:

[MST Mar 31 12:05:20] error : 'apache' failed protocol test [APACHESTATUS] at [localhost]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found

What I understand from this message is its looking for the word 'scoreboard' in server-status page Interesting server-status page doesn't contain the word 'scoreboard' above the lines:

LL_LWLLLLL_LLLLLLLLL_LLLLLLLLL.LW_LL_LLLLLLL_L.LLLLLLLLLL_LLLLLL
L_LLLLLLLLLLLW_LL.LLLLLL_LRLLLL_L_LLLLLL__WL__LLLLLLL_LL_LLLLLLL
LLR_LLLLLLLLLL.LLLLL_RLLLLLLLLL_LLLLLL__LLLLLLL_LLLLL_LLLLLLLLLR
LLLLLLLLL_LL_L_LLLCLLLLLLLLLLLLL.LLLLLLLLLLLLLWLLLLLLL_LLL_LLLLL

However the word 'scoreboard' is present in the link server-status?auto.

Appreciate any help. Here is the systems we have: CentOS 2.6.32-573.12.1.el6.x86_64 GNU/Linux Apache version: Apache/2.4.18 (Unix)

[Tildeslash]

Monit uses by default "/server-status?auto" to retrieve the apache statistics. If your server status is in different path, please upgrade monit to 5.17.1 ... the 5.17 added support for a "path" option (https://mmonit.com/monit/documentation/monit.html#APACHE-STATUS).

Example:

if failed port 80
    protocol apache-status
    path "/my/server-status"
    loglimit > 60%
then restart

If the problem will persist, please get a network trace between monit and target apache server (for example using wireshark or tcpdump) and send it to [e-mail]

[Tony]

Thank you tildeslash. server status is in the same path. /server-status. And /server-status?auto is just fine.

Just upgraded to 5.17.1, by the way:

root@phoenix [/opt/monit-5.17.1/bin]# ./monit -v Adding host allow 'localhost' Skipping redundant host 'localhost' Adding credentials for user 'admin' Runtime constants: Control file = /opt/monit-5.17.1/bin/monitrc Log file = /var/log/monit

---

Still experiencing the same:

[MST Apr 1 10:38:23] error : 'apache' failed to stop (exit status -1) -- no output [MST Apr 1 10:38:53] info : 'apache' process is running after previous exec error (slow starting or manually recovered?) [MST Apr 1 10:38:53] error : 'apache' failed protocol test [APACHESTATUS] at [localhost]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found [MST Apr 1 10:38:53] info : 'apache' trying to restart [MST Apr 1 10:38:53] info : 'apache' stop: /etc/init.d/httpd

---

I have not used wireshart/tcpdump before; will get the network trace.

Thanks again !

[Tildeslash]

I think the problem could be in the configuration ... when you try "http://www.example.com/server-status?auto", you access your webserver via host "www.example.com", but your monit configuration doesn't specify any host:

if failed port 80
    protocol apache-status loglimit > 60%
then restart

In such case monit connects via localhost (127.0.0.1), which may have server-status disabled.

Please try to modify the monit configuration by adding the host:

if failed
    host www.example.com
    port 80
    protocol apache-status loglimit > 60%
then restart

[Tony]

Thank you tildeslash. Did the config change

 if failed
 host phoenix.example.com
 port 80
 protocol apache-status loglimit > 60%
 then restart

Reinitializing Monit - Control file '/opt/monit-5.17.1/bin/monitrc' [MST Apr 4 06:17:07] info : Shutting down Monit HTTP server [MST Apr 4 06:17:08] info : Monit HTTP server stopped [MST Apr 4 06:17:08] debug : Adding host allow 'localhost' [MST Apr 4 06:17:08] debug : Skipping redundant host 'localhost' [MST Apr 4 06:17:08] debug : Adding credentials for user 'admin' [MST Apr 4 06:17:08] info : Starting Monit HTTP server at [localhost]:2812 [MST Apr 4 06:17:08] info : Monit HTTP server started [MST Apr 4 06:17:08] info : 'phoenix.example.com' Monit reloaded [MST Apr 4 06:17:08] debug : 'apache' process is running with pid 6397 [MST Apr 4 06:17:08] debug : 'apache' zombie check succeeded [MST Apr 4 06:17:13] debug : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- no scoreboard found [MST Apr 4 06:17:13] error : 'apache' failed protocol test [APACHESTATUS] at [phoenix.vwclient.com]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found [MST Apr 4 06:17:13] info : 'apache' trying to restart [MST Apr 4 06:17:13] info : 'apache' stop: /etc/init.d/httpd [MST Apr 4 06:17:43] error : 'apache' failed to stop (exit status -1) -- no output [MST Apr 4 06:18:13] debug : 'apache' process is running with pid 6397 [MST Apr 4 06:18:13] info : 'apache' process is running after previous exec error (slow starting or manually recovered?) [MST Apr 4 06:18:13] debug : 'apache' zombie check succeeded [MST Apr 4 06:18:18] debug : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- no scoreboard found

Correct server names are taken. Essentially facing the same situation. "no scorecard found."

[Tony]

Hello tildeslash, the monit process and apache server are running in the same server. Therefore not sure how I can take tcpdump even if that is possible.

One other thing we noticed is ever since we added the host name, we are consistently seeing 'Socket test failed'.

[MST Apr  5 14:22:35] debug    : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- no scoreboard found
[MST Apr  5 14:22:35] error    : 'apache' failed protocol test [APACHESTATUS] at [phoenix.example.com]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found

I tried to establish socket connection at 80 using command line and it worked.

root@phoenix [/usr/local/apache/conf]# curl -v 198.xx.xx.xx8
* About to connect() to 198.xx.xx.xx8 port 80 (#0)
*   Trying 198.xx.xx.xx8... connected

Not sure why Monit has trouble establishing port 80 socket to localhost and maybe that leads to not finding the scorecard ..

[Tildeslash]

The problem is not with socket connection itself - according to the error message, monit was able to connect to the given socket, but the server-status was not found.

With the "host" added, monit connects to the correct host (198.xx.xx.xx8) instead of default 127.0.0.1 when no host is specified, but the problem could be in the Host header, if the Apache server-status is configured for some specific name-based virtual host only. The apache-status protocol test will set the Host header based on name associated with the remote ip (should correspond to "host 198.xx.xx.xx8" output), which may be different then your server-status' virtual host name if that machine has multiple hosts.

Please take a tcpdump and send it to [e-mail]:

tcpdump -i any -s 0 -w /tmp/apache_status.pcap port 80

Catch a monit apache-status test in it + do also "curl http://www.example.com/server-status?auto" while tcpdump is active.

Alternatively if that host is remotely accessible via HTTP, please send connection details to [e-mail], so we can try to replicate the problem.

[Tony]

The httpd.conf had

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from localhost
    Allow from 98.xx.xx.x03
</Location>

Since apache and monit are running in the same server I thought it will work fine since I have localhost in this config. (98.xx.xx.x03 is the ip-address of our office) I guess I still needed to explicitly mention the same server address and I made it like this:

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from localhost
    Allow from 198.xx.xx.xx8
    Allow from 98.xx.xx.x03
</Location>

And it worked:

[MST Apr 11 11:53:10] debug    : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- 83 percent of processes are logging
[MST Apr 11 11:53:10] error    : 'apache' failed protocol test [APACHESTATUS] at [phoenix.example.com]:80 [TCP/IP] -- APACHE-STATUS: error -- 83 percent of processes are logging
[MST Apr 11 11:53:10] debug    : M/Monit: event message sent to http://[108.xx.xx.x94]:8080/collector

Thanks a lot tildeslash. I think it looks good now !

[Tildeslash]

Removing version: 5.14 (automated comment)