Monit fails with message APACHE-STATUS: error — no scoreboard found

Issue #341 resolved
Former user created an issue

We have mod_status setup and we have the weblink monitor link working real well. both the http://www.example.com/server-status?refresh=5 and http://www.example.com/server-status?auto.

We just setup Monit with the following in /etc/monit.conf:

check process apache with pidfile /usr/local/apache/logs/httpd.pid
    start "/etc/init.d/httpd start"
    stop  "/etc/init.d/httpd stop"
    if failed port 80
        protocol apache-status loglimit > 60%
    then restart

Unfortunately Monit fails with the following message in /var/log/monit:

[MST Mar 31 12:05:20] error : 'apache' failed protocol test [APACHESTATUS] at [localhost]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found

What I understand from this message is its looking for the word 'scoreboard' in server-status page Interesting server-status page doesn't contain the word 'scoreboard' above the lines:

LL_LWLLLLL_LLLLLLLLL_LLLLLLLLL.LW_LL_LLLLLLL_L.LLLLLLLLLL_LLLLLL
L_LLLLLLLLLLLW_LL.LLLLLL_LRLLLL_L_LLLLLL__WL__LLLLLLL_LL_LLLLLLL
LLR_LLLLLLLLLL.LLLLL_RLLLLLLLLL_LLLLLL__LLLLLLL_LLLLL_LLLLLLLLLR
LLLLLLLLL_LL_L_LLLCLLLLLLLLLLLLL.LLLLLLLLLLLLLWLLLLLLL_LLL_LLLLL

However the word 'scoreboard' is present in the link server-status?auto.

Appreciate any help. Here is the systems we have: CentOS 2.6.32-573.12.1.el6.x86_64 GNU/Linux Apache version: Apache/2.4.18 (Unix)

Comments (10)

  1. Tildeslash repo owner

    Monit uses by default "/server-status?auto" to retrieve the apache statistics. If your server status is in different path, please upgrade monit to 5.17.1 ... the 5.17 added support for a "path" option (https://mmonit.com/monit/documentation/monit.html#APACHE-STATUS).

    Example:

    if failed port 80
        protocol apache-status
        path "/my/server-status"
        loglimit > 60%
    then restart
    

    If the problem will persist, please get a network trace between monit and target apache server (for example using wireshark or tcpdump) and send it to support@mmonit.com

  2. Tony Vibrant

    Thank you tildeslash.
    server status is in the same path. /server-status.
    And /server-status?auto is just fine.

    Just upgraded to 5.17.1, by the way:

    root@phoenix [/opt/monit-5.17.1/bin]# ./monit -v
    Adding host allow 'localhost'
    Skipping redundant host 'localhost'
    Adding credentials for user 'admin'
    Runtime constants:
    Control file = /opt/monit-5.17.1/bin/monitrc
    Log file = /var/log/monit


    Still experiencing the same:

    [MST Apr 1 10:38:23] error : 'apache' failed to stop (exit status -1) -- no output
    [MST Apr 1 10:38:53] info : 'apache' process is running after previous exec error (slow starting or manually recovered?)
    [MST Apr 1 10:38:53] error : 'apache' failed protocol test [APACHESTATUS] at [localhost]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found
    [MST Apr 1 10:38:53] info : 'apache' trying to restart
    [MST Apr 1 10:38:53] info : 'apache' stop: /etc/init.d/httpd


    I have not used wireshart/tcpdump before; will get the network trace.

    Thanks again !

  3. Tildeslash repo owner

    I think the problem could be in the configuration ... when you try "http://www.example.com/server-status?auto", you access your webserver via host "www.example.com", but your monit configuration doesn't specify any host:

    if failed port 80
        protocol apache-status loglimit > 60%
    then restart
    

    In such case monit connects via localhost (127.0.0.1), which may have server-status disabled.

    Please try to modify the monit configuration by adding the host:

    if failed
        host www.example.com
        port 80
        protocol apache-status loglimit > 60%
    then restart
    
  4. Tony Vibrant

    Thank you tildeslash. Did the config change

     if failed
     host phoenix.example.com
     port 80
     protocol apache-status loglimit > 60%
     then restart
    

    Reinitializing Monit - Control file '/opt/monit-5.17.1/bin/monitrc'
    [MST Apr 4 06:17:07] info : Shutting down Monit HTTP server
    [MST Apr 4 06:17:08] info : Monit HTTP server stopped
    [MST Apr 4 06:17:08] debug : Adding host allow 'localhost'
    [MST Apr 4 06:17:08] debug : Skipping redundant host 'localhost'
    [MST Apr 4 06:17:08] debug : Adding credentials for user 'admin'
    [MST Apr 4 06:17:08] info : Starting Monit HTTP server at [localhost]:2812
    [MST Apr 4 06:17:08] info : Monit HTTP server started
    [MST Apr 4 06:17:08] info : 'phoenix.example.com' Monit reloaded
    [MST Apr 4 06:17:08] debug : 'apache' process is running with pid 6397
    [MST Apr 4 06:17:08] debug : 'apache' zombie check succeeded
    [MST Apr 4 06:17:13] debug : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- no scoreboard found
    [MST Apr 4 06:17:13] error : 'apache' failed protocol test [APACHESTATUS] at [phoenix.vwclient.com]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found
    [MST Apr 4 06:17:13] info : 'apache' trying to restart
    [MST Apr 4 06:17:13] info : 'apache' stop: /etc/init.d/httpd
    [MST Apr 4 06:17:43] error : 'apache' failed to stop (exit status -1) -- no output
    [MST Apr 4 06:18:13] debug : 'apache' process is running with pid 6397
    [MST Apr 4 06:18:13] info : 'apache' process is running after previous exec error (slow starting or manually recovered?)
    [MST Apr 4 06:18:13] debug : 'apache' zombie check succeeded
    [MST Apr 4 06:18:18] debug : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- no scoreboard found

    Correct server names are taken. Essentially facing the same situation.
    "no scorecard found."

  5. Tony Vibrant

    Hello tildeslash, the monit process and apache server are running in the same server.
    Therefore not sure how I can take tcpdump even if that is possible.

    One other thing we noticed is ever since we added the host name, we are consistently seeing 'Socket test failed'.

    [MST Apr  5 14:22:35] debug    : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- no scoreboard found
    [MST Apr  5 14:22:35] error    : 'apache' failed protocol test [APACHESTATUS] at [phoenix.example.com]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found
    

    I tried to establish socket connection at 80 using command line and it worked.

    root@phoenix [/usr/local/apache/conf]# curl -v 198.xx.xx.xx8
    * About to connect() to 198.xx.xx.xx8 port 80 (#0)
    *   Trying 198.xx.xx.xx8... connected
    

    Not sure why Monit has trouble establishing port 80 socket to localhost and maybe that leads to not finding the scorecard ..

  6. Tildeslash repo owner

    The problem is not with socket connection itself - according to the error message, monit was able to connect to the given socket, but the server-status was not found.

    With the "host" added, monit connects to the correct host (198.xx.xx.xx8) instead of default 127.0.0.1 when no host is specified, but the problem could be in the Host header, if the Apache server-status is configured for some specific name-based virtual host only. The apache-status protocol test will set the Host header based on name associated with the remote ip (should correspond to "host 198.xx.xx.xx8" output), which may be different then your server-status' virtual host name if that machine has multiple hosts.

    Please take a tcpdump and send it to support@mmonit.com:

    tcpdump -i any -s 0 -w /tmp/apache_status.pcap port 80
    

    Catch a monit apache-status test in it + do also "curl http://www.example.com/server-status?auto" while tcpdump is active.

    Alternatively if that host is remotely accessible via HTTP, please send connection details to support@mmonit.com, so we can try to replicate the problem.

  7. Tony Vibrant

    The httpd.conf had

    <Location /server-status>
        SetHandler server-status
        Order deny,allow
        Deny from all
        Allow from localhost
        Allow from 98.xx.xx.x03
    </Location>
    

    Since apache and monit are running in the same server I thought it will work fine since I have localhost in this config.
    (98.xx.xx.x03 is the ip-address of our office)
    I guess I still needed to explicitly mention the same server address and I made it like this:

    <Location /server-status>
        SetHandler server-status
        Order deny,allow
        Deny from all
        Allow from localhost
        Allow from 198.xx.xx.xx8
        Allow from 98.xx.xx.x03
    </Location>
    

    And it worked:

    [MST Apr 11 11:53:10] debug    : Socket test failed for [198.xx.xx.xx8]:80 -- APACHE-STATUS: error -- 83 percent of processes are logging
    [MST Apr 11 11:53:10] error    : 'apache' failed protocol test [APACHESTATUS] at [phoenix.example.com]:80 [TCP/IP] -- APACHE-STATUS: error -- 83 percent of processes are logging
    [MST Apr 11 11:53:10] debug    : M/Monit: event message sent to http://[108.xx.xx.x94]:8080/collector
    

    Thanks a lot tildeslash. I think it looks good now !

  8. Log in to comment