Monit 5.22.0 Err#9 EBADF Solaris 10 with latest Patches

[Klaus Jürgens]

After applying latest solaris patches monit loops with Err#9 EBADF. A locally compiled version does the same.

root#sol-backup1:/opt/monit/monit-5.22.0/bin:>cat /etc/release 
                    Oracle Solaris 10 1/13 s10x_u11wos_24a X86
  Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.
                            Assembled 17 January 2013

root#sol-backup1:/opt/monit/monit-5.22.0/bin:>uname -a
SunOS sol-backup1 5.10 Generic_150401-50 i86pc i386 i86pc

root#sol-backup1:/opt/monit/monit-5.22.0/bin:>ldd ./monit 
        libm.so.2 =>     /lib/64/libm.so.2
        libpam.so.1 =>   /lib/64/libpam.so.1
        libkstat.so.1 =>         /lib/64/libkstat.so.1
        libnvpair.so.1 =>        /lib/64/libnvpair.so.1
        libzfs.so.2 =>   /lib/64/libzfs.so.2
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libresolv.so.2 =>        /lib/64/libresolv.so.2
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libcmd.so.1 =>   /lib/64/libcmd.so.1
        libdevinfo.so.1 =>       /lib/64/libdevinfo.so.1
        libdevid.so.1 =>         /lib/64/libdevid.so.1
        libgen.so.1 =>   /lib/64/libgen.so.1
        libuutil.so.1 =>         /lib/64/libuutil.so.1
        libavl.so.1 =>   /lib/64/libavl.so.1
        libefi.so.1 =>   /lib/64/libefi.so.1
        libadm.so.1 =>   /lib/64/libadm.so.1
        libmd.so.1 =>    /lib/64/libmd.so.1
        libumem.so.1 =>  /lib/64/libumem.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libscf.so.1 =>   /lib/64/libscf.so.1
        libsec.so.1 =>   /lib/64/libsec.so.1
        libuuid.so.1 =>  /lib/64/libuuid.so.1
        libdoor.so.1 =>  /lib/64/libdoor.so.1

root#sol-backup1:/opt/monit/monit-5.22.0/bin:>ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
open files                      (-n) unlimited
pipe size            (512 bytes, -p) 10
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 27589
virtual memory          (kbytes, -v) unlimited

[Tildeslash]

When monit starts it tries to close all filedescriptors except stdin, stdout and stderr up to maximum allowed number of filedescriptors. The limit is returned by getdtablesize() call - in your case ulimit reports "unlimited", as getdtablesize() return "signed int", it may be up to 2^31 (2 147 483 648), which will probably block monit in the close loop for some time.

Please can you try to set filedescriptors limit for monit to for example 1024? (monit needs usually just few filedescriptors).

[Klaus Jürgens]

Thank you for the explanation of monits behaviour. Setting the fd limit to 1024 is the solution.

root#sol-backup1:/opt/monit/monit-5.22.0:>ulimit -n 1024     
root#sol-backup1:/opt/monit/monit-5.22.0:>bin/monit -t  
Cannot find the Monit control file at ~/.monitrc, /etc/monitrc, /usr/local/etc/monitrc, /usr/local/etc/monitrc or at ./monitrc 

[Klaus Jürgens]

working solution

[Tildeslash]

Fixed: Issue #616: If no filedescriptors limit is set when Monit starts, Monit may freeze for a long time during startup and when executing start/stop methods or program checks.

→ <<cset 616cf523912f>>