monit fails to identify filesystem stat file

[Former user]

Since updating to 5.25.1 from 5.14 on CentOS7, I am observing following error:

# monit -t
filesystem statistic error: cannot read /sys/class/block/rootfs/stat -- No such file or directory
Control file syntax OK

The cause is this config directive:

check filesystem rootfs with path /
    if space free < 50 GB for 10 cycles then alert

If path "/" is replaced with device name, the error goes away.

# grep " / " /etc/mtab
rootfs / rootfs rw 0 0
/dev/mapper/vg1-root / ext4 rw,seclabel,noatime,discard,data=ordered 0 0

Maybe the following change is the root cause:

src/device/sysdep_LINUX.c
-        return IS(mountpoint, mnt->mnt_dir) && ! IS(mnt->mnt_fsname, "rootfs");
+        return IS(mountpoint, mnt->mnt_dir);

https://bitbucket.org/tildeslash/monit/commits/b4f1dd2b8530

# uname -a
Linux xxx.xxx.xxx 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# rpm -q monit
monit-5.25.1-1.el7.x86_64

[Tildeslash]

I cannot reproduce the issue (CentOS Linux release 7.4.1708) with the current monit version - my filesystems setup is similar (using XFS instead of EXT4):

$ grep " / " /etc/mtab
rootfs / rootfs rw 0 0
/dev/mapper/centos-root / xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0

The code has little bit changed since https://bitbucket.org/tildeslash/monit/commits/b4f1dd2b8530 ... it basically scans the whole mount table to support overlay mounts like for rootfs and the exception is no longer necessary.

Is it really monit 5.25.1? Please can you verify the version?:

monit -V

If it is monit 5.25.1, please can you run it in debug mode and send output?:

monit -vI

[Ivan Pesin]

Hi, I am seeing the same issue:

# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

# monit -V
This is Monit version 5.25.1
Built with ssl, with ipv6, with compression, with pam and with large files
Copyright (C) 2001-2017 Tildeslash Ltd. All Rights Reserved.

# grep " / " /etc/mtab
rootfs / rootfs rw 0 0
/dev/vda3 / xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0

# monit -vI
Adding 'allow localhost' -- host resolved to [::1]
Adding 'allow localhost' -- host resolved to [::ffff:127.0.0.1]
Adding credentials for user 'admin'
Reloading mount information for filesystem '/'
filesystem statistic error: cannot read /sys/class/block/rootfs/stat -- No such file or directory
Runtime constants:
 Control file       = /etc/monitrc
 Log file           = /var/log/monit.log
 Pid file           = /run/monit.pid
 Id file            = /root/.monit.id
 State file         = /root/.monit.state
 Debug              = True
 Log                = True
 Use syslog         = False
 Is Daemon          = True
 Use process engine = True
 Limits             = {
                    =   programOutput:     512 B                                                                                                                                                             [4/444]
                    =   sendExpectBuffer:  256 B
                    =   fileContentBuffer: 512 B
                    =   httpContentBuffer: 1 MB
                    =   networkTimeout:    5 s
                    =   programTimeout:    5 m
                    =   stopTimeout:       30 s
                    =   startTimeout:      30 s
                    =   restartTimeout:    30 s
                    = }
 On reboot          = start
 Poll time          = 30 seconds with start delay 0 seconds
 Start monit httpd  = True
 httpd bind address = localhost
 httpd portnumber   = 2812
 httpd signature    = Enabled
 httpd auth. style  = Basic Authentication and Host/Net allow list

The service list contains the following entries:

Filesystem Name       = rootfs
 Path                 = /
 Monitoring mode      = active
 On reboot            = start
 Existence            = if does not exist then restart
 Filesystem flags     = if changed then alert
 Space free limit     = if less than 50 GB for 10 cycles then alert

System Name           = bastion.devenv
 Monitoring mode      = active
 On reboot            = start

-------------------------------------------------------------------------------
Monit daemon with PID 5318 awakened

I have more than a dozen machines that exhibit this behaviour, both VMs and bare-metal. All machines are centos7.

Is there anything I can do to help debug the reason why this is happening?

[Lutz Mader]

Hello,
I find similar Errors in the monit.log on an Linux system, but sometimes only.

[CET Mar 19 15:03:27] info     : Reinitializing monit daemon
[CET Mar 19 15:03:27] info     : Reinitializing Monit -- control file '/usd/as12345a/home/was12345/.monitrc'
[CET Mar 19 15:03:27] info     : 'D100ABCD0123' Monit reloaded
[CET Mar 19 15:03:27] error    : filesystem statistic error: cannot read /sys/class/block/as12345avol/stat -- No such file or directory
[CET Mar 19 15:47:33] error    : filesystem statistic error: cannot read /sys/class/block/as12345avol/stat -- No such file or directory
[CET Mar 19 15:53:33] error    : filesystem statistic error: cannot read /sys/class/block/as12345avol/stat -- No such file or directory
[CET Mar 19 18:47:27] info     : Reinitializing monit daemon

The errors depend on the following check.

check filesystem User_Fs with path "/opt/as12345a"
  if space usage > 80% for 15 cycles then exec "/opt/as12345a/monit/scripts/wlpdump.sh"
     repeat every 60 cycles

For a user application filesystem.

df -m .
Filesystem                        1M-blocks  Used Available Use% Mounted on
/dev/vx/dsk/as12345dg/as12345avol    204759 17793    175284  10% /opt/as12345a

/etc/fstab
/dev/vx/dsk/as12345dg/as12345avol /usd/as12345a vxfs noauto,_netdev 0 0

My Problem of understandig is, monit check the filesystem every cycle (every minute), but I find the error sometimes one, three or five times a day. And sometimes I can’t find the error the whole week.

For your Information only,
Lutz

[Christian Hack]

I also have this issue running 5.25.3

mount | grep opt
/dev/mapper/optfs on /opt type ext4 (rw,noatime,data=ordered)

df -m .
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/optfs 6764 200 6479 3% /opt

config looks like

check device opt with path /opt
    if SPACE free < 20% then alert

But is the same with /dev/mapper/optfs.

monit -V
This is Monit version 5.25.3
Built with ssl, without ipv6, with compression, without pam and with large files
Copyright (C) 2001-2019 Tildeslash Ltd. All Rights Reserved.

Error looks like

monit[8704]: filesystem statistic error: cannot read /sys/class/block/optfs/stat -- No such file or directory

I’m running a Yocto build with kernel 4.14

[Tildeslash]

Duplicate of #937.