fsflags monitoring does not detect if kernel remounted filesystem read-only.
Context: For example, on vmware environment, storage sometimes may get delayed and, because of timeouts, kernel interprets these delays as errors and remount filesystem read-only. Monit is unable to detect this condition with
changed fsflags monitoring. This is important condition for server health but monit is unexpectedly quiet about it.
It could be reproduced with
echo u > /proc/sysrq-trigger, after that observe that
monit status still shows
filesystem flags 0x1000. In the same time
/proc/mounts shows that filesystem is actually changed flags to
Example on Centos 7.3.1611 system:
root@test:/# echo u > /proc/sysrq-trigger -- dmesg -- [222022.193510] SysRq : Emergency Remount R/O [222022.274584] EXT4-fs (sdb): re-mounted. Opts: (null) [222022.567667] EXT4-fs (loop0): re-mounted. Opts: (null) [222022.567695] Emergency Remount complete root@test:/# grep -w ro /proc/mounts tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 /dev/mapper/centos_test2-root / xfs ro,relatime,attr2,inode64,noquota 0 0 /dev/sda1 /boot xfs ro,relatime,attr2,inode64,noquota 0 0 /dev/sdb /space ext4 ro,relatime 0 0 root@test:/# monit status|grep flags filesystem flags 0x1000 filesystem flags 0x1000
Some thoughts: Not necessarily
statvfs() syscall is working incorrectly, because,
f_flag value is defined as
/* mount flags */, i.e. it could be interpreted as mount time flags, and not as current filesystem state flags.
You state in documentation:
This test is useful for detecting changes of filesystem flags such as if the filesystem become read-only (on disk error) or mount flags were changed (such as nosuid).
"on disk error" is exactly incorrect statement.