fsflags monitoring does not detect when fs became read-only on Linux

Issue #560 resolved
Don
created an issue

On linux fsflags monitoring does not detect if kernel remounted filesystem read-only.

Context: For example, on vmware environment, storage sometimes may get delayed and, because of timeouts, kernel interprets these delays as errors and remount filesystem read-only. Monit is unable to detect this condition with changed fsflags monitoring. This is important condition for server health but monit is unexpectedly quiet about it.

It could be reproduced with echo u > /proc/sysrq-trigger, after that observe that monit status still shows filesystem flags 0x1000. In the same time /proc/mounts shows that filesystem is actually changed flags to ro.

Example on Centos 7.3.1611 system:

root@test:/# echo u > /proc/sysrq-trigger
-- dmesg --
[222022.193510] SysRq : Emergency Remount R/O
[222022.274584] EXT4-fs (sdb): re-mounted. Opts: (null)
[222022.567667] EXT4-fs (loop0): re-mounted. Opts: (null)
[222022.567695] Emergency Remount complete

root@test:/# grep -w ro /proc/mounts
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
/dev/mapper/centos_test2-root / xfs ro,relatime,attr2,inode64,noquota 0 0
/dev/sda1 /boot xfs ro,relatime,attr2,inode64,noquota 0 0
/dev/sdb /space ext4 ro,relatime 0 0

root@test:/# monit status|grep flags
  filesystem flags                  0x1000
  filesystem flags                  0x1000

Some thoughts: Not necessarily statvfs() syscall is working incorrectly, because, f_flag value is defined as /* mount flags */, i.e. it could be interpreted as mount time flags, and not as current filesystem state flags.

You state in documentation:

This test is useful for detecting changes of filesystem flags such as if the filesystem become read-only (on disk error) or mount flags were changed (such as nosuid).

"on disk error" is exactly incorrect statement.

Comments (4)

  1. Tildeslash repo owner

    Thank you for report, will try to reproduce. We are testing the mount flags test using manual remount:

    mount -o remount,ro /myfilesystem
    

    Monit detects mount flags change in such case without problem.

  2. Don reporter

    Monit detects mount flags change in such case without problem.

    I know. Currently, if filesystem remounts ro (by decision of kernel, such as "on disk error"), monit does not detect it. Well, I wish monit would detect that, as this is exact purpose of monitoring to detect such an event.

  3. Tildeslash repo owner

    Fixed: Issue #560: Monit may miss the filesystem flags change if it was triggered by kernel (TODO: LINUX only in this checkin, update all other platforms)

    New: Show textual representation of filesystem flags instead of numeric.

    → <<cset cc9450e10ac5>>

  4. Log in to comment