fsflags monitoring does not detect when fs became read-only on Linux
On linux fsflags
monitoring does not detect if kernel remounted filesystem read-only.
Context: For example, on vmware environment, storage sometimes may get delayed and, because of timeouts, kernel interprets these delays as errors and remount filesystem read-only. Monit is unable to detect this condition with changed fsflags
monitoring. This is important condition for server health but monit is unexpectedly quiet about it.
It could be reproduced with echo u > /proc/sysrq-trigger
, after that observe that monit status
still shows filesystem flags 0x1000
. In the same time /proc/mounts
shows that filesystem is actually changed flags to ro
.
Example on Centos 7.3.1611 system:
root@test:/# echo u > /proc/sysrq-trigger -- dmesg -- [222022.193510] SysRq : Emergency Remount R/O [222022.274584] EXT4-fs (sdb): re-mounted. Opts: (null) [222022.567667] EXT4-fs (loop0): re-mounted. Opts: (null) [222022.567695] Emergency Remount complete root@test:/# grep -w ro /proc/mounts tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 /dev/mapper/centos_test2-root / xfs ro,relatime,attr2,inode64,noquota 0 0 /dev/sda1 /boot xfs ro,relatime,attr2,inode64,noquota 0 0 /dev/sdb /space ext4 ro,relatime 0 0 root@test:/# monit status|grep flags filesystem flags 0x1000 filesystem flags 0x1000
Some thoughts: Not necessarily statvfs()
syscall is working incorrectly, because, f_flag
value is defined as /* mount flags */
, i.e. it could be interpreted as mount time flags, and not as current filesystem state flags.
You state in documentation:
This test is useful for detecting changes of filesystem flags such as if the filesystem become read-only (on disk error) or mount flags were changed (such as nosuid).
"on disk error" is exactly incorrect statement.
Comments (4)
-
repo owner -
reporter Monit detects mount flags change in such case without problem.
I know. Currently, if filesystem remounts ro (by decision of kernel, such as "on disk error"), monit does not detect it. Well, I wish monit would detect that, as this is exact purpose of monitoring to detect such an event.
-
repo owner - changed status to resolved
Fixed: Issue
#560: Monit may miss the filesystem flags change if it was triggered by kernel (TODO: LINUX only in this checkin, update all other platforms)New: Show textual representation of filesystem flags instead of numeric.
→ <<cset cc9450e10ac5>>
-
reporter Thanks for quick fix!
- Log in to comment
Thank you for report, will try to reproduce. We are testing the mount flags test using manual remount:
mount -o remount,ro /myfilesystem
Monit detects mount flags change in such case without problem.