unable to read filesystem 'XX' state / filesystem flags changed to ro,relatime
On two of our Ubuntus 20.04 (test/prod) we got repeatable (at least once per day, often two times) one of this alerts:
-
monit alert -- Filesystem flags changed XXX
- filesystem flags changed to ro,relatime
-
monit alert -- Does not exist XXX
- unable to read filesystem '/dev/XXX' state
Directly with the next check (2mins) this got resolved. And we got the corresponding message.
My monit configuration on both systems is:
check filesystem XXX with path /dev/XXX
if space usage > 80% for 5 times within 15 cycles then alert
if inode usage > 50% then alert
if read rate > 1 MB/s for 5 cycles then alert
if read rate > 500 operations/s for 5 cycles then alert
if write rate > 1 MB/s for 5 cycles then alert
if write rate > 500 operations/s for 5 cycles then alert
if service time > 10 milliseconds for 3 times within 5 cycles then alert
We already have this problem since 5.27.0. Also with 5.27.1. And still with 5.28.0.
We already checked with our server administrator, but it looks like a monitoring problem by monit, because everything else is working. And it’s resolved automatically directly by the next check.
Do you observed similar behaviors?
The problem is not major, but I try to teach our team to look at this alerts if I’m not available, and they are getting confused, if there were alerts.
Best, Michael
Comments (10)
-
repo owner -
reporter Hi.
Thanks a lot for your response.
Hm not sure how to check this. My server admin tolds me: we don’t use it (but also he doesn’t know about).
cat /etc/fstab:
UUID=d95ad723-e295-4141-b771-afbe7fe90682 / ext4 defaults 0 0 /swap.img none swap sw 0 0
mount -l | grep sda:
/dev/sda2 on / type ext4 (rw,relatime)
Do you have us a hint? :)
-
repo owner Please can you check the “mount” output when the problem occurs? You can modify monit configuration to execute a script on fsflags change, for example something like:
check filesystem XXX with path /dev/XXX if changed fsflags then exec "/bin/bash -c 'mount >>/tmp/mount.log 2>&1 && ls -l /dev/XXX >>/tmp/mount.log 2>&1'"
-
reporter Oh cool. Thanks!
Added this check and will write here if it happens again.
-
reporter Ok catched some. But it is looking always the same. (I also added the timestamp and the email text at the top of each file.)
-
tracking for changed fslags
- monit_mh_mount_fsflags.log
- monit_mh_dev_sda2_fsflags.log
-
tracking for sda2 not exists
- monit_mh_mount_not_exist.log
- monit_mh_dev_sda2_not_exist.log
monit_mh_mount_fsflags.log
# Thu, 27 May 2021 21:15:30: filesystem flags changed to ro,relatime sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=4031516k,nr_inodes=1007879,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755) /dev/sda2 on / type ext4 (rw,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=3423116k,nr_inodes=855779) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=855779) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=3423116k,nr_inodes=855779,mode=755) cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1988) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) /var/lib/snapd/snaps/core18_1997.snap on /snap/core18/1997 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/snapd_11588.snap on /snap/snapd/11588 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/lxd_20400.snap on /snap/lxd/20400 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/snapd_11841.snap on /snap/snapd/11841 type squashfs (ro,nodev,relatime,x-gdu.hide) tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755) nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw) overlay on /var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/U3PJWBS4IN7CHYZEDRKIELGOFH:/var/lib/docker/overlay2/l/MHTD4I4NG7GPBNLTS54TPCCJX4,upperdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/diff,workdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/work,xino=off) overlay on /var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/FCCHPMOLSWMBTCKJQSDXWTFK2E:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/diff,workdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/work,xino=off) overlay on /var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LIEICPXUFSLQXK4LZQJYCOEXRK:/var/lib/docker/overlay2/l/PN6XHPKERATKK7O3HKGEWAHECZ:/var/lib/docker/overlay2/l/B7CRPJGPZ36HJRWXQVF6B7BTF3:/var/lib/docker/overlay2/l/Z5SWZFFF5L32BTRYU2LHUJV5B7:/var/lib/docker/overlay2/l/FVGMU44VDQ5F5PMPWDHO4SQTCL:/var/lib/docker/overlay2/l/QODJR3S7U4TPYYHC5CZVOHXTS5:/var/lib/docker/overlay2/l/5MGTV44L72TRLAO6VGVHFJ35JD:/var/lib/docker/overlay2/l/ZVU7MSJJ3H56DJ44DAM442EPA7:/var/lib/docker/overlay2/l/2L7OB354Z6RT3BI5BZTR4EKKSW:/var/lib/docker/overlay2/l/BBHXUQROYCC66RAUU2JVXAKSUF,upperdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/diff,workdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/work,xino=off) overlay on /var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PNV4P2HJ75HNFXHTXLBY43WM4L:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/diff,workdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/work,xino=off) overlay on /var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QR5RHA62YCRIVSE2N44VWJRSVY:/var/lib/docker/overlay2/l/3IGCMBG3OJ557EJBCHB5327FZY:/var/lib/docker/overlay2/l/3AUA22SERVKSKUYOJBS4GUQPJY:/var/lib/docker/overlay2/l/PERPMOI3E3Y3YB3A6I75EX2QHD:/var/lib/docker/overlay2/l/XGG777UKIPVO7HV6MJCXP6A5HV:/var/lib/docker/overlay2/l/K5C2FIPFO36JDE5GIVVWUHNGZ7:/var/lib/docker/overlay2/l/IQKLLJ5PTM76BSI3UF3QR645LF:/var/lib/docker/overlay2/l/CKOFI42WRX5DKLMSMSJKDIUH34:/var/lib/docker/overlay2/l/FKZ5FMYZRNOLZFHZEZAEMC2RIH:/var/lib/docker/overlay2/l/Q3CD76A5FT2M5GB4OBOANSY3FV:/var/lib/docker/overlay2/l/O4NHZLUOKBFW36BLIX7IBENA2N:/var/lib/docker/overlay2/l/WUQLKUNMNHLY3VFQCGWK62QAUC:/var/lib/docker/overlay2/l/FDIDP5WWXVSKDFU3ETHI6FBUCN:/var/lib/docker/overlay2/l/A2OCVV423OVGTR72KMP3GEK3DR:/var/lib/docker/overlay2/l/LGZ3CSBU5Q5P5KZXKMAWQHOHBC,upperdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/diff,workdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/work,xino=off) overlay on /var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/M6LPMD7J7IZLHPLWY77GBQ3KJG:/var/lib/docker/overlay2/l/GZGBLTA4I4RKZIY2MNYZTB4L7N:/var/lib/docker/overlay2/l/DFY3KHMZRZQBRQASS25SDLUS43:/var/lib/docker/overlay2/l/KDTL5P2LOGF5TM7SGQJFKHJA47:/var/lib/docker/overlay2/l/PP3E5CYLUB7TWCMPM4YHC6ZVE5:/var/lib/docker/overlay2/l/PSCML74WYZBUMBDENHMVGZB632:/var/lib/docker/overlay2/l/ZMX5E2AWWLB6BUITOG37BBLSDR:/var/lib/docker/overlay2/l/ZCR6XOPFBO4EZVUFMCCZRIB7FG:/var/lib/docker/overlay2/l/MAFJSYYASUB7SE3IZK5BSIDYQQ:/var/lib/docker/overlay2/l/MDKA3RN666VTZ4DHRRZAXY7DCW:/var/lib/docker/overlay2/l/YVI7AZAEJWVM62KI7RATSZH4P7:/var/lib/docker/overlay2/l/6Q4GLO5UN2NG4KQZXGNARNFF22:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/diff,workdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/work,xino=off) nsfs on /run/docker/netns/b6fed2a811ac type nsfs (rw) nsfs on /run/docker/netns/106bef75f708 type nsfs (rw) nsfs on /run/docker/netns/22537a0e813d type nsfs (rw) nsfs on /run/docker/netns/e51f7dc5bdd3 type nsfs (rw) nsfs on /run/docker/netns/5b4f4345b1f4 type nsfs (rw) nsfs on /run/docker/netns/dbf87ce558a2 type nsfs (rw) /var/lib/snapd/snaps/lxd_20450.snap on /snap/lxd/20450 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/core18_2066.snap on /snap/core18/2066 type squashfs (ro,nodev,relatime,x-gdu.hide) tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=186036k,mode=700,uid=1001,gid=1001) # Thu, 27 May 2021 21:16:00: Description: filesystem flags changed to rw,relatime sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=4031516k,nr_inodes=1007879,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755) /dev/sda2 on / type ext4 (rw,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=3423116k,nr_inodes=855779) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=855779) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=3423116k,nr_inodes=855779,mode=755) cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1988) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) /var/lib/snapd/snaps/core18_1997.snap on /snap/core18/1997 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/snapd_11588.snap on /snap/snapd/11588 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/lxd_20400.snap on /snap/lxd/20400 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/snapd_11841.snap on /snap/snapd/11841 type squashfs (ro,nodev,relatime,x-gdu.hide) tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755) nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw) overlay on /var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/U3PJWBS4IN7CHYZEDRKIELGOFH:/var/lib/docker/overlay2/l/MHTD4I4NG7GPBNLTS54TPCCJX4,upperdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/diff,workdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/work,xino=off) overlay on /var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/FCCHPMOLSWMBTCKJQSDXWTFK2E:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/diff,workdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/work,xino=off) overlay on /var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LIEICPXUFSLQXK4LZQJYCOEXRK:/var/lib/docker/overlay2/l/PN6XHPKERATKK7O3HKGEWAHECZ:/var/lib/docker/overlay2/l/B7CRPJGPZ36HJRWXQVF6B7BTF3:/var/lib/docker/overlay2/l/Z5SWZFFF5L32BTRYU2LHUJV5B7:/var/lib/docker/overlay2/l/FVGMU44VDQ5F5PMPWDHO4SQTCL:/var/lib/docker/overlay2/l/QODJR3S7U4TPYYHC5CZVOHXTS5:/var/lib/docker/overlay2/l/5MGTV44L72TRLAO6VGVHFJ35JD:/var/lib/docker/overlay2/l/ZVU7MSJJ3H56DJ44DAM442EPA7:/var/lib/docker/overlay2/l/2L7OB354Z6RT3BI5BZTR4EKKSW:/var/lib/docker/overlay2/l/BBHXUQROYCC66RAUU2JVXAKSUF,upperdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/diff,workdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/work,xino=off) overlay on /var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PNV4P2HJ75HNFXHTXLBY43WM4L:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/diff,workdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/work,xino=off) overlay on /var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QR5RHA62YCRIVSE2N44VWJRSVY:/var/lib/docker/overlay2/l/3IGCMBG3OJ557EJBCHB5327FZY:/var/lib/docker/overlay2/l/3AUA22SERVKSKUYOJBS4GUQPJY:/var/lib/docker/overlay2/l/PERPMOI3E3Y3YB3A6I75EX2QHD:/var/lib/docker/overlay2/l/XGG777UKIPVO7HV6MJCXP6A5HV:/var/lib/docker/overlay2/l/K5C2FIPFO36JDE5GIVVWUHNGZ7:/var/lib/docker/overlay2/l/IQKLLJ5PTM76BSI3UF3QR645LF:/var/lib/docker/overlay2/l/CKOFI42WRX5DKLMSMSJKDIUH34:/var/lib/docker/overlay2/l/FKZ5FMYZRNOLZFHZEZAEMC2RIH:/var/lib/docker/overlay2/l/Q3CD76A5FT2M5GB4OBOANSY3FV:/var/lib/docker/overlay2/l/O4NHZLUOKBFW36BLIX7IBENA2N:/var/lib/docker/overlay2/l/WUQLKUNMNHLY3VFQCGWK62QAUC:/var/lib/docker/overlay2/l/FDIDP5WWXVSKDFU3ETHI6FBUCN:/var/lib/docker/overlay2/l/A2OCVV423OVGTR72KMP3GEK3DR:/var/lib/docker/overlay2/l/LGZ3CSBU5Q5P5KZXKMAWQHOHBC,upperdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/diff,workdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/work,xino=off) overlay on /var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/M6LPMD7J7IZLHPLWY77GBQ3KJG:/var/lib/docker/overlay2/l/GZGBLTA4I4RKZIY2MNYZTB4L7N:/var/lib/docker/overlay2/l/DFY3KHMZRZQBRQASS25SDLUS43:/var/lib/docker/overlay2/l/KDTL5P2LOGF5TM7SGQJFKHJA47:/var/lib/docker/overlay2/l/PP3E5CYLUB7TWCMPM4YHC6ZVE5:/var/lib/docker/overlay2/l/PSCML74WYZBUMBDENHMVGZB632:/var/lib/docker/overlay2/l/ZMX5E2AWWLB6BUITOG37BBLSDR:/var/lib/docker/overlay2/l/ZCR6XOPFBO4EZVUFMCCZRIB7FG:/var/lib/docker/overlay2/l/MAFJSYYASUB7SE3IZK5BSIDYQQ:/var/lib/docker/overlay2/l/MDKA3RN666VTZ4DHRRZAXY7DCW:/var/lib/docker/overlay2/l/YVI7AZAEJWVM62KI7RATSZH4P7:/var/lib/docker/overlay2/l/6Q4GLO5UN2NG4KQZXGNARNFF22:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/diff,workdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/work,xino=off) nsfs on /run/docker/netns/b6fed2a811ac type nsfs (rw) nsfs on /run/docker/netns/106bef75f708 type nsfs (rw) nsfs on /run/docker/netns/22537a0e813d type nsfs (rw) nsfs on /run/docker/netns/e51f7dc5bdd3 type nsfs (rw) nsfs on /run/docker/netns/5b4f4345b1f4 type nsfs (rw) nsfs on /run/docker/netns/dbf87ce558a2 type nsfs (rw) /var/lib/snapd/snaps/lxd_20450.snap on /snap/lxd/20450 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/core18_2066.snap on /snap/core18/2066 type squashfs (ro,nodev,relatime,x-gdu.hide) tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=186036k,mode=700,uid=1001,gid=1001)
monit_mh_dev_sda2_fsflags.log
brw-rw---- 1 root disk 8, 2 May 15 09:56 /dev/sda2 brw-rw---- 1 root disk 8, 2 May 15 09:56 /dev/sda2
monit_mh_mount_not_exist.log
# Thu, 27 May 2021 23:24:42: unable to read filesystem '/dev/sda2' state sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=4031516k,nr_inodes=1007879,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755) /dev/sda2 on / type ext4 (rw,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=3423116k,nr_inodes=855779) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=855779) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=3423116k,nr_inodes=855779,mode=755) cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1988) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) /var/lib/snapd/snaps/core18_1997.snap on /snap/core18/1997 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/snapd_11588.snap on /snap/snapd/11588 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/lxd_20400.snap on /snap/lxd/20400 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/snapd_11841.snap on /snap/snapd/11841 type squashfs (ro,nodev,relatime,x-gdu.hide) tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755) nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw) overlay on /var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/U3PJWBS4IN7CHYZEDRKIELGOFH:/var/lib/docker/overlay2/l/MHTD4I4NG7GPBNLTS54TPCCJX4,upperdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/diff,workdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/work,xino=off) overlay on /var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/FCCHPMOLSWMBTCKJQSDXWTFK2E:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/diff,workdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/work,xino=off) overlay on /var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LIEICPXUFSLQXK4LZQJYCOEXRK:/var/lib/docker/overlay2/l/PN6XHPKERATKK7O3HKGEWAHECZ:/var/lib/docker/overlay2/l/B7CRPJGPZ36HJRWXQVF6B7BTF3:/var/lib/docker/overlay2/l/Z5SWZFFF5L32BTRYU2LHUJV5B7:/var/lib/docker/overlay2/l/FVGMU44VDQ5F5PMPWDHO4SQTCL:/var/lib/docker/overlay2/l/QODJR3S7U4TPYYHC5CZVOHXTS5:/var/lib/docker/overlay2/l/5MGTV44L72TRLAO6VGVHFJ35JD:/var/lib/docker/overlay2/l/ZVU7MSJJ3H56DJ44DAM442EPA7:/var/lib/docker/overlay2/l/2L7OB354Z6RT3BI5BZTR4EKKSW:/var/lib/docker/overlay2/l/BBHXUQROYCC66RAUU2JVXAKSUF,upperdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/diff,workdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/work,xino=off) overlay on /var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PNV4P2HJ75HNFXHTXLBY43WM4L:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/diff,workdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/work,xino=off) overlay on /var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QR5RHA62YCRIVSE2N44VWJRSVY:/var/lib/docker/overlay2/l/3IGCMBG3OJ557EJBCHB5327FZY:/var/lib/docker/overlay2/l/3AUA22SERVKSKUYOJBS4GUQPJY:/var/lib/docker/overlay2/l/PERPMOI3E3Y3YB3A6I75EX2QHD:/var/lib/docker/overlay2/l/XGG777UKIPVO7HV6MJCXP6A5HV:/var/lib/docker/overlay2/l/K5C2FIPFO36JDE5GIVVWUHNGZ7:/var/lib/docker/overlay2/l/IQKLLJ5PTM76BSI3UF3QR645LF:/var/lib/docker/overlay2/l/CKOFI42WRX5DKLMSMSJKDIUH34:/var/lib/docker/overlay2/l/FKZ5FMYZRNOLZFHZEZAEMC2RIH:/var/lib/docker/overlay2/l/Q3CD76A5FT2M5GB4OBOANSY3FV:/var/lib/docker/overlay2/l/O4NHZLUOKBFW36BLIX7IBENA2N:/var/lib/docker/overlay2/l/WUQLKUNMNHLY3VFQCGWK62QAUC:/var/lib/docker/overlay2/l/FDIDP5WWXVSKDFU3ETHI6FBUCN:/var/lib/docker/overlay2/l/A2OCVV423OVGTR72KMP3GEK3DR:/var/lib/docker/overlay2/l/LGZ3CSBU5Q5P5KZXKMAWQHOHBC,upperdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/diff,workdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/work,xino=off) overlay on /var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/M6LPMD7J7IZLHPLWY77GBQ3KJG:/var/lib/docker/overlay2/l/GZGBLTA4I4RKZIY2MNYZTB4L7N:/var/lib/docker/overlay2/l/DFY3KHMZRZQBRQASS25SDLUS43:/var/lib/docker/overlay2/l/KDTL5P2LOGF5TM7SGQJFKHJA47:/var/lib/docker/overlay2/l/PP3E5CYLUB7TWCMPM4YHC6ZVE5:/var/lib/docker/overlay2/l/PSCML74WYZBUMBDENHMVGZB632:/var/lib/docker/overlay2/l/ZMX5E2AWWLB6BUITOG37BBLSDR:/var/lib/docker/overlay2/l/ZCR6XOPFBO4EZVUFMCCZRIB7FG:/var/lib/docker/overlay2/l/MAFJSYYASUB7SE3IZK5BSIDYQQ:/var/lib/docker/overlay2/l/MDKA3RN666VTZ4DHRRZAXY7DCW:/var/lib/docker/overlay2/l/YVI7AZAEJWVM62KI7RATSZH4P7:/var/lib/docker/overlay2/l/6Q4GLO5UN2NG4KQZXGNARNFF22:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/diff,workdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/work,xino=off) nsfs on /run/docker/netns/b6fed2a811ac type nsfs (rw) nsfs on /run/docker/netns/106bef75f708 type nsfs (rw) nsfs on /run/docker/netns/22537a0e813d type nsfs (rw) nsfs on /run/docker/netns/e51f7dc5bdd3 type nsfs (rw) nsfs on /run/docker/netns/5b4f4345b1f4 type nsfs (rw) nsfs on /run/docker/netns/dbf87ce558a2 type nsfs (rw) /var/lib/snapd/snaps/lxd_20450.snap on /snap/lxd/20450 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/core18_2066.snap on /snap/core18/2066 type squashfs (ro,nodev,relatime,x-gdu.hide) tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=186036k,mode=700,uid=1001,gid=1001)
monit_mh_dev_sda2_not_exist.log
brw-rw---- 1 root disk 8, 2 May 15 09:56 /dev/sda2
-
-
repo owner Thanks for data. The log doesn’t show any problem, even though there are some overlays, it is not related to the monitored filesystem.
We have added support for reporting the original filesystem flags to the development version, it may help to find the root cause. It seems to me the fsflags change is secondary problem, i think the main root cause is the “unable to read filesystem '/dev/XXX' state”. But that should be preceded by one of these messages:
- either “Filesystem 'XXX' not mounted”
- or “Cannot dereference filesystem 'XXX' (symlink) -- <error>
Please can you make sure the “set log syslog” (or “set log <file>”) statement is present in your monit configuration file and send us the monit related log part to support@mmonit.com?
-
Sure thanks a lot. I wrote you an email
-
repo owner Status update: it turned out, that the problem is caused by a temporary docker mount, which mounts the /dev/sda2 (that is already mounted to “/”) to a temporary mountpoint. Symptoms:
monit[3513479]: Mount table change detected monit[3513479]: Reloading mount information for filesystem '/dev/sda2' monit[3513479]: device /dev/sda2 matches filesystem /dev/sda2 (mounted on /) monit[3513479]: device /dev/sda2 matches filesystem /dev/sda2 (mounted on /run/docker/runtime-runc/moby/b12feed0387afb581c5a8c281da2d1c9d74c6cb8c21f10363e8748def9fab2f1/runc.EDzC9W) monit[3513479]: Error getting usage statistics for filesystem '/run/docker/runtime-runc/moby/b12feed0387afb581c5a8c281da2d1c9d74c6cb8c21f10363e8748def9fab2f1/runc.EDzC9W' -- No such file or directory monit[3513479]: Filesystem '/dev/sda2' not mounted monit[3513479]: 'sda2' unable to read filesystem '/dev/sda2' state
That rw+ro multipoint mount looks dangerous (cache coherency needs to be maintained), not sure if ext4 (and similar filesystems) can handle such use case.
Workaround: use the specific mountpoint in the monit test rather then device:
check filesystem XXX with path /
We don’t plan to fix the filesystem scan algorithm right now, due to doubts about the rw+ro mesh sanity. We have only improved the debug mode to allow to discover the root cause of the problem (snip above).
We may need to revisit this problem in the future, if it’ll turn out that such multipoint mount usage is correct.
-
repo owner - changed status to resolved
workaround provided (the test needs to be specific)
-
repo owner yet note:
If the same filesystem is mounted to two different mount points where at least one of them is rw, the filesystem would need to keep cache coherency. That may work for cluster filesystems that are designed for such purpose, but it is very risky for filesystems like ext4: the read-only mount may fail to read data if the block was updated via the first rw mount. It may work fine if all mounts of this filesystem are read-only.
It seems that docker does this rw+ro mesh mount for very short time frame (which is when monit sporadically catches the problem), but i think it is kind of Russian roulette.
- Log in to comment
Hello Michael, do you use overlay mounts for the same filesystem on your system? If there are overlays, Monit will evaluate only the last fstab entry for the given filesystem. The overlay mount may have different mount flags … if it is just temporary, it may explain the problem.