unable to read filesystem 'XX' state / filesystem flags changed to ro,relatime

Issue #983 resolved
Michael Hadorn created an issue

On two of our Ubuntus 20.04 (test/prod) we got repeatable (at least once per day, often two times) one of this alerts:

  • monit alert -- Filesystem flags changed XXX

    • filesystem flags changed to ro,relatime
  • monit alert -- Does not exist XXX

    • unable to read filesystem '/dev/XXX' state

Directly with the next check (2mins) this got resolved. And we got the corresponding message.

My monit configuration on both systems is:

check filesystem XXX with path /dev/XXX
   if space usage > 80% for 5 times within 15 cycles then alert
   if inode usage > 50% then alert
   if read rate > 1 MB/s for 5 cycles then alert
   if read rate > 500 operations/s for 5 cycles then alert
   if write rate > 1 MB/s for 5 cycles then alert
   if write rate > 500 operations/s for 5 cycles then alert
   if service time > 10 milliseconds for 3 times within 5 cycles then alert

We already have this problem since 5.27.0. Also with 5.27.1. And still with 5.28.0.

We already checked with our server administrator, but it looks like a monitoring problem by monit, because everything else is working. And it’s resolved automatically directly by the next check.

Do you observed similar behaviors?

The problem is not major, but I try to teach our team to look at this alerts if I’m not available, and they are getting confused, if there were alerts.

Best, Michael

Comments (10)

  1. Tildeslash repo owner

    Hello Michael, do you use overlay mounts for the same filesystem on your system? If there are overlays, Monit will evaluate only the last fstab entry for the given filesystem. The overlay mount may have different mount flags … if it is just temporary, it may explain the problem.

  2. Michael Hadorn reporter

    Hi.

    Thanks a lot for your response.

    Hm not sure how to check this. My server admin tolds me: we don’t use it (but also he doesn’t know about).

    cat /etc/fstab:

    UUID=d95ad723-e295-4141-b771-afbe7fe90682 / ext4 defaults 0 0
    /swap.img       none    swap    sw      0       0
    

    mount -l | grep sda:

    /dev/sda2 on / type ext4 (rw,relatime)
    

    Do you have us a hint? :)

  3. Tildeslash repo owner

    Please can you check the “mount” output when the problem occurs? You can modify monit configuration to execute a script on fsflags change, for example something like:

    check filesystem XXX with path /dev/XXX
        if changed fsflags then exec "/bin/bash -c 'mount >>/tmp/mount.log 2>&1 && ls -l /dev/XXX >>/tmp/mount.log 2>&1'"
    
  4. Michael Hadorn reporter

    Ok catched some. But it is looking always the same. (I also added the timestamp and the email text at the top of each file.)

    • tracking for changed fslags

      • monit_mh_mount_fsflags.log
      • monit_mh_dev_sda2_fsflags.log
    • tracking for sda2 not exists

      • monit_mh_mount_not_exist.log
      • monit_mh_dev_sda2_not_exist.log

    monit_mh_mount_fsflags.log

    # Thu, 27 May 2021 21:15:30: filesystem flags changed to ro,relatime
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=4031516k,nr_inodes=1007879,mode=755)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755)
    /dev/sda2 on / type ext4 (rw,relatime)
    securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=3423116k,nr_inodes=855779)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=855779)
    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=3423116k,nr_inodes=855779,mode=755)
    cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
    cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
    pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
    none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
    cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
    cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
    cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
    cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
    cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
    cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
    cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
    cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
    cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
    cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
    systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1988)
    hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
    mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
    debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
    tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
    fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
    configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
    /var/lib/snapd/snaps/core18_1997.snap on /snap/core18/1997 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/snapd_11588.snap on /snap/snapd/11588 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/lxd_20400.snap on /snap/lxd/20400 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/snapd_11841.snap on /snap/snapd/11841 type squashfs (ro,nodev,relatime,x-gdu.hide)
    tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755)
    nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
    overlay on /var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/U3PJWBS4IN7CHYZEDRKIELGOFH:/var/lib/docker/overlay2/l/MHTD4I4NG7GPBNLTS54TPCCJX4,upperdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/diff,workdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/work,xino=off)
    overlay on /var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/FCCHPMOLSWMBTCKJQSDXWTFK2E:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/diff,workdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/work,xino=off)
    overlay on /var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LIEICPXUFSLQXK4LZQJYCOEXRK:/var/lib/docker/overlay2/l/PN6XHPKERATKK7O3HKGEWAHECZ:/var/lib/docker/overlay2/l/B7CRPJGPZ36HJRWXQVF6B7BTF3:/var/lib/docker/overlay2/l/Z5SWZFFF5L32BTRYU2LHUJV5B7:/var/lib/docker/overlay2/l/FVGMU44VDQ5F5PMPWDHO4SQTCL:/var/lib/docker/overlay2/l/QODJR3S7U4TPYYHC5CZVOHXTS5:/var/lib/docker/overlay2/l/5MGTV44L72TRLAO6VGVHFJ35JD:/var/lib/docker/overlay2/l/ZVU7MSJJ3H56DJ44DAM442EPA7:/var/lib/docker/overlay2/l/2L7OB354Z6RT3BI5BZTR4EKKSW:/var/lib/docker/overlay2/l/BBHXUQROYCC66RAUU2JVXAKSUF,upperdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/diff,workdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/work,xino=off)
    overlay on /var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PNV4P2HJ75HNFXHTXLBY43WM4L:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/diff,workdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/work,xino=off)
    overlay on /var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QR5RHA62YCRIVSE2N44VWJRSVY:/var/lib/docker/overlay2/l/3IGCMBG3OJ557EJBCHB5327FZY:/var/lib/docker/overlay2/l/3AUA22SERVKSKUYOJBS4GUQPJY:/var/lib/docker/overlay2/l/PERPMOI3E3Y3YB3A6I75EX2QHD:/var/lib/docker/overlay2/l/XGG777UKIPVO7HV6MJCXP6A5HV:/var/lib/docker/overlay2/l/K5C2FIPFO36JDE5GIVVWUHNGZ7:/var/lib/docker/overlay2/l/IQKLLJ5PTM76BSI3UF3QR645LF:/var/lib/docker/overlay2/l/CKOFI42WRX5DKLMSMSJKDIUH34:/var/lib/docker/overlay2/l/FKZ5FMYZRNOLZFHZEZAEMC2RIH:/var/lib/docker/overlay2/l/Q3CD76A5FT2M5GB4OBOANSY3FV:/var/lib/docker/overlay2/l/O4NHZLUOKBFW36BLIX7IBENA2N:/var/lib/docker/overlay2/l/WUQLKUNMNHLY3VFQCGWK62QAUC:/var/lib/docker/overlay2/l/FDIDP5WWXVSKDFU3ETHI6FBUCN:/var/lib/docker/overlay2/l/A2OCVV423OVGTR72KMP3GEK3DR:/var/lib/docker/overlay2/l/LGZ3CSBU5Q5P5KZXKMAWQHOHBC,upperdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/diff,workdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/work,xino=off)
    overlay on /var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/M6LPMD7J7IZLHPLWY77GBQ3KJG:/var/lib/docker/overlay2/l/GZGBLTA4I4RKZIY2MNYZTB4L7N:/var/lib/docker/overlay2/l/DFY3KHMZRZQBRQASS25SDLUS43:/var/lib/docker/overlay2/l/KDTL5P2LOGF5TM7SGQJFKHJA47:/var/lib/docker/overlay2/l/PP3E5CYLUB7TWCMPM4YHC6ZVE5:/var/lib/docker/overlay2/l/PSCML74WYZBUMBDENHMVGZB632:/var/lib/docker/overlay2/l/ZMX5E2AWWLB6BUITOG37BBLSDR:/var/lib/docker/overlay2/l/ZCR6XOPFBO4EZVUFMCCZRIB7FG:/var/lib/docker/overlay2/l/MAFJSYYASUB7SE3IZK5BSIDYQQ:/var/lib/docker/overlay2/l/MDKA3RN666VTZ4DHRRZAXY7DCW:/var/lib/docker/overlay2/l/YVI7AZAEJWVM62KI7RATSZH4P7:/var/lib/docker/overlay2/l/6Q4GLO5UN2NG4KQZXGNARNFF22:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/diff,workdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/work,xino=off)
    nsfs on /run/docker/netns/b6fed2a811ac type nsfs (rw)
    nsfs on /run/docker/netns/106bef75f708 type nsfs (rw)
    nsfs on /run/docker/netns/22537a0e813d type nsfs (rw)
    nsfs on /run/docker/netns/e51f7dc5bdd3 type nsfs (rw)
    nsfs on /run/docker/netns/5b4f4345b1f4 type nsfs (rw)
    nsfs on /run/docker/netns/dbf87ce558a2 type nsfs (rw)
    /var/lib/snapd/snaps/lxd_20450.snap on /snap/lxd/20450 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/core18_2066.snap on /snap/core18/2066 type squashfs (ro,nodev,relatime,x-gdu.hide)
    tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=186036k,mode=700,uid=1001,gid=1001)
    
    
    # Thu, 27 May 2021 21:16:00: Description: filesystem flags changed to rw,relatime
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=4031516k,nr_inodes=1007879,mode=755)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755)
    /dev/sda2 on / type ext4 (rw,relatime)
    securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=3423116k,nr_inodes=855779)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=855779)
    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=3423116k,nr_inodes=855779,mode=755)
    cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
    cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
    pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
    none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
    cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
    cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
    cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
    cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
    cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
    cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
    cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
    cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
    cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
    cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
    systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1988)
    hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
    mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
    debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
    tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
    fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
    configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
    /var/lib/snapd/snaps/core18_1997.snap on /snap/core18/1997 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/snapd_11588.snap on /snap/snapd/11588 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/lxd_20400.snap on /snap/lxd/20400 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/snapd_11841.snap on /snap/snapd/11841 type squashfs (ro,nodev,relatime,x-gdu.hide)
    tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755)
    nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
    overlay on /var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/U3PJWBS4IN7CHYZEDRKIELGOFH:/var/lib/docker/overlay2/l/MHTD4I4NG7GPBNLTS54TPCCJX4,upperdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/diff,workdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/work,xino=off)
    overlay on /var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/FCCHPMOLSWMBTCKJQSDXWTFK2E:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/diff,workdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/work,xino=off)
    overlay on /var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LIEICPXUFSLQXK4LZQJYCOEXRK:/var/lib/docker/overlay2/l/PN6XHPKERATKK7O3HKGEWAHECZ:/var/lib/docker/overlay2/l/B7CRPJGPZ36HJRWXQVF6B7BTF3:/var/lib/docker/overlay2/l/Z5SWZFFF5L32BTRYU2LHUJV5B7:/var/lib/docker/overlay2/l/FVGMU44VDQ5F5PMPWDHO4SQTCL:/var/lib/docker/overlay2/l/QODJR3S7U4TPYYHC5CZVOHXTS5:/var/lib/docker/overlay2/l/5MGTV44L72TRLAO6VGVHFJ35JD:/var/lib/docker/overlay2/l/ZVU7MSJJ3H56DJ44DAM442EPA7:/var/lib/docker/overlay2/l/2L7OB354Z6RT3BI5BZTR4EKKSW:/var/lib/docker/overlay2/l/BBHXUQROYCC66RAUU2JVXAKSUF,upperdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/diff,workdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/work,xino=off)
    overlay on /var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PNV4P2HJ75HNFXHTXLBY43WM4L:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/diff,workdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/work,xino=off)
    overlay on /var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QR5RHA62YCRIVSE2N44VWJRSVY:/var/lib/docker/overlay2/l/3IGCMBG3OJ557EJBCHB5327FZY:/var/lib/docker/overlay2/l/3AUA22SERVKSKUYOJBS4GUQPJY:/var/lib/docker/overlay2/l/PERPMOI3E3Y3YB3A6I75EX2QHD:/var/lib/docker/overlay2/l/XGG777UKIPVO7HV6MJCXP6A5HV:/var/lib/docker/overlay2/l/K5C2FIPFO36JDE5GIVVWUHNGZ7:/var/lib/docker/overlay2/l/IQKLLJ5PTM76BSI3UF3QR645LF:/var/lib/docker/overlay2/l/CKOFI42WRX5DKLMSMSJKDIUH34:/var/lib/docker/overlay2/l/FKZ5FMYZRNOLZFHZEZAEMC2RIH:/var/lib/docker/overlay2/l/Q3CD76A5FT2M5GB4OBOANSY3FV:/var/lib/docker/overlay2/l/O4NHZLUOKBFW36BLIX7IBENA2N:/var/lib/docker/overlay2/l/WUQLKUNMNHLY3VFQCGWK62QAUC:/var/lib/docker/overlay2/l/FDIDP5WWXVSKDFU3ETHI6FBUCN:/var/lib/docker/overlay2/l/A2OCVV423OVGTR72KMP3GEK3DR:/var/lib/docker/overlay2/l/LGZ3CSBU5Q5P5KZXKMAWQHOHBC,upperdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/diff,workdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/work,xino=off)
    overlay on /var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/M6LPMD7J7IZLHPLWY77GBQ3KJG:/var/lib/docker/overlay2/l/GZGBLTA4I4RKZIY2MNYZTB4L7N:/var/lib/docker/overlay2/l/DFY3KHMZRZQBRQASS25SDLUS43:/var/lib/docker/overlay2/l/KDTL5P2LOGF5TM7SGQJFKHJA47:/var/lib/docker/overlay2/l/PP3E5CYLUB7TWCMPM4YHC6ZVE5:/var/lib/docker/overlay2/l/PSCML74WYZBUMBDENHMVGZB632:/var/lib/docker/overlay2/l/ZMX5E2AWWLB6BUITOG37BBLSDR:/var/lib/docker/overlay2/l/ZCR6XOPFBO4EZVUFMCCZRIB7FG:/var/lib/docker/overlay2/l/MAFJSYYASUB7SE3IZK5BSIDYQQ:/var/lib/docker/overlay2/l/MDKA3RN666VTZ4DHRRZAXY7DCW:/var/lib/docker/overlay2/l/YVI7AZAEJWVM62KI7RATSZH4P7:/var/lib/docker/overlay2/l/6Q4GLO5UN2NG4KQZXGNARNFF22:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/diff,workdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/work,xino=off)
    nsfs on /run/docker/netns/b6fed2a811ac type nsfs (rw)
    nsfs on /run/docker/netns/106bef75f708 type nsfs (rw)
    nsfs on /run/docker/netns/22537a0e813d type nsfs (rw)
    nsfs on /run/docker/netns/e51f7dc5bdd3 type nsfs (rw)
    nsfs on /run/docker/netns/5b4f4345b1f4 type nsfs (rw)
    nsfs on /run/docker/netns/dbf87ce558a2 type nsfs (rw)
    /var/lib/snapd/snaps/lxd_20450.snap on /snap/lxd/20450 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/core18_2066.snap on /snap/core18/2066 type squashfs (ro,nodev,relatime,x-gdu.hide)
    tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=186036k,mode=700,uid=1001,gid=1001)
    

    monit_mh_dev_sda2_fsflags.log

    brw-rw---- 1 root disk 8, 2 May 15 09:56 /dev/sda2
    brw-rw---- 1 root disk 8, 2 May 15 09:56 /dev/sda2
    

    monit_mh_mount_not_exist.log

    # Thu, 27 May 2021 23:24:42: unable to read filesystem '/dev/sda2' state
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=4031516k,nr_inodes=1007879,mode=755)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755)
    /dev/sda2 on / type ext4 (rw,relatime)
    securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=3423116k,nr_inodes=855779)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=855779)
    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=3423116k,nr_inodes=855779,mode=755)
    cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
    cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
    pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
    none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
    cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
    cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
    cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
    cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
    cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
    cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
    cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
    cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
    cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
    cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
    systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1988)
    hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
    mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
    debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
    tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
    fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
    configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
    /var/lib/snapd/snaps/core18_1997.snap on /snap/core18/1997 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/snapd_11588.snap on /snap/snapd/11588 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/lxd_20400.snap on /snap/lxd/20400 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/snapd_11841.snap on /snap/snapd/11841 type squashfs (ro,nodev,relatime,x-gdu.hide)
    tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=776580k,nr_inodes=970723,mode=755)
    nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
    overlay on /var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/U3PJWBS4IN7CHYZEDRKIELGOFH:/var/lib/docker/overlay2/l/MHTD4I4NG7GPBNLTS54TPCCJX4,upperdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/diff,workdir=/var/lib/docker/overlay2/8f0a388a25484015bd0fe51b5cd97c7e827d46e083cac3afbbd5273e25cd9ec7/work,xino=off)
    overlay on /var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/FCCHPMOLSWMBTCKJQSDXWTFK2E:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/diff,workdir=/var/lib/docker/overlay2/47bd1da99ddee4d06627c377fb7a6940977d295be239f27ef6ac727214b41cad/work,xino=off)
    overlay on /var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LIEICPXUFSLQXK4LZQJYCOEXRK:/var/lib/docker/overlay2/l/PN6XHPKERATKK7O3HKGEWAHECZ:/var/lib/docker/overlay2/l/B7CRPJGPZ36HJRWXQVF6B7BTF3:/var/lib/docker/overlay2/l/Z5SWZFFF5L32BTRYU2LHUJV5B7:/var/lib/docker/overlay2/l/FVGMU44VDQ5F5PMPWDHO4SQTCL:/var/lib/docker/overlay2/l/QODJR3S7U4TPYYHC5CZVOHXTS5:/var/lib/docker/overlay2/l/5MGTV44L72TRLAO6VGVHFJ35JD:/var/lib/docker/overlay2/l/ZVU7MSJJ3H56DJ44DAM442EPA7:/var/lib/docker/overlay2/l/2L7OB354Z6RT3BI5BZTR4EKKSW:/var/lib/docker/overlay2/l/BBHXUQROYCC66RAUU2JVXAKSUF,upperdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/diff,workdir=/var/lib/docker/overlay2/6caf74bb055e3a223024d3a1c614fc6888a826ad09488e5bd0ba1fbfcc79a9cb/work,xino=off)
    overlay on /var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PNV4P2HJ75HNFXHTXLBY43WM4L:/var/lib/docker/overlay2/l/SPXBNWIWELWYRKLB7FRPCVG7XQ:/var/lib/docker/overlay2/l/3PUTCAFDKUY4QUJEMAIS4KTPBX:/var/lib/docker/overlay2/l/Z2UAFKUUN6DEZKGY42LTEHN2YQ:/var/lib/docker/overlay2/l/SPKQFW6ITAKO554XLNZY6VR4DE:/var/lib/docker/overlay2/l/REXCLJRM6WK2I3GKBNUSXB6J7L:/var/lib/docker/overlay2/l/6DLEAH6KZVZN5Q252ZG4JFIVWW:/var/lib/docker/overlay2/l/DEVCYPUOSECCTECS7YQIWFYXOY:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/diff,workdir=/var/lib/docker/overlay2/755aa8446fbbe68a443f00cae3077e16954227cc9201001571704936aa706180/work,xino=off)
    overlay on /var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QR5RHA62YCRIVSE2N44VWJRSVY:/var/lib/docker/overlay2/l/3IGCMBG3OJ557EJBCHB5327FZY:/var/lib/docker/overlay2/l/3AUA22SERVKSKUYOJBS4GUQPJY:/var/lib/docker/overlay2/l/PERPMOI3E3Y3YB3A6I75EX2QHD:/var/lib/docker/overlay2/l/XGG777UKIPVO7HV6MJCXP6A5HV:/var/lib/docker/overlay2/l/K5C2FIPFO36JDE5GIVVWUHNGZ7:/var/lib/docker/overlay2/l/IQKLLJ5PTM76BSI3UF3QR645LF:/var/lib/docker/overlay2/l/CKOFI42WRX5DKLMSMSJKDIUH34:/var/lib/docker/overlay2/l/FKZ5FMYZRNOLZFHZEZAEMC2RIH:/var/lib/docker/overlay2/l/Q3CD76A5FT2M5GB4OBOANSY3FV:/var/lib/docker/overlay2/l/O4NHZLUOKBFW36BLIX7IBENA2N:/var/lib/docker/overlay2/l/WUQLKUNMNHLY3VFQCGWK62QAUC:/var/lib/docker/overlay2/l/FDIDP5WWXVSKDFU3ETHI6FBUCN:/var/lib/docker/overlay2/l/A2OCVV423OVGTR72KMP3GEK3DR:/var/lib/docker/overlay2/l/LGZ3CSBU5Q5P5KZXKMAWQHOHBC,upperdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/diff,workdir=/var/lib/docker/overlay2/7736591f8d0058c71d5a96fc2ff804040f16a6592217b213d02f45805548f5e4/work,xino=off)
    overlay on /var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/M6LPMD7J7IZLHPLWY77GBQ3KJG:/var/lib/docker/overlay2/l/GZGBLTA4I4RKZIY2MNYZTB4L7N:/var/lib/docker/overlay2/l/DFY3KHMZRZQBRQASS25SDLUS43:/var/lib/docker/overlay2/l/KDTL5P2LOGF5TM7SGQJFKHJA47:/var/lib/docker/overlay2/l/PP3E5CYLUB7TWCMPM4YHC6ZVE5:/var/lib/docker/overlay2/l/PSCML74WYZBUMBDENHMVGZB632:/var/lib/docker/overlay2/l/ZMX5E2AWWLB6BUITOG37BBLSDR:/var/lib/docker/overlay2/l/ZCR6XOPFBO4EZVUFMCCZRIB7FG:/var/lib/docker/overlay2/l/MAFJSYYASUB7SE3IZK5BSIDYQQ:/var/lib/docker/overlay2/l/MDKA3RN666VTZ4DHRRZAXY7DCW:/var/lib/docker/overlay2/l/YVI7AZAEJWVM62KI7RATSZH4P7:/var/lib/docker/overlay2/l/6Q4GLO5UN2NG4KQZXGNARNFF22:/var/lib/docker/overlay2/l/BGHG5DA4OGNOKWZMFSFMD3HLKR,upperdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/diff,workdir=/var/lib/docker/overlay2/b6c6c53bfbf73d4620dbf073c27927fe4304cb7956963cdfbc113ae3aadb01a6/work,xino=off)
    nsfs on /run/docker/netns/b6fed2a811ac type nsfs (rw)
    nsfs on /run/docker/netns/106bef75f708 type nsfs (rw)
    nsfs on /run/docker/netns/22537a0e813d type nsfs (rw)
    nsfs on /run/docker/netns/e51f7dc5bdd3 type nsfs (rw)
    nsfs on /run/docker/netns/5b4f4345b1f4 type nsfs (rw)
    nsfs on /run/docker/netns/dbf87ce558a2 type nsfs (rw)
    /var/lib/snapd/snaps/lxd_20450.snap on /snap/lxd/20450 type squashfs (ro,nodev,relatime,x-gdu.hide)
    /var/lib/snapd/snaps/core18_2066.snap on /snap/core18/2066 type squashfs (ro,nodev,relatime,x-gdu.hide)
    tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=186036k,mode=700,uid=1001,gid=1001)
    

    monit_mh_dev_sda2_not_exist.log

    brw-rw---- 1 root disk 8, 2 May 15 09:56 /dev/sda2
    

  5. Tildeslash repo owner

    Thanks for data. The log doesn’t show any problem, even though there are some overlays, it is not related to the monitored filesystem.

    We have added support for reporting the original filesystem flags to the development version, it may help to find the root cause. It seems to me the fsflags change is secondary problem, i think the main root cause is the “unable to read filesystem '/dev/XXX' state”. But that should be preceded by one of these messages:

    1. either “Filesystem 'XXX' not mounted”
    2. or “Cannot dereference filesystem 'XXX' (symlink) -- <error>

    Please can you make sure the “set log syslog” (or “set log <file>”) statement is present in your monit configuration file and send us the monit related log part to support@mmonit.com?

  6. Tildeslash repo owner

    Status update: it turned out, that the problem is caused by a temporary docker mount, which mounts the /dev/sda2 (that is already mounted to “/”) to a temporary mountpoint. Symptoms:

    monit[3513479]: Mount table change detected
    monit[3513479]: Reloading mount information for filesystem '/dev/sda2'
    monit[3513479]: device /dev/sda2 matches filesystem /dev/sda2 (mounted on /)
    monit[3513479]: device /dev/sda2 matches filesystem /dev/sda2 (mounted on /run/docker/runtime-runc/moby/b12feed0387afb581c5a8c281da2d1c9d74c6cb8c21f10363e8748def9fab2f1/runc.EDzC9W)
    monit[3513479]: Error getting usage statistics for filesystem '/run/docker/runtime-runc/moby/b12feed0387afb581c5a8c281da2d1c9d74c6cb8c21f10363e8748def9fab2f1/runc.EDzC9W' -- No such file or directory
    monit[3513479]: Filesystem '/dev/sda2' not mounted
    monit[3513479]: 'sda2' unable to read filesystem '/dev/sda2' state
    

    That rw+ro multipoint mount looks dangerous (cache coherency needs to be maintained), not sure if ext4 (and similar filesystems) can handle such use case.

    Workaround: use the specific mountpoint in the monit test rather then device:

    check filesystem XXX with path /
    

    We don’t plan to fix the filesystem scan algorithm right now, due to doubts about the rw+ro mesh sanity. We have only improved the debug mode to allow to discover the root cause of the problem (snip above).

    We may need to revisit this problem in the future, if it’ll turn out that such multipoint mount usage is correct.

  7. Tildeslash repo owner

    yet note:

    If the same filesystem is mounted to two different mount points where at least one of them is rw, the filesystem would need to keep cache coherency. That may work for cluster filesystems that are designed for such purpose, but it is very risky for filesystems like ext4: the read-only mount may fail to read data if the block was updated via the first rw mount. It may work fine if all mounts of this filesystem are read-only.

    It seems that docker does this rw+ro mesh mount for very short time frame (which is when monit sporadically catches the problem), but i think it is kind of Russian roulette.

  8. Log in to comment