Wrong memory usage reported in LXC container

Issue #843 resolved
Former user created an issue

Hi!

This bug was already reported and fixed in #474 - but it seems it got reintroduced in 5.22. I tried a couple of Ubuntu 18.04.2, a Debian 8.11 and an Ubuntu 16.04.6 container. Memory usage gets reported like:

memory usage                 16 EB [429496729600.0%]

Downgrading to 5.21 fixes that:

memory usage                 364.8 MB [8.9%]

I tried 5.26, 5.25.1, 5.25, 5.23, 5.22, 5.21, 5.20 & 5.17.1 - from 5.17.1 to 5.21 the usage is reported correctly.

However - I have a single Ubuntu 18.04.1 container, which does not show this behaviour using 5.25.1 or 5.26 (didn't try other versions there).

I use different kernel across the containers versions:

Linux 4.15.18-11-pve #1 SMP PVE 4.15.18-33 (Tue, 05 Feb 2019 07:36:16 +0100) x86_64 x86_64 x86_64 GNU/Linux
Linux 4.15.18-12-pve #1 SMP PVE 4.15.18-36 (Fri, 05 Apr 2019 18:47:13 +0200) x86_64 x86_64 x86_64 GNU/Linux
Linux 4.15.18-10-pve #1 SMP PVE 4.15.18-32 (Sat, 19 Jan 2019 10:09:37 +0100) x86_64 GNU/Linux
Usage per /proc/meminfo and free is always reported correctly.

If you point me in the right direction I'am happy to provide you with more details.

Comments (10)

  1. Mindbender666

    I stand corrected... the usage in the 18.04.1 container is reported wrong as well.

    5.26

    memory usage                 1.2 GB [15.2%]
    

    but it should be

    5.21

    memory usage                 7.2 GB [90.2%]
    
  2. Martin Pala

    Hi,

    i think it could be related to the change, which was done in 5.22.0:

    Subtract ZFS ARC size from the system memory usage.

    If the host has a ZFS filesystem, it i possible that LXC exposes system-wide ZFS ARC statistics to the container and it exceeds the memory assigned to the container.

    Monit reads the ZFS statistics from /proc/spl/kstat/zfs/arcstats - please can you check if that file exists in the container?

  3. Mindbender666

    Hi,

    the underlying file system is indeed ZFS and /proc/spl/kstat/zfs/arcstats is present in the containers as well.

    Maybe just omit substracting the ARC size when monit runs within a container?

  4. Marcel Novotny

    Hello, we have same problem - using proxmox as host, zfs disk and in lxc as guest, and sometimes (4 times per hour) got message:

    Description: mem usage of 214748364800.0 % matches resource limit [mem usage > 90.0%]
    

    Monit version 5.25.1

    do you have some solution ?

  5. Mindbender666

    do you have some solution ?

    Besides downgrading to 5.21? I fear not 😞
    Its actually the same setup with us, and I downgraded all containers down to 5.21. I’am waiting on a solution as well…

  6. Mindbender666

    Hi there!

    Thanks for the fix 🙂

    I just tested it with one lxc container - seems to work fine:

    # 5.26.0
    System 'xxx'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      load average                 [1.92] [2.52] [2.16]
      cpu                          0.0%us 0.0%sy 0.0%wa
      memory usage                 16 EB [122713350144.0%]
      swap usage                   140 kB [0.0%]
      uptime                       28d 10h 35m
      boot time                    Tue, 10 Mar 2020 22:32:30
      data collected               Wed, 08 Apr 2020 10:08:03
    
    # Patched 5.26.0
    System 'xxx'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      load average                 [3.41] [2.77] [2.18]
      cpu                          0.0%us 0.0%sy 0.0%wa
      memory usage                 9.0 GB [64.2%]
      swap usage                   140 kB [0.0%]
      uptime                       28d 10h 34m
      boot time                    Tue, 10 Mar 2020 22:32:30
      data collected               Wed, 08 Apr 2020 10:06:15
    

  7. Tildeslash repo owner

    Fixed: Issue #843: Linux: Wrong memory usage reported in LXC container when ZFS filesystem is used. Monit now uses MemAvailable on Linux when calculating the memory usage, so it will provide more accurate statistics. Thanks to Joe LeVeque for patch.

    → <<cset 0a8069cd6662>>

  8. Log in to comment