Wrong memory usage reported in LXC container

[Former user]

Hi!

This bug was already reported and fixed in #474 - but it seems it got reintroduced in 5.22. I tried a couple of Ubuntu 18.04.2, a Debian 8.11 and an Ubuntu 16.04.6 container. Memory usage gets reported like:

memory usage                 16 EB [429496729600.0%]

Downgrading to 5.21 fixes that:

memory usage                 364.8 MB [8.9%]

I tried 5.26, 5.25.1, 5.25, 5.23, 5.22, 5.21, 5.20 & 5.17.1 - from 5.17.1 to 5.21 the usage is reported correctly.

However - I have a single Ubuntu 18.04.1 container, which does not show this behaviour using 5.25.1 or 5.26 (didn't try other versions there).

I use different kernel across the containers versions:

Linux 4.15.18-11-pve #1 SMP PVE 4.15.18-33 (Tue, 05 Feb 2019 07:36:16 +0100) x86_64 x86_64 x86_64 GNU/Linux
Linux 4.15.18-12-pve #1 SMP PVE 4.15.18-36 (Fri, 05 Apr 2019 18:47:13 +0200) x86_64 x86_64 x86_64 GNU/Linux
Linux 4.15.18-10-pve #1 SMP PVE 4.15.18-32 (Sat, 19 Jan 2019 10:09:37 +0100) x86_64 GNU/Linux

Usage per /proc/meminfo and free is always reported correctly.

If you point me in the right direction I'am happy to provide you with more details.

[Mindbender666]

I stand corrected... the usage in the 18.04.1 container is reported wrong as well.

5.26

memory usage                 1.2 GB [15.2%]

but it should be

5.21

memory usage                 7.2 GB [90.2%]

[Martin Pala]

Hi,

i think it could be related to the change, which was done in 5.22.0:

Subtract ZFS ARC size from the system memory usage.

If the host has a ZFS filesystem, it i possible that LXC exposes system-wide ZFS ARC statistics to the container and it exceeds the memory assigned to the container.

Monit reads the ZFS statistics from /proc/spl/kstat/zfs/arcstats - please can you check if that file exists in the container?

[Mindbender666]

Hi,

the underlying file system is indeed ZFS and /proc/spl/kstat/zfs/arcstats is present in the containers as well.

Maybe just omit substracting the ARC size when monit runs within a container?

[Marcel Novotny]

Hello, we have same problem - using proxmox as host, zfs disk and in lxc as guest, and sometimes (4 times per hour) got message:

Description: mem usage of 214748364800.0 % matches resource limit [mem usage > 90.0%]

Monit version 5.25.1

do you have some solution ?

‌

[Mindbender666]

do you have some solution ?

Besides downgrading to 5.21? I fear not
Its actually the same setup with us, and I downgraded all containers down to 5.21. I’am waiting on a solution as well…

‌

[Tildeslash]

Hello, it should be fixed now (https://bitbucket.org/tildeslash/monit/commits/1e420ea44a416cf23027fa850faf616d48a26cd6)

‌

If you want to test, you can checkout the code from Bitbucket and compile it this way:

./bootstrap
./configure
make

‌

[Mindbender666]

Hi there!

Thanks for the fix

I just tested it with one lxc container - seems to work fine:

# 5.26.0
System 'xxx'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  load average                 [1.92] [2.52] [2.16]
  cpu                          0.0%us 0.0%sy 0.0%wa
  memory usage                 16 EB [122713350144.0%]
  swap usage                   140 kB [0.0%]
  uptime                       28d 10h 35m
  boot time                    Tue, 10 Mar 2020 22:32:30
  data collected               Wed, 08 Apr 2020 10:08:03

# Patched 5.26.0
System 'xxx'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  load average                 [3.41] [2.77] [2.18]
  cpu                          0.0%us 0.0%sy 0.0%wa
  memory usage                 9.0 GB [64.2%]
  swap usage                   140 kB [0.0%]
  uptime                       28d 10h 34m
  boot time                    Tue, 10 Mar 2020 22:32:30
  data collected               Wed, 08 Apr 2020 10:06:15

‌

[Tildeslash]

Fixed: Issue #843: Linux: Wrong memory usage reported in LXC container when ZFS filesystem is used. Monit now uses MemAvailable on Linux when calculating the memory usage, so it will provide more accurate statistics. Thanks to Joe LeVeque for patch.

→ <<cset 0a8069cd6662>>

[Tildeslash]

Issue #474 was marked as a duplicate of this issue.

[Tildeslash]

Issue #385 was marked as a duplicate of this issue.