Total upload bytes test - strange behavior

[Rychu]

First of all thank you @tildeslash and the team for this great piece of software. I have been a happy user for almost a year now.

Since the very beginning of my journey with monit I have been observing odd total upload test behavior. My current configuration of the network monitoring is as follows:

  set daemon 120            # check services at 2-minute intervals
    with start delay 30     # delay the first check by 0.5 minute

[…]

  check network enp3s0_network_interface with interface enp3s0
    start program = "/bin/ip link set dev enp3s0 up"
    stop  program = "/bin/ip link set dev enp3s0 down"
    if failed link then restart
    if changed link capacity then alert
    if saturation > 70% then alert
    if download > 100 KB/s for 3 times within 5 cycles then alert
    if upload > 100 KB/s for 3 times within 5 cycles then alert not every "* 6 * * *" # disable check when backup is on
    if total downloaded > 600 MB in last hour then alert
    if total uploaded > 600 MB in last 15 minutes then alert not every "* 6 * * *" # disable check when backup is on

So as you probably noticed from the config file that everyday at 6AM I have a backup scheduled and monit set not to alert me during that hour.

Now let’s take today as an example. Yesterday I updated lot of packages so today’s backup was bigger than usual and took longer to complete and send to the destination places.
Monit correctly notified me about that at 7AM that the total upload was close to 1GB and cleared the alert at 07:35 (let’s say the upload took longer). Odd things start to happen from here. For the next 5 hours monit had been constantly alerting and clearing the alert about the same thing only adding ~1MB to the rate every time.

[CEST Jun 18 07:00:53] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:02:55] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:04:55] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:06:55] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:08:56] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:10:56] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:12:56] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:14:56] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:16:57] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:18:57] error    : 'enp3s0_network_interface' total upload 998.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:20:57] error    : 'enp3s0_network_interface' total upload 999.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:22:58] error    : 'enp3s0_network_interface' total upload 999.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:24:58] error    : 'enp3s0_network_interface' total upload 999.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:26:58] error    : 'enp3s0_network_interface' total upload 999.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:28:59] error    : 'enp3s0_network_interface' total upload 999.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:30:59] error    : 'enp3s0_network_interface' total upload 999.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:32:59] error    : 'enp3s0_network_interface' total upload 999.7 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:35:00] info     : 'enp3s0_network_interface' total upload check succeeded [current upload rate 218.0 kB in last 15 minute]
[CEST Jun 18 07:49:03] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:51:04] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:53:04] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:55:04] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:57:05] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 07:59:05] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:01:05] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:03:06] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:05:06] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:07:06] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:09:07] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:11:07] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:13:07] error    : 'enp3s0_network_interface' total upload 958.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:15:08] info     : 'enp3s0_network_interface' total upload check succeeded [current upload rate 1.0 MB in last 15 minute]
[CEST Jun 18 08:49:14] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:51:15] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:53:15] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:55:15] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:57:16] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 08:59:16] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:01:16] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:03:17] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:05:17] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:07:17] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:09:18] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:11:18] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:13:18] error    : 'enp3s0_network_interface' total upload 960.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:15:19] info     : 'enp3s0_network_interface' total upload check succeeded [current upload rate 2.4 MB in last 15 minute]
[CEST Jun 18 09:49:25] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:51:26] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:53:26] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:55:27] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:57:27] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 09:59:27] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:01:27] error    : 'enp3s0_network_interface' total upload 961.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:03:28] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:05:28] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:07:28] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:09:29] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:11:29] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:13:29] error    : 'enp3s0_network_interface' total upload 961.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:15:30] info     : 'enp3s0_network_interface' total upload check succeeded [current upload rate 3.3 MB in last 15 minute]
[CEST Jun 18 10:49:36] error    : 'enp3s0_network_interface' total upload 962.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:51:37] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:53:37] error    : 'enp3s0_network_interface' total upload 962.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:55:38] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:57:38] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 10:59:38] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:01:39] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:03:39] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:05:39] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:07:39] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:09:40] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:11:40] error    : 'enp3s0_network_interface' total upload 961.9 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:13:40] error    : 'enp3s0_network_interface' total upload 962.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:15:41] info     : 'enp3s0_network_interface' total upload check succeeded [current upload rate 4.4 MB in last 15 minute]
[CEST Jun 18 11:49:47] error    : 'enp3s0_network_interface' total upload 963.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:51:48] error    : 'enp3s0_network_interface' total upload 963.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:53:48] error    : 'enp3s0_network_interface' total upload 963.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:55:49] error    : 'enp3s0_network_interface' total upload 963.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:57:49] error    : 'enp3s0_network_interface' total upload 963.0 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 11:59:49] error    : 'enp3s0_network_interface' total upload 963.3 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:01:50] error    : 'enp3s0_network_interface' total upload 963.2 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:03:50] error    : 'enp3s0_network_interface' total upload 963.3 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:05:50] error    : 'enp3s0_network_interface' total upload 963.3 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:07:51] error    : 'enp3s0_network_interface' total upload 963.3 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:09:51] error    : 'enp3s0_network_interface' total upload 963.3 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:11:51] error    : 'enp3s0_network_interface' total upload 963.3 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:13:51] error    : 'enp3s0_network_interface' total upload 963.3 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:15:52] info     : 'enp3s0_network_interface' total upload check succeeded [current upload rate 5.4 MB in last 15 minute]
[CEST Jun 18 12:49:58] error    : 'enp3s0_network_interface' total upload 964.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:51:59] error    : 'enp3s0_network_interface' total upload 964.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:53:59] error    : 'enp3s0_network_interface' total upload 964.1 MB matches limit [upload rate > 600 MB in last 15 minute]
[CEST Jun 18 12:56:00] info     : 'enp3s0_network_interface' total upload check succeeded [current upload rate 190.5 kB in last 15 minute]

I was investigating this issue few times in the past with iftop and tcptrack turned on in real time and I’m pretty sure that there wasn’t anything else uploading such amount of data from the server during this time.