Dashboard top user is not correct

[Oleyley]

On my dashboard sometimes the top user stats is not correct: it doesn’t list some users with a much higher number of total emails (received + sent) than those listed in the top.

What happens is apparently a bunch of the sent emails are sometimes not counted in redis “top|user|total|date” keys, whereas they are counted in the “limit|address|date” (= box out stat) and “stats|in|address|date” keys (= box in stat).

For instance if I do:

redis-cli zrange "top|user|total|20200724" 1 1000 WITHSCORES > total.txt

And then I search for a particular user for which I identified the issue, I’ll get “5” as the total value. If I now check the in and out values:

redis-cli zrange "top|user|in|20200724" 1 1000 WITHSCORES > in.txt

redis-cli zrange "top|user|out|20200724" 1 1000 WITHSCORES > out.txt

I’ll get “2” in and “3” out, which add up to 5. So far, so good. But then if I do (replacing address with the address of that user of course):

redis-cli mget “limits|address|20200724“

I get “113”!

I looked at the haraka plugin which increment all of those numbers in redis but I couldn’t find where the issue comes from.

[Oleyley]

I think I got what’s wrong. The top stat counts sent emails to multiple recipients as 1 email sent only, whereas the box out stat counts each recipient as 1 email. The mismatch between the box page and the dashboard top users happens when a user sends an email to multiple recipients.

In Haraka plugin limits.js the box out stat (“limit|address|date”) in incremented by “connection.transaction.rcpt_to.length”.

In Haraka plugin stats.js the “increase_top” function increments the top out (“top|user|out|date”) and top total (“top|user|total|date”) stats by 1 only!

[Oleyley]

So the fix for this was pretty easy, just increment the top total stat and top user stat by connection.transaction.rcpt_to.length instead of 1.

Now I also identified a mismatch between the in stat of the dashboard top users and the in stat shown in some users page. Looking again at stats.js:

• the top in stat is increased by 1 in the hook_queue_ok hook.
• the user in stat is increased by 1 in the hook_delivered hook (only if mode === 'LMTP').

Those probably don’t trigger exactly the same number of times, explaining the mismatch, but I’m not sure why. Perhaps some emails get queued but are never delivered? Or can a user receive an email without mode === 'LMTP'?

‌