Retention Rules missbehaviour

[Thomas Helmrich]

Hello,

i recognized that my retention rule is sometimes working, sometimes not - really strange behaviour.

For example:

Simple Retention Rule:

‌

pilertest shows correct retention rule:

pilertest -c /usr/local/etc/piler/piler.conf -m Mail.eml
locale: en_US.UTF-8
build: 998
parsing...
post parsing...
message-id: <[e-mail]> / 4e505c96c623e37fafa04c72f7f7600e1b0d050cbab8e42ee45d87ac46d047a8
from: *root [e-mail] root senderserver de  (senderserver.de)*
to: *[e-mail] mail mail com  (mail.com )*
....
retention period: 1591693098 (14 days)
i:1, name=*FILE.csv*, type: *text/csv*, size: 746307, int.name: Mail.eml.a1, digest: d315aa398561ee413ebcca3731477239d0f4cd0ce567f41e2619e0f86a097509
...

But when i go in metadata:

 select subject, arrived, retained from metadata where message_id LIKE '[e-mail]%';
+------------+------------+------------+
| subject    | arrived    | retained   |
+------------+------------+------------+
| Mail       | 1588504941 | 1904296941 |
+------------+------------+------------+

A lot of Mails have the wrong retention time, but also some mails have a correct one.
Pretty Strange.

[Thomas Helmrich]

More debugging:

‌

 select arrived, retained, deleted, size from metadata where arrived > 1588484480 and arrived < 1588959142 and subject = 'Mail' order by arrived asc limit 0,20;
+------------+------------+---------+--------+
| arrived    | retained   | deleted | size   |
+------------+------------+---------+--------+
| 1588485150 | 1904277150 |       0 | 748465 |
| 1588486042 | 1904278042 |       0 | 748442 |
| 1588486945 | 1904278945 |       0 | 748442 |
| 1588487842 | 1904279842 |       0 | 748442 |
| 1588488741 | 1904280741 |       0 | 748442 |
| 1588489645 | 1904281645 |       0 | 748442 |
| 1588490542 | 1904282542 |       0 | 748684 |
| 1588491441 | 1589701041 |       1 | 748442 |
| 1588492342 | 1904284342 |       0 | 748684 |
| 1588493250 | 1589702850 |       1 | 748465 |
| 1588494142 | 1904286142 |       0 | 748442 |
| 1588495043 | 1589704643 |       1 | 748684 |
| 1588495942 | 1904287942 |       0 | 748442 |
| 1588496841 | 1589706441 |       1 | 748684 |
| 1588497744 | 1589707344 |       1 | 748684 |
| 1588498642 | 1589708242 |       1 | 748404 |
| 1588499540 | 1589709140 |       1 | 748442 |
| 1588500442 | 1589710042 |       1 | 748404 |
| 1588501348 | 1589710948 |       1 | 748707 |
| 1588502242 | 1589711842 |       1 | 748684 |
+------------+------------+---------+--------+

‌

[Janos SUTO]

The retention value is calculated at archiving time, while pilertest shows the retention value based on the current retention rules. Is it possible that you applied the 14 day rule later?

[Janos SUTO]

No news is good news.

[Thomas Helmrich]

Oops - sorry.

Is it possible that you applied the 14 day rule later?

No - i added it minimum a half year ago.

[Thomas Helmrich]

It seems that it happens random, what is really weird, that some are retained and some not:

Correct:

Received: 05/20/2020 @ 2:52am
Retained: 06/03/2020 @ 2:52am

Wrong:

Received: 05/20/2020 @ 3:07am
Retained: 05/20/2020 @ 3:22am

Wrong:

Received: 05/20/2020 @ 3:37 
Retained: 05/23/2030 @ 3:37

Correct:

Received: 05/20/2020 @ 4:07am
Retained: 06/03/2020 @ 4:07am  

‌

‌

[Thomas Helmrich]

Hi @Janos SUTO
could we have a look on this, please?

Still have messages that are marked as deleted and some not. Seems like randomly?
Anything i could do to debug it?

Thanks!

[Thomas Helmrich]

Seems like it is fixed since Update!
A few days after update i hade 2-3 wrong retention recognitions, but until 30.5. i got only retention=14 in my mail.log, before update there where a lot of retention=DEFAULT_RET and retention=0