Messages discarded with status=duplicate

[Veaceslav Grecea]

Hello Janos,

Here is our instance -
piler 1.3.8, build 998, Janos SUTO [e-mail]

Build Date: Tue Mar 31 20:52:54 AEDT 2020

ldd version: ldd (GNU libc) 2.17

gcc version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)

OS: Linux mailarchiver.ljhfairfield.com.au 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Configure command: ./configure --localstatedir=/var --with-database=mysql --enable-clamd --enable-memcached

MySQL client library version: 10.4.13

Extractors: /usr/bin/pdftotext /usr/bin/catdoc /usr/bin/catppt /usr/bin/xls2csv /usr/local/bin/ppthtml /usr/bin/unrtf /usr/bin/tnef libzip

Sphynx - 3.2.1

Without any reasons once in a while incoming messages are not indexed any longer and discarded with status=duplicate
Stored messages have a valid references in maillog, those discarded have reference field empty.

1. Jul 24 10:23:05 mailarchiver piler[85193]: 1/3GNHDFS1WDNVKX83: 400000005f1a29f316118b34000000007043, size=1527813/0, attachments=0, reference=, message-id=null, retention=0, folder=0, delay

   =0.0114, status=duplicate

2. Jul 24 10:26:06 mailarchiver piler[87353]: 1/SHPI8W49O9WAV6D0: 400000005f1a2aa812ce27d400ded9fa1425, size=59435/10008, attachments=9, reference=<ME4P282MB08232B1B8A04099BF16CC429E2770@ME4P2

82MB0823.AUSP282.PROD.OUTLOOK.COM>, message-id=[e-mail]>, retention=2557, folder=0, delay=0.0927, status=stored

If I restart piler daemon messages begin to be stored again.
I don’t quite understand what’s going on and why piler stops storing the messages. Do you have any idea where the problem comes from ?
Thank you !

[Janos SUTO]

Hello Veaceslav, I suspect that those problematic emails has no Message-ID lines in the header. Can you check this theory based on the delivery date and perhaps size?

‌

[Veaceslav Grecea]

Hello Janos, the problem is that after the first such discarded message only piler restart helps. I usually find about this only via gui statistics.
Anything else we can think about ? Because if we get any wrong message (zero sized or with wrong headers), piler should discard it and continue to store other “good“ messages, but it does not. Everything else which comes is automatically discarded with duplicate status and piler restart allows to store input messages again. I’ll talk with admin who sends messages to check your theory.
Thanks !

[Veaceslav Grecea]

Hello Janos,
I have located a few of those messages that have been discarded by piler, started from the first one. They all had proper headers, including message_id.
Any other clues of why piler refuses to store messages at some point ?
Thanks.

[Janos SUTO]

Odd. Check if mysql daemon works properly. Look for any errors in the mysql logs, also check if you have big enough innodb buffers, etc. The FAQ http://www.mailpiler.org/wiki/faq has some starter settings. Don’t be afraid to allocate more memory if you have the resources.

[Veaceslav Grecea]

Thanks, Janos.

I’m going to play with mysql and come back with the results.

Please don’t close the ticket till then :)

[Janos SUTO]

OK, I won’t :-)

[Janos SUTO]

Any update on the matter?

[Veaceslav Grecea]

Hello Janos,

I think you can close the ticket. It seems to be a problem on a client side.

Thank you for your support !