Missing Mails, File Exists in Log

[Thomas Helmrich]

Hi,

i recognized that some Mails with a PDF Attachment are missing. I have f.e. those entries in the Log:

Aug 3 08:46:52 host piler[26124]: 400000005f27b2e62d82b02c003bac28ec4e: touch 409fd69a491b1a321d6cac526a06ba5d175bc498e708f1b4f371205c2e5ac34a FAILED ([e-mail]), error: File exists

When importing the Mail manually, the output is empty. i also can Re-Import the Mail and always no Ouput (Not-Even Duplicate). In the Interface the Mail is displayed the amount of import commands ( in my case three times).

‌

‌

‌

[Thomas Helmrich]

The “File exists” is appearing a lot of Times, the specific Mail got a “discarding: duplicate message“ on receive via smtp

[Janos SUTO]

The ‘touch’ is part of checking if the email is already being processed. You have at least 2+ piler child processes running. Let’s say they both are processing emails with the same message-id. So when piler processes a message, it creates a hash value from the message-id, and creates a temp file. If it can’t open ('touch') the file, it it thinks some other piler process is already processing the same email, and it returns ERR_EXITS.

So in a nutshell I suspect that you mail server presents the same email in multiple copies to piler, perhaps one copy for each recipients. If you also have a log entry status=stored for the message-id [e-mail] then I’d say piler works properly.

[Thomas Helmrich]

Ok, thanks for the info regarding duplicate message.

What about that a Mail with a PDF is not indexed via SMTP?
Only after i added it with manual Import. But then, the manual import didn´t gave any output and when i rerunned the import, no Duplicate Message warning was there and the Mail was in the Interface two times ( i runned import twice).

[Janos SUTO]

You should be able to find the pdf email in the logs sent via smtp. I suspect that it’s not there. Also I suspect that the message has no message-id, that’s the reason pilerimport was able to process it twice. I’d like to see the full headers of the pdf emails, if possible.

[Thomas Helmrich]

My Mail-Server did redirected the Mail to piler. How to grep in pilerlog for the Message?

I’d like to see the full headers of the pdf emails, if possible.

I can provide you the original Mail (via Mail or whatever), if you want.

‌

[Thomas Helmrich]

Also I suspect that the message has no message-id, that’s the reason pilerimport was able to process it twice. I’d like to see the full headers of the pdf emails, if possible.

Yes, no message ID. Is that the reason why piler didn´t imported it?

‌

[Janos SUTO]

Yes. The piler daemon cares about the message-id, the pilerimport utility doesn’t. If you want to archive such flawed messages, then consider setting archive_emails_not_having_message_id=1 in piler.conf

[Thomas Helmrich]

Good to know, thanks!

Have to restart something after change this config?

[Janos SUTO]

Yes, restart the piler daemons after fixing piler.conf