Disable deduplication based on Message-ID again

[Tomem]

Hi,

First the background. Our users archive their emails with special script that get the messages from their mailboxes and send them to piler with SMTP. In order to preserve the email owner (a user that logs to Piler wants to see all his/her emails), the script adds X-Envelope-To header to the mail.

Everything works ok, but the problem is with mails that have been sent to multiple users. When the second user tries to archive the email, it is deduplicated and the information about the owner is lost.

I've found the issue 602: https://bitbucket.org/jsuto/piler/issues/602/disable-deduplication-based-on-message-id, but I need to concat my custom header value with mail-id and create hash. Is this header (X-Envelope-To) value available in post_parse() function?

Or maybe you have completely different idea to complete the user requirements?

[Janos SUTO]

I won't disable the message-id deduplication. Note that the message-id is a unique column in the metadata table, and each message should have a unique id. I also don't understand why any owner info is lost. The owner info is derived from the From/To/Cc and X-Envelope-To headers, and a given owner of the email should be either of them. Btw why the users do archive emails? They shouldn't archive anything, the process should be automatic without their intervention. Perhaps you need some more clarification what's going on at your site.

[Tomem]

I don't want you to disable message-id. I would like to prepare special version of piler for our internal use. I clarify our process:

We are using piler in different way that it is planned. We gave the user the choice to archive mails from their mailbox with our script. Mails are not sent to piler on the MTA level (we don't want to archive all emails). Mails on mailboxes are new (with X-Envelope headers added by MTA) and there are also old ones that were migrated from previous mail system (without X-envelope header).

So, we struggle with such problem: 1/ Two users wants to archive (send to piler) the mail that has no X-Envelope-To header and both users were in BCC: field. 2/ First user: our script adds X-Envelope-To header and sends it to piler. Everything is OK (the mail is archived and the owner can see it) 2/ Second user: our script adds X-Envelope-To header and sends it to piler. Piler doesn't archive it (the same message-id) but the second user can't see this email because his email is not in X-Envelope-To header.

I think that changing hash function would help to store the email for second user. (sacrificing deduplication in special cases like this one above) The question is if the value of X-Envelope-To header is available in post_parse() function.

[Janos SUTO]

Piler processes not only X-Envelope-To, but From, To and Cc as well. If the recipient wasn't in any of them, then you may still add -a email@address to pilerimport. Then pilerimport would not store the message the 2nd time, but append the -a address to the rcpt table, so the 2nd user could see it in the gui.

[Janos SUTO]

Conclusion: won't fix, sorry. However the workaround may work out. Let me know how it goes.

[Tomem]

Unfortunatelly "pilerimport -e mymsg.eml -a [e-mail]" doesn't update rcpt table. My Piler version: 1.2.0-master build 949

[Janos SUTO]

Try setting process_rcpt_to_addresses=1 in piler.conf