Fals duplicate message

[János Csárdi-Braunstein]

We receive a mail, but piler say, this is a duplicate message:

W do not receive mail from this sender before, and i search for this sender, and i found nothing.

This destination mail is an alias.

{{{ piler[2787]: 400000004ff4999c3292ccdc2015433076f8: from=[e-mail] USER DOMAIN TLD, to=[e-mail] DESTUSER DESTDOMAIN com, subj= ..., message-id=null, reference= piler[2787]: 400000004ff4999c3292ccdc2015433076f8: found message_id:null(4) null=0 piler[2787]: 400000004ff4999c3292ccdc2015433076f8: discarding duplicate message piler[2787]: 400000004ff4999c3292ccdc2015433076f8: [e-mail] got mail size=4573, body digest=e35e991f105b7e5cee175e802322f51f5050367e0e134123443e5db4fc19394a, delay=0.38, delays=0.38/0.00/0.00/0.00/0.00/0.00 }}}

[Janos SUTO]

could you please show me the email header? You can screen out any sensitive information, but the message-id line must be included.

According to the logs piler couldn't find any message-id in the header, and if you had any similar message in the past, then it's a duplicate from piler's point of view.

[János Csárdi-Braunstein]

This is the header from Thunderbird:

Return-path: <[e-mail]>
Envelope-to: REALUSER@DESTDOMAIN
Delivery-date: Wed, 04 Jul 2012 21:28:29 +0200
Received: by SAJATSZERVEREM.DOMAIN.TLD with esmtps (TLSv1:AES256-SHA:256) id 1SmVFV-0008F4-TC for REALUSER@DESTDOMAIN; Wed, 04 Jul 2012 21:28:29 +0200
Received: from cp.DOMAIN.co ([KULDO IP]) by mx1.SAJATSZERVEREM.DOMAIN.TLD with esmtps (TLSv1:AES256-SHA:256) (envelope-from <[e-mail]>) id 1SmVGM-0001Gx-KO for [e-mail]; Wed, 04 Jul 2012 21:29:22 +0200
Received: from cm-XXXXX.YY.getinternet.TLD ([XXXXX.YY]:60970 helo=lap) by cp.DOMAIN.co with esmtpa (Exim 4.77) (envelope-from <[e-mail]>) id 1SmRML-0006Q6-TR for [e-mail]; Wed, 04 Jul 2012 17:19:18 +0200
MIME-Version: 1.0
From: [e-mail]
To: [e-mail]
Date: 4 Jul 2012 17:19:15 +0200
Subject: **
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
X-AntiAbuse: Primary Hostname - cp.DOMAIN.co
X-AntiAbuse: Original Domain - DOMAIN.TLD
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - DOMAIN.TLD
X-Source:
X-Source-Args:
X-Source-Dir: 

My MX send a copy to piler from a mail.

[Janos SUTO]

thanks for the header. I couldn't see the Message-Id: line, so that's why piler discarded the message. Do you often get messages without Message-Id? If so, and you need these emails to be archived, then it's possible to add a configuration variable that makes piler to assign a random message-id if it can't find any.

[János Csárdi-Braunstein]

What is this configuration variable?

[Janos SUTO]

it's not implemented yet. I wanted to say that I would make it if you need it because you get lots of legitim messages without a valid Message-Id. So do you need this feature?

[János Csárdi-Braunstein]

I think, this is a good feature, because RFC 2821 says:

"As discussed in section 2.4.1, a relay SMTP has no need to inspect or act upon the headers or body of the message data and MUST NOT do so except to add its own "Received:" header (section 4.4) and, optionally, to attempt to detect looping in the mail system (see section 6.2)."

So, i think, i no need to add missing message id in my relay server (and in my infrastrukture the MX server is only a relay).

And yes, i need this feature :)

[Janos SUTO]

ok, the following configuration option solves this problem.

archive_emails_not_having_message_id=1

Please download the master branch (https://bitbucket.org/jsuto/piler/get/master.tar.gz) to use this feature.

And if you are upgrading from <0.1.19, then please review the upgrade instructions, too (http://www.mailpiler.org/wiki/doku.php/current:upgrade)

[János Csárdi-Braunstein]

I upgrade piler from master, but i give this: Jul 6 14:56:45 mailarchive0 piler[25912]: unknown key: "archive_emails_not_having_message_id"

[Janos SUTO]

hmmm, please show me

piler -v

then add the following (fake) variable (only temporarily) to piler.conf:

aaaaaa=1

and finally show me please

pilerconf -n | grep "unknown key"

I know that you've done but just to make sure: have you also updated the libpiler.* files, too?

[János Csárdi-Braunstein]

piler 0.1.19, Janos SUTO <[e-mail]>

Build Date: Fri Jul 6 14:52:10 CEST 2012
Configure command: ./configure --enable-memcached --localstatedir=/var

# pilerconf -n | grep "unknown key"
unknown key: "aaaaaa"

[Janos SUTO]

ok, it looks fine. Do you still have a complaint for archive_emails_not_having_message_id?