verp addressing with bcc not working

Hi Janos,

I have already posted this in ~~#335~~, because I thought it fits there. But as that issue is already closed I thought it would be better to open a new issue.

It seems that at least in 1.3.4 there is an issue with verp addressing. A mail to several BCC receipients is only stored for the first receipient and the subsequent mails piler receives for the other receipients are rejected because they are duplicate.

I sent a mail to the following receipients: from: test1@XXXX.local to: test1@XXXX.local bcc: test2@XXXX.local, test3@XXXX.local

Mail log from piler looks like this:

Jun  1 18:25:09 piler-test piler-smtp[2535]: connected from 10.253.57.114:60690 on descriptor 6 (active connections:1)
Jun  1 18:25:09 piler-test piler-smtp[2535]: received: Y05TL0XP261NOJ4B, from=test1@XXXX.local, size=1760, client=10.253.57.114
Jun  1 18:25:09 piler-test piler-smtp[2535]: disconnected from 10.253.57.114 on descriptor 6 (0 active connections)
Jun  1 18:25:09 piler-test piler-smtp[2535]: connected from 10.253.57.114:60704 on descriptor 6 (active connections:1)
Jun  1 18:25:09 piler-test piler[2547]: 9/Y05TL0XP261NOJ4B: 400000005b11736f1cd81dfc00ffedef1516, size=1802/888, attachments=0, reference=, message-id=<765b69baf7aff678bb258d76daba3393@XXXX.local>, retention=2557, folder=0, delay=0.0
214, status=stored
Jun  1 18:25:09 piler-test piler-smtp[2535]: received: YQ0BUN6LL32I6V8R, from=test1@XXXX.local, size=1760, client=10.253.57.114
Jun  1 18:25:09 piler-test piler-smtp[2535]: disconnected from 10.253.57.114 on descriptor 6 (0 active connections)
Jun  1 18:25:09 piler-test piler-smtp[2535]: connected from 10.253.57.114:60706 on descriptor 6 (active connections:1)
Jun  1 18:25:09 piler-test piler-smtp[2535]: received: YFBZG832L18R8Q1C, from=test1@XXXX.local, size=1760, client=10.253.57.114
Jun  1 18:25:10 piler-test piler[2547]: 9/YQ0BUN6LL32I6V8R: discarding: duplicate message, id: 5, message-id: <765b69baf7aff678bb258d76daba3393@XXXX.local>
Jun  1 18:25:10 piler-test piler[2547]: 9/YQ0BUN6LL32I6V8R: 400000005b1173701e2746c400410545cdd9, size=1802/0, attachments=0, reference=, message-id=<765b69baf7aff678bb258d76daba3393@XXXX.local>, retention=0, folder=0, delay=0.0291,
status=duplicate
Jun  1 18:25:10 piler-test piler[2547]: 9/YFBZG832L18R8Q1C: discarding: duplicate message, id: 5, message-id: <765b69baf7aff678bb258d76daba3393@XXXX.local>
Jun  1 18:25:10 piler-test piler[2547]: 9/YFBZG832L18R8Q1C: 400000005b1173701fe36de4000b333fa27a, size=1802/0, attachments=0, reference=, message-id=<765b69baf7aff678bb258d76daba3393@XXXX.local>, retention=0, folder=0, delay=0.0042,
status=duplicate

So the message is only stored and visible to the first receipient:

$ pilerget 400000005b1173701e2746c400410545cdd9

X-Piler-Envelope-To: test1@XXXX.local
Received: from iredmail-test.XXXX.local (localhost [127.0.0.1])
        by iredmail-test.XXXX.local (Postfix) with ESMTP id 9107E361389
        for <archive+test1=XXXX.local@piler-test.XXXX.local>; Fri,  1 Jun 2018 18:25:09 +0200 (CEST)
Authentication-Results: iredmail-test.XXXX.local (amavisd-new); dkim=pass
        reason="pass (just generated, assumed good)" header.d=XXXX.local
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=XXXX.local;
         h=user-agent:message-id:subject:subject:to:from:from:date:date
        :content-transfer-encoding:content-type:content-type
        :mime-version; s=dkim; t=1527870309; x=1530462310; bh=3g7dCsSnBa
        /3DzRzTpCh0KHYt2q+S7U/Pqk0vBBbOxc=; b=sYBhGRqcp7QvSxiXGbb1VvzT0p
        WsyNUSN+O+D+uRG16OoL+cA8YUUpe5EmDZS7avm+7vo8RNNEEF4ES32jcVdFoFcK
        a3yvGwq4xt8TwPGtkiTuJEkEmSkqB4+vUV/oVz/7g7qt8nDUU57Watf70SOTAhYp
        DTnXsQTTmr5XJGy8U=
X-Virus-Scanned: Debian amavisd-new at iredmail-test.XXXX.local
Received: from iredmail-test.XXXX.local ([127.0.0.1])
        by iredmail-test.XXXX.local (iredmail-test.XXXX.local [127.0.0.1]) (amavisd-new, port 10026)
        with ESMTP id hX6RPvz-Kian
        for <archive+test1=XXXX.local@piler-test.XXXX.local>;
        Fri,  1 Jun 2018 18:25:09 +0200 (CEST)
Received: from _ (localhost [127.0.0.1])
        by iredmail-test.XXXX.local (Postfix) with ESMTPSA id 6234E361386;
        Fri,  1 Jun 2018 18:25:09 +0200 (CEST)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Fri, 01 Jun 2018 18:25:09 +0200
From: test1@XXXX.local
To: test1 <test1@XXXX.local>
Subject: test2
Message-ID: <765b69baf7aff678bb258d76daba3393@XXXX.local>
X-Sender: test1@XXXX.local
User-Agent: Roundcube Webmail

test2

The other receipients are not added to the rcpt table:

MariaDB [piler]> select * from metadata where piler_id='400000005b11736f1cd81dfc00ffedef1516';
+----+----------------------+----------------+---------+------+------------+------------+------------+---------+------+------+-----------+-------------+--------------------------------------+---------------------------------------------------+-----------+------------------------------------------------------------------+------------------------------------------------------------------+------------------------------------------------------------------+
| id | from                 | fromdomain     | subject | spam | arrived    | sent       | retained   | deleted | size | hlen | direction | attachments | piler_id                             | message_id                                        | reference | digest                                                           | bodydigest                                                       | vcode                                                            |
+----+----------------------+----------------+---------+------+------------+------------+------------+---------+------+------+-----------+-------------+--------------------------------------+---------------------------------------------------+-----------+------------------------------------------------------------------+------------------------------------------------------------------+------------------------------------------------------------------+
|  5 | test1@XXXX.local | XXXX.local | test2   |    0 | 1527870309 | 1527870309 | 1748795109 |       0 | 1802 | 1795 |         0 |           0 | 400000005b11736f1cd81dfc00ffedef1516 | <765b69baf7aff678bb258d76daba3393@XXXX.local> |           | 30940484efd4924374bd3e044112038d54f895d4372064d54edae9b4ebaef8b3 | de0edd0ac4a705aff70f34734e90a1d0a1d8b76abe4bb53f3ea934bc105b3b17 | 46c94a890ea89ea8b4f2a77073a86d3e1787d082708b6f8a0942c11762809935 |
+----+----------------------+----------------+---------+------+------------+------------+------------+---------+------+------+-----------+-------------+--------------------------------------+---------------------------------------------------+-----------+------------------------------------------------------------------+------------------------------------------------------------------+------------------------------------------------------------------+
1 row in set (0.00 sec)


MariaDB [piler]> select * from rcpt;
+----+----------------------------------+----------------+
| id | to                               | todomain       |
+----+----------------------------------+----------------+
|  2 | test2@XXXX.local             | XXXX.local |
|  3 | test1@XXXX.local             | XXXX.local |
|  4 | test1@XXXX.local             | XXXX.local |
|  5 | test1@XXXX.local             | XXXX.local |
|  1 | undisclosed-recipients@no.domain | no.domain      |
+----+----------------------------------+----------------+
5 rows in set (0.00 sec)

This is my piler.conf on a freshly installed piler instance with version 1.3.4:

archive_emails_not_having_message_id=0
archive_only_mydomains=0
backlog=20
cipher_list=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
clamd_socket=/tmp/clamd
default_retention_days=2557
enable_chunking=0
enable_cjk=0
enable_folders=0
encrypt_messages=1
extract_attachments=1
extra_to_field=X-Envelope-To:
helper_timeout=20
hostid=piler-test.XXXX.local
listen_addr=0.0.0.0
listen_port=25
max_requests_per_child=1000
memcached_servers=127.0.0.1
memcached_to_db_interval=900
memcached_ttl=86400
min_message_size=100
min_word_len=1
mmap_dedup_test=0
mysqlhost=127.0.0.1
mysqlport=3306
mysql_connect_timeout=2
mysqldb=piler
mysqlpwd=XXXXXXXXXX
mysqlsocket=/var/run/mysqld/mysqld.sock
mysqluser=piler
number_of_worker_processes=10
pemfile=/usr/local/etc/piler/piler.pem
pidfile=/var/run/piler/piler.pid
piler_header_field=X-piler-id:
process_rcpt_to_addresses=1
server_id=0
spam_header_line=
syslog_recipients=0
tls_enable=0
tweak_sent_time_offset=0
update_counters_to_memcached=0
username=piler
verbosity=1
workdir=/var/piler/tmp
mysqlcharset=utf8
queuedir=/var/piler/store

The test message I sent to piler comes from a freshly installed iredmail server using a bcc_map config I took from one of the comments in this thread

#/etc/postfix/bcc_map
#
/^([^+]+).*@(.*)$/      archive+$1=$2@piler-test.XXXX.local

I am trying to use verp adressing because I hope to solve the bcc reveal problem. Please note, that this issue is important for me because of privacy concerns and I am also willing to pay for your support if necessary. In that case just send me a contact info so we can discuss the expected price and necessary steps.

Kind Regards, Martin

Comments (14)