[Postfix related]: x-envelope-to reveals bcc
When using postfix, prepending the X-Envelope-To header will reveal all BCC recipients.
I might not have found the right place for prepending the header yet (smtpd_recipient_restrictions). Please correct me if there is another way!
As an alternative one could use recipient_bcc_maps and allow sub-addressing to take care of the original envelope recipient (which would be part of the sub-address).
E.g. my route to pile is archive@archive.local, sub-addressing of the original recipient alex@example.org would be archive+alex=example.org@archive.local.
For this to work the piler parser needs to be aware of the type of sub-addressing used.
Alex
Comments (70)
-
repo owner -
Account Deleted You would use "recipient_bcc_maps" instead of "always_bcc"
# /etc/postfix/main.cf # recipient_delimiter = + recipient_bcc_maps = pcre:/etc/postfix/bcc_map
The map would look something like:
# /etc/postfix/bcc_map # /^(.*)@(.*)$/ archive+$1=$2@myarchive.local
(given that you have a transport for "myarchive.local")
Any given envelope address would rewrite to a bcc-recipient in the form of archive+<name>=<domain>@myarchive.local
For example an email delivered to "test@example.org" would be bcc'ed to archive+test=example.org@myarchive.local
The "+" delimiter starts the sub-address and the "@" of the original recipient address is replaced by "=" as only one "@" is obviously allowed in email addresses.
Your parser could for instance know about the sub-address delimiter (config option!?) and about the "@" replacement character in order to form a valid address.
I found a statement more than a decade old by Wietse Venema about him wanting to address that issue - don't know what became of it. Old Post by Wietse
There might be another way by now that I don't know of.
-
repo owner Thanks for the hint, I'll check it out. However note that processing the same message several times (ie. one occasion for a single recipient) is a bit wasting of resources. With 10+ recipients much waste.
-
repo owner OK, check out the latest master branch, see the Download link above. Let me know how it works for you.
-
Account Deleted Wow, that was a prompt reply!
I just checked it, yet it doesn't quite work (yet)...
Jun 30 17:39:25 mail piler[21633]: 4000000053b184b635237a3c00a8dc181426: from=testuser@example.com, size=43726/6208, attachments=2, reference=, message-id=<dcbe5815-1223-44cf-8846-f9567a5329ee@WG-EX01.somecorp.local>, retention=2557, delay=0.08, delays=0.07/0.00/0.01/0.00/0.00/0.00, status=stored Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=example-1.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0,status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <archiv+testuser@example-1.org>) Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=example-2.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0, status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <archiv+testuser@example-1.org>) Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=xn--ltzimmer-n4a.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0, status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <archiv+testuser@example-1.org>)
The message I sent:
- From: testuser@example.com
- To: testuser@example.com.
- BCC: testuser@example-1.org, testuser@example-2.org, testuser@lötzimmer.org
In Piler I cannot see any BCC information, I only see a single mail from testuser@example.com to testuser@example.com
Piler did get the message but instead of finding the additional Information in the X-Envelope-to header, it would have to add the recipient information passed by the envelope address.
-
repo owner OK, try again. I think I've nailed it this time.
-
Account Deleted That must've been quite some changes in your commit...
We are almost there - it works fine but now I always get the archive-mail address (archiv@myarchive.local) as the first recipient in piler (even when using X-Envelope-To).
-
repo owner What's the "hostid" in piler.conf?
-
Account Deleted It was not "myarchive.local" ;)
Changed it, and it works now! Let me test for another week and if I don't find any problems you might consider this an alternative to the current postfix header prepending way.
-
repo owner OK, take your time. I hope it will be ok.
-
Account Deleted Strangest things are happening... After having been away for a few days I checked my queue and noticed quite a few mails stuck, not being delivered to piler.
Most of them end as below (piler in debug mode):
Jul 7 15:17:51 mail postfix/qmgr[24140]: 69EE58400EB: from=<notification+aqqkkkgx@facebookmail.com>, size=12569, nrcpt=2 (queue active) Jul 7 15:17:51 mail piler[24041]: connection from 1.2.3.4 Jul 7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: sent: 220 myarchive.local ESMTP#015 Jul 7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: got: EHLO mx.example.com Jul 7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: sent: 250-myarchive.local#015#012250-PIPELINING#015#012250-SIZE#015#012250 8BITMIME#015 Jul 7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: got: MAIL FROM:<notification+aqqkkkgx@facebookmail.com> SIZE=12569 BODY=7BIT Jul 7 15:17:51 mail piler[24222]: forked a child (pid: 24222) Jul 7 15:17:51 mail piler[24222]: child (pid: 24222) started main() Jul 7 15:17:51 mail postfix/smtp[24172]: 69EE58400EB: to=<archiv+test.user=example.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=319443, delays=319443/0/0/0, dsn=4.4.2, status=deferred (lost connection with 127.0.0.1[127.0.0.1] while sending MAIL FROM)
Another thing I noticed is the following (piler in normal mode):
Jul 1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: mysql_stmt_execute error: *Duplicate entry '288736-test.user@example.org' for key 'id'* Jul 1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: from=sender@somesender.org, size=42616/6960, attachments=1, reference=<C5F711ED0309444ABD2404B33220CCC3F8073D6717@pamail01.gvm.somesender.org>, message-id=<C5F711ED0309444ABD2404B33220CCC3F8073D672F@pamail01.gvm.somesender.org>, retention=2557, delay=0.08, delays=0.06/0.00/0.02/0.00/0.00/0.00, status=stored Jul 1 16:35:00 mail postfix/smtp[9098]: 460968400C1: to=<archiv+test.user=example.org@wissler-archiv.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.3, delays=0.05/0/0/0.25, dsn=2.0.0, status=sent (250 Ok 4000000053b2c71e13f0ce1400f725273a52 <test.user@example.org>) Jul 1 16:35:00 mail postfix/qmgr[9096]: 460968400C1: removed
-
repo owner - attached misc.c
-
repo owner Yes, I made a text processing error. Please update src/misc.c with the attached file, and recompile piler. It should fix the issue.
-
Account Deleted Again, thanks for the amazingly quick response!
I recompiled yesterday night and the interruption of the mail delivery has vanished.
Yet, I still get the second error about mail duplication. I have the feeling those messages do not show up in the archive:
Jul 1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: mysql_stmt_execute error: *Duplicate entry '288736-test.user@example.org' for key 'id'* Jul 1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: from=sender@somesender.org, size=42616/6960, attachments=1, reference=<C5F711ED0309444ABD2404B33220CCC3F8073D6717@pamail01.gvm.somesender.org>, message-id=<C5F711ED0309444ABD2404B33220CCC3F8073D672F@pamail01.gvm.somesender.org>, retention=2557, delay=0.08, delays=0.06/0.00/0.02/0.00/0.00/0.00, status=stored Jul 1 16:35:00 mail postfix/smtp[9098]: 460968400C1: to=<archiv+test.user=example.org@wissler-archiv.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.3, delays=0.05/0/0/0.25, dsn=2.0.0, status=sent (250 Ok 4000000053b2c71e13f0ce1400f725273a52 <test.user@example.org>) Jul 1 16:35:00 mail postfix/qmgr[9096]: 460968400C1: removed
-
repo owner Please check if you can retrieve the message with "pilerget 4000000053b2c71e13f0ce1400f725273a52". Also I'd like to see "grep C5F711ED0309444ABD2404B33220CCC3F8073D672F@pamail01.gvm.somesender.org /var/log/maillog" output to see how many emails with the given message-id you received. And finally check the rcpt table in the piler database whether it has all recipients. To do this execute the following query:
mysql> select * from rcpt where id=(select id from metadata where piler_id='4000000053b2c71e13f0ce1400f725273a52');
I suspect that 288736-test.user@example.org was added earlier.
-
Account Deleted My observation is that this error only appears after applying the VERP-Patches last week.
Above mail was received once. And the query returns exactly the only recipient. I learned to use "pilerget" (perfekt!), and the mail was actually saved in piler.
While using the web-frontend, these mails do not show up - in fact I can only see two mails received today.
Btw, what is the point of the "rcpt" table? Is it supposed to add a recipient for each mail? (Except for the ID, most of the other information is quite redundant.) No criticism intended - just wondering... ;)
-
repo owner The purpose of the rcpt table is to store the recipients of the email, including bcc stuff. Since an email may have several recipients, I decided to store them in a separate table, rather to add a blob(8192) not null type column to metadata.
Please do the following. Save the message with pilerget, eg. pilerget 40000.... > /tmp/1.eml Then run pilertest to parse it, eg. pilertest /tmp/1.eml. It should parse the email, and check the 'to:' line if it seems correct, and includes all recipients' emails.
Btw. do you have other, recent "duplicate entry" error emails? The verp stuff should not cause any trouble like that, since it merely adds another email address (coming from the mail envelope).
-
Account Deleted My rcpt table looks sth. like:
id | to | domain -------------------------------------------------------------- 29001 | test.user@example.org | example.org 29002 | test.user@example.org | example.org 29003 | test.user@example.org | example.org 29004 | user2@example.org | example.org
I was wondering if the redundant data was intended, but if ID is the internal message id, I see what you are getting at.
pilertest does not report any error. Here is the to: line:
to: *user test test.user@example.org test user example org (example.org )*
I get this "duplicate entry" error on most of the mails now. I don't see anything obvious about them, e.g. sometimes those mails have multiple recipients, sometimes there's only one. I do notice, that while the mails are stored in piler (I know to use pilerget now), they do not show up in the web interface.
-
repo owner I've just checked the demo site, and indeed, it also produces the "Duplicate entry" error messages for some messages, however those messages are spam with invalid or missing message-id fields. Other than that I can't see an error, though those emails don't use verp addressing, just plain old aaa@aaa.fu like addresses. I'll add some messages with verp addressing as well. We'll see.
For the web gui, check if you have /var/piler/sphinx files, and they are several MBs large (or even bigger).
-
Account Deleted Alright, I switched back to "always_bcc" and I don't get any "Duplicate entry" error. Those mails also appear in the WebUI. (sphinx indexing seems to work)
1.) VERP addressing is responsible for the "Duplicate entry" error message (99,9% of all mails) 2.) in case the error is thrown, the respective message does not show in the WebUI
I hope this error description is of any help! Sorry for giving you a hard time ;)
-
repo owner I'll give it a deeper test, I think it can be fixed. I'll let you know when it's ok.
-
repo owner - attached parser.c
-
repo owner Please overwrite src/parser.c with the attached file, and recompile piler. I think it will solve the "duplicate entry" issue. If the message is actually archived, and you can retrieve it with pilerget, then we will access the message in the gui. But first, let's fix the "duplicate entry" bug.
-
Account Deleted That seems to have done the trick!
No more "duplicate entry" logs, and all new mails accessable in the webUI.
Any chance I can make those "inbetween" mails visible in the webUI as well? I tried pilerexport -> pilerimport but didn't get any changes as all messages had been archived already.
-
repo owner OK, 1 step closer. The cause of the duplicate entry issue was a minor bug that made the given email address to present twice in the "to" buffer, and trying to insert it to the rcpt table twice. However I don't think it prevents the gui to access the given email, because I think it's stored, archived by piler, and you could retrieve it by pilerget.
To check things, please get the serial id of the message:
mysql> select id from metadata where piler_id='4000....';
Now check if it's know by sphinx:
$ mysql -h 127.0.0.1 -P 9306 mysql> select * from main1,dailydelta1,delta1 where id=12345;
where 12345 is the number you got from the metadata table.
-
Account Deleted The metadata is filled with all information. The sphinx database does not have any knowledge of the mail-id; an empty set is returned.
If I understand the mechanism correctly, then there might have been an error preventing the data to be written into sph_index. Seems to be fixed now anyways.
I would delete all sphinx files and run the reindex utility?
-
repo owner That's exactly what happened. Try the following: instead reindexing everything, reindex only the problematic messages:
cd /tmp reindex -f 12345 -t 54321
where 12345 is the first affected id, and 54321 is the last affected id. After that you should be able to see all emails in the gui.
-
Account Deleted Alright, learned a lot!
There is one minor thing I am not sure whether it should be discarded; as you stated before, the use of VERP scheme requires each message to be delivered to piler as many times as there are recipients to the original mail. For all I can see this will continuously generate "Duplicate entry" errors as the second (third, etc.) recipient has already been logged in the "rcpt" table from parsing the first recipient's mail. This of course is not the case if we're talking about proper bcc recipients. Then the original mail should have no knowledge of the bcc line.
Regarding the documentation; you call this type of addressing Variable Envelope Return Path (VERP). VERP is actually a special application of sub-addressing, commonly used for mailing lists (or rather for bounces thereof). Replacing the "@" with "=" seems to be an old qmail convention.
I had a good read at Sieve Extensions and directly at the Postfix page recipient_delimiter and recipient_bcc_maps
Finally, realizing that someone else might use sub-addressing for his own purpose, I changed the bcc_map:
# /etc/postfix/bcc_map # /^([^+]+).*@(.*)$/ archive+$1=$2@myarchive.local
-
repo owner I think I'll fix the message processing flow not to return an error in case of a duplicate entry issue. Unfortunately it's not easy to prevent the duplicate error if you deliver the same actual message many times to piler.
-
repo owner - changed status to resolved
I think the issue is resolved.
-
hello.the attached file is error.
error:conflicting types for ‘split’;
thank.
-
repo owner Could you elaborate the problem? Which attached file is 'error'? Why is that? Where can you see the "error:conflicting types for ‘split’;" message?
-
parser.c
line 638,
An error occurred when running 'make'
-
Is there a config-option in piler.conf, which i must enable to use the feature "Sub-Addressing"? I have two problems:
- If i enable "archive_only_mydomains=1", then piler discard the emails
- If i disable "archive_only_mydomains=0", piler will accept this message. But the email will not assign to the user with the bcc-address. So they cant view the email in die WebUI.
-
repo owner Please elaborate the problem, perhaps you may give a detailed example, and what you mean by 'sub-addressing'.
-
i mean this feature, was discuss in this thread ;-) If i send a mail, who has user@domain.de in BCC, it will send by postfix to archive+user=domain.de@host.example.com. But piler discard this mail, if i have the option "archive_only_mydomains=1" on. Allthough "domain.de" is under the Domains added.
If i disable the option, then piler accept this email, but dont assign to the user "user@domain.de". So the User cant see this email in his WebUI-Account.
-
repo owner What piler version do you have?
-
1.3.0
-
Do you need any more Informations from me? Can you replicate my problem?
-
repo owner I couldn't investigate the issue yet. I need more time.
-
repo owner - changed status to open
-
Hello. I have the same problem as Thomas Czarnetzki.
Piler v 1.3.1
I can see "to" and "bcc" email going towards mailpiler in this form
MAIL FROM:archive+sender=domain.com@myarchive.local
got: RCPT TO:archive+to_user=domain.de@myarchive.local
got: RCPT TO:archive+bcc_user=domain.co.uk@myarchive.local
but i cannot see the email in bcc_user@domain.co.uk, only to_user@domain.de
mailpiler logs "added 1 recipients". Shouldn't it be 2?
Thanks
-
repo owner Try setting process_rcpt_to_addresses=1 in piler.conf
-
Thanks Janos. It's working!
-
repo owner Great news!
-
repo owner - changed status to resolved
I conclude that the issue is solved. If you think otherwise, Thomas, please reopen it.
-
Hi Janos. One last thing about BCC.
MAIL FROM:archive+sender=domain.com@myarchive.local
got: RCPT TO:archive+to_user=domain.de@myarchive.local
got: RCPT TO:archive+bcc_user=domain.co.uk@myarchive.local
If I login as auditor I can see this email:
MAIL FROM:sender@domain.com
RCPT TO:to_user@domain.de
and next to the recipient an icon (group of people). If I move the mouse on the icon I can see both rcpt and bcc email. Everything is correct
But... if I login with sender account (regular user) I cannot see that group icon so the sender user doesn't know if there was a bcc user in the original email. However if I search bcc_user@domain.co.uk I can find the email with sender sender@domain.com and recipient to_user@domain.de.
Is it possible to have that group icon even with regular user (or something that could show the bcc email)
Thanks
-
repo owner hmm, it seems that the sender envelope address is also verp-like. However, piler reads the mail header From: line. What is it like?
-
Hi. I think that the problem is that i login only as sender, not as sender@domain.com. I had to modify this file to view the emails(as seen in another post):
/var/www/piler/model/user/auth.php
//$emails = array($username);
emails = array($username . '@domain.com');
-
repo owner OK, great.
-
Dear Janos,
i have testet it again (with the parameter: process_rcpt_to_addresses=1), but nothing has changed. If my postfix send a mail in the format: "archive+to_user=domain.de@myarchive.local" to piler, AND archive_only_mydomains=1 then piler dont accept the mail:
Dec 12 13:23:35 xxx piler[11519]: xxx: discarding: not on mydomains, from=xxx@xxx, message-id=xxx@xxx
If i set "archive_only_mydomains=0", then it will be accepted.
-
repo owner It seems that 'mydomain' check is done before the verp address fix. I assume you use the prefork version of piler. What version is it?
Btw. I've just noticed that when I rewrote piler using the epoll model I forgot to add the mydomain support. I'll add it again.
-
[root@piler]# piler -v 1.3.0 build 955
-
repo owner - changed status to open
-
any update about this issue?
-
repo owner Not yet. I'll try to make it in a few days.
-
repo owner A question: where is the verp email address is specified? In the email headers or in the mail from or rcpt to (=envelope) addresses?
-
repo owner Anyway if the verp address is in the mail headers (To or Cc) try the following: edit src/parser.c, and find the last strtolower(puf); line (around the 731st line), and add the following line after it:
if(strchr(puf, '@')) extract_verp_address(puf);
Then recompile, and update the libpiler.so* files. Let me know how it goes.
-
I have modify the code and debug it (add a syslog line), but it does't work. I have send the following Mail: "To: address@domain1.tld, bcc: address@domain2.tld" Piler Is Running for domain2.tld
Feb 20 06:53:38 mailarchive piler[26177]: puf is: address@domain1.tld Feb 20 06:53:38 mailarchive piler[26177]: ...: discarding: not on mydomains, from=xxx@xxx, message-id=xxx@xxx
Of course "domain2.tld" is in mydomains. It looks like, that piler doen't search the email-address at the right place.
-
Additional Info:
Feb 20 07:10:54 mailarchive piler[26844]: xxx: got: RCPT TO:archive+address=domain2.tld@xxx.tld
so the right place to search is "RCPT TO"
-
repo owner OK. Get the message from the archive, eg pilerget 4000000......xxxxx > /tmp/1.eml, then run pilertest on it, and verify that domain2.tld is recognized in the recipient list.
-
no, in the mail-source is the Address not included. Because it is a "bcc" Address. it will not show in the source. I think the verp-address-format (RCPT TO) will only be available within the smtp-communication.
-
repo owner It's odd: if process_rcpt_to_addresses=1 is set in piler.conf, then the parser should process the envelope addresses, and it should find the fixed address in domain2.tld.
-
i have set this parameter allready: "process_rcpt_to_addresses=1"
-
repo owner OK, I think I've nailed it finally. Edit src/parser.c, then fix the parse_message() function as follows:
There's an if condition around the 48th line:
if(state.tolen < MAXBUFSIZE-len-1){
Add the following before the above line:
if(is_email_address_on_my_domains(puf, data) == 1) sdata->internal_recipient = 1;
Then recompile, and check again.
-
first test looks fine! I will do our full checks asap, and give you a finally feedback shortly. btw: the first fix generate the follwing warning at compile:
parser.c: In function ‘parse_line’: parser.c:734:10: warning: implicit declaration of function ‘extract_verp_address’ [-Wimplicit-function-declaration] if(strchr(puf, '@')) extract_verp_address(puf);
Is the first fix also need for the bug, or only the second?
-
repo owner Only the second.
-
repo owner - changed status to resolved
I conclude the issue resolved.
-
Hi! Apparantly this is missing from the documentation: https://www.mailpiler.org/wiki/current:index
-
repo owner Thanks for noticing. I’ve updated https://www.mailpiler.org/wiki/current:installation. Let me know if it needs more clarification.
- Log in to comment
The purpose of the X-Envelope-To prepended header is to track all possible recipients of the message. You are right, that if X-Envelope-To is present, then it may reveal even bcc recipients as well. Unfortunately I can't think of another way to solve this issue, because without it, postfix won't copy the bcc addresses, and you have to rely on the mail header To/Cc lines. So yes, that's a trade-off.
Regarding the "sub-addressing" how would this work, and how do you configure it?