[Postfix related]: x-envelope-to reveals bcc

Issue #335 resolved
Former user created an issue

When using postfix, prepending the X-Envelope-To header will reveal all BCC recipients.

I might not have found the right place for prepending the header yet (smtpd_recipient_restrictions). Please correct me if there is another way!

As an alternative one could use recipient_bcc_maps and allow sub-addressing to take care of the original envelope recipient (which would be part of the sub-address).

E.g. my route to pile is archive@archive.local, sub-addressing of the original recipient alex@example.org would be archive+alex=example.org@archive.local.

For this to work the piler parser needs to be aware of the type of sub-addressing used.

Alex

Comments (68)

  1. Janos SUTO repo owner

    The purpose of the X-Envelope-To prepended header is to track all possible recipients of the message. You are right, that if X-Envelope-To is present, then it may reveal even bcc recipients as well. Unfortunately I can't think of another way to solve this issue, because without it, postfix won't copy the bcc addresses, and you have to rely on the mail header To/Cc lines. So yes, that's a trade-off.

    Regarding the "sub-addressing" how would this work, and how do you configure it?

  2. Former user Account Deleted

    You would use "recipient_bcc_maps" instead of "always_bcc"

    # /etc/postfix/main.cf
    #
    recipient_delimiter = +
    recipient_bcc_maps = pcre:/etc/postfix/bcc_map
    

    The map would look something like:

    # /etc/postfix/bcc_map
    #
    /^(.*)@(.*)$/      archive+$1=$2@myarchive.local
    

    (given that you have a transport for "myarchive.local")

    Any given envelope address would rewrite to a bcc-recipient in the form of
    archive+<name>=<domain>@myarchive.local

    For example an email delivered to "test@example.org" would be bcc'ed to
    archive+test=example.org@myarchive.local

    The "+" delimiter starts the sub-address and the "@" of the original recipient address is replaced by "=" as only one "@" is obviously allowed in email addresses.

    Your parser could for instance know about the sub-address delimiter (config option!?) and about the "@" replacement character in order to form a valid address.

    I found a statement more than a decade old by Wietse Venema about him wanting to address that issue - don't know what became of it.
    Old Post by Wietse

    There might be another way by now that I don't know of.

  3. Janos SUTO repo owner

    Thanks for the hint, I'll check it out. However note that processing the same message several times (ie. one occasion for a single recipient) is a bit wasting of resources. With 10+ recipients much waste.

  4. Janos SUTO repo owner

    OK, check out the latest master branch, see the Download link above. Let me know how it works for you.

  5. Former user Account Deleted

    Wow, that was a prompt reply!

    I just checked it, yet it doesn't quite work (yet)...

    Jun 30 17:39:25 mail piler[21633]: 4000000053b184b635237a3c00a8dc181426: from=testuser@example.com, size=43726/6208, attachments=2, reference=, message-id=<dcbe5815-1223-44cf-8846-f9567a5329ee@WG-EX01.somecorp.local>, retention=2557, delay=0.08, delays=0.07/0.00/0.01/0.00/0.00/0.00, status=stored
    Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=example-1.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0,status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <archiv+testuser@example-1.org>)
    Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=example-2.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0, status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <archiv+testuser@example-1.org>)
    Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=xn--ltzimmer-n4a.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0, status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <archiv+testuser@example-1.org>)
    

    The message I sent:

    • From: testuser@example.com
    • To: testuser@example.com.
    • BCC: testuser@example-1.org, testuser@example-2.org, testuser@lötzimmer.org

    In Piler I cannot see any BCC information, I only see a single mail from testuser@example.com to testuser@example.com

    Piler did get the message but instead of finding the additional Information in the X-Envelope-to header, it would have to add the recipient information passed by the envelope address.

  6. Former user Account Deleted

    That must've been quite some changes in your commit...

    We are almost there - it works fine but now I always get the archive-mail address (archiv@myarchive.local) as the first recipient in piler (even when using X-Envelope-To).

  7. Former user Account Deleted

    It was not "myarchive.local" ;)

    Changed it, and it works now!
    Let me test for another week and if I don't find any problems you might consider this an alternative to the current postfix header prepending way.

  8. Former user Account Deleted

    Strangest things are happening...
    After having been away for a few days I checked my queue and noticed quite a few mails stuck, not being delivered to piler.

    Most of them end as below (piler in debug mode):

    Jul  7 15:17:51 mail postfix/qmgr[24140]: 69EE58400EB: from=<notification+aqqkkkgx@facebookmail.com>, size=12569, nrcpt=2 (queue active)
    Jul  7 15:17:51 mail piler[24041]: connection from 1.2.3.4
    Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: sent: 220 myarchive.local ESMTP#015
    Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: got: EHLO mx.example.com
    Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: sent: 250-myarchive.local#015#012250-PIPELINING#015#012250-SIZE#015#012250 8BITMIME#015
    Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: got: MAIL FROM:<notification+aqqkkkgx@facebookmail.com> SIZE=12569 BODY=7BIT
    Jul  7 15:17:51 mail piler[24222]: forked a child (pid: 24222)
    Jul  7 15:17:51 mail piler[24222]: child (pid: 24222) started main()
    Jul  7 15:17:51 mail postfix/smtp[24172]: 69EE58400EB: to=<archiv+test.user=example.org@myarchive.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=319443, delays=319443/0/0/0, dsn=4.4.2, status=deferred (lost connection with 127.0.0.1[127.0.0.1] while sending MAIL FROM)
    

    Another thing I noticed is the following (piler in normal mode):

    Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: mysql_stmt_execute error: *Duplicate entry '288736-test.user@example.org' for key 'id'*
    Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: from=sender@somesender.org, size=42616/6960, attachments=1, reference=<C5F711ED0309444ABD2404B33220CCC3F8073D6717@pamail01.gvm.somesender.org>, message-id=<C5F711ED0309444ABD2404B33220CCC3F8073D672F@pamail01.gvm.somesender.org>, retention=2557, delay=0.08, delays=0.06/0.00/0.02/0.00/0.00/0.00, status=stored
    Jul  1 16:35:00 mail postfix/smtp[9098]: 460968400C1: to=<archiv+test.user=example.org@wissler-archiv.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.3, delays=0.05/0/0/0.25, dsn=2.0.0, status=sent (250 Ok 4000000053b2c71e13f0ce1400f725273a52 <test.user@example.org>)
    Jul  1 16:35:00 mail postfix/qmgr[9096]: 460968400C1: removed
    
  9. Janos SUTO repo owner

    Yes, I made a text processing error. Please update src/misc.c with the attached file, and recompile piler. It should fix the issue.

  10. Former user Account Deleted

    Again, thanks for the amazingly quick response!

    I recompiled yesterday night and the interruption of the mail delivery has vanished.

    Yet, I still get the second error about mail duplication. I have the feeling those messages do not show up in the archive:

    Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: mysql_stmt_execute error: *Duplicate entry '288736-test.user@example.org' for key 'id'*
    Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: from=sender@somesender.org, size=42616/6960, attachments=1, reference=<C5F711ED0309444ABD2404B33220CCC3F8073D6717@pamail01.gvm.somesender.org>, message-id=<C5F711ED0309444ABD2404B33220CCC3F8073D672F@pamail01.gvm.somesender.org>, retention=2557, delay=0.08, delays=0.06/0.00/0.02/0.00/0.00/0.00, status=stored
    Jul  1 16:35:00 mail postfix/smtp[9098]: 460968400C1: to=<archiv+test.user=example.org@wissler-archiv.local>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.3, delays=0.05/0/0/0.25, dsn=2.0.0, status=sent (250 Ok 4000000053b2c71e13f0ce1400f725273a52 <test.user@example.org>)
    Jul  1 16:35:00 mail postfix/qmgr[9096]: 460968400C1: removed
    
  11. Janos SUTO repo owner

    Please check if you can retrieve the message with "pilerget 4000000053b2c71e13f0ce1400f725273a52". Also I'd like to see "grep C5F711ED0309444ABD2404B33220CCC3F8073D672F@pamail01.gvm.somesender.org /var/log/maillog" output to see how many emails with the given message-id you received.
    And finally check the rcpt table in the piler database whether it has all recipients. To do this execute the following query:

    mysql> select * from rcpt where id=(select id from metadata where piler_id='4000000053b2c71e13f0ce1400f725273a52');

    I suspect that 288736-test.user@example.org was added earlier.

  12. Former user Account Deleted

    My observation is that this error only appears after applying the VERP-Patches last week.

    Above mail was received once. And the query returns exactly the only recipient.
    I learned to use "pilerget" (perfekt!), and the mail was actually saved in piler.

    While using the web-frontend, these mails do not show up - in fact I can only see two mails received today.

    Btw, what is the point of the "rcpt" table? Is it supposed to add a recipient for each mail? (Except for the ID, most of the other information is quite redundant.)
    No criticism intended - just wondering... ;)

  13. Janos SUTO repo owner

    The purpose of the rcpt table is to store the recipients of the email, including bcc stuff. Since an email may have several recipients, I decided to store them in a separate table, rather to add a blob(8192) not null type column to metadata.

    Please do the following. Save the message with pilerget, eg. pilerget 40000.... > /tmp/1.eml
    Then run pilertest to parse it, eg. pilertest /tmp/1.eml. It should parse the email, and check the 'to:' line if it seems correct, and includes all recipients' emails.

    Btw. do you have other, recent "duplicate entry" error emails? The verp stuff should not cause any trouble like that, since it merely adds another email address (coming from the mail envelope).

  14. Former user Account Deleted

    My rcpt table looks sth. like:

    id         |   to                                 |  domain
    --------------------------------------------------------------
    29001  | test.user@example.org | example.org
    29002  | test.user@example.org | example.org
    29003  | test.user@example.org | example.org
    29004  | user2@example.org      | example.org
    

    I was wondering if the redundant data was intended, but if ID is the internal message id, I see what you are getting at.

    pilertest does not report any error. Here is the to: line:

    to: *user test  test.user@example.org test user example org  (example.org )*
    

    I get this "duplicate entry" error on most of the mails now. I don't see anything obvious about them, e.g. sometimes those mails have multiple recipients, sometimes there's only one.
    I do notice, that while the mails are stored in piler (I know to use pilerget now), they do not show up in the web interface.

  15. Janos SUTO repo owner

    I've just checked the demo site, and indeed, it also produces the "Duplicate entry" error messages for some messages, however those messages are spam with invalid or missing message-id fields. Other than that I can't see an error, though those emails don't use verp addressing, just plain old aaa@aaa.fu like addresses.
    I'll add some messages with verp addressing as well. We'll see.

    For the web gui, check if you have /var/piler/sphinx files, and they are several MBs large (or even bigger).

  16. Former user Account Deleted

    Alright, I switched back to "always_bcc" and I don't get any "Duplicate entry" error.
    Those mails also appear in the WebUI. (sphinx indexing seems to work)

    1.) VERP addressing is responsible for the "Duplicate entry" error message (99,9% of all mails)
    2.) in case the error is thrown, the respective message does not show in the WebUI

    I hope this error description is of any help!
    Sorry for giving you a hard time ;)

  17. Janos SUTO repo owner

    Please overwrite src/parser.c with the attached file, and recompile piler. I think it will solve the "duplicate entry" issue. If the message is actually archived, and you can retrieve it with pilerget, then we will access the message in the gui. But first, let's fix the "duplicate entry" bug.

  18. Former user Account Deleted

    That seems to have done the trick!

    No more "duplicate entry" logs, and all new mails accessable in the webUI.

    Any chance I can make those "inbetween" mails visible in the webUI as well? I tried pilerexport -> pilerimport but didn't get any changes as all messages had been archived already.

  19. Janos SUTO repo owner

    OK, 1 step closer. The cause of the duplicate entry issue was a minor bug that made the given email address to present twice in the "to" buffer, and trying to insert it to the rcpt table twice.
    However I don't think it prevents the gui to access the given email, because I think it's stored, archived by piler, and you could retrieve it by pilerget.

    To check things, please get the serial id of the message:

    mysql> select id from metadata where piler_id='4000....';

    Now check if it's know by sphinx:

    $ mysql -h 127.0.0.1 -P 9306
    mysql> select * from main1,dailydelta1,delta1 where id=12345;

    where 12345 is the number you got from the metadata table.

  20. Former user Account Deleted

    The metadata is filled with all information.
    The sphinx database does not have any knowledge of the mail-id; an empty set is returned.

    If I understand the mechanism correctly, then there might have been an error preventing the data to be written into sph_index. Seems to be fixed now anyways.

    I would delete all sphinx files and run the reindex utility?

  21. Janos SUTO repo owner

    That's exactly what happened. Try the following: instead reindexing everything, reindex only the problematic messages:

    cd /tmp
    reindex -f 12345 -t 54321

    where 12345 is the first affected id, and 54321 is the last affected id. After that you should be able to see all emails in the gui.

  22. Former user Account Deleted

    Alright, learned a lot!

    There is one minor thing I am not sure whether it should be discarded;
    as you stated before, the use of VERP scheme requires each message to be delivered to piler as many times as there are recipients to the original mail.
    For all I can see this will continuously generate "Duplicate entry" errors as the second (third, etc.) recipient has already been logged in the "rcpt" table from parsing the first recipient's mail.
    This of course is not the case if we're talking about proper bcc recipients. Then the original mail should have no knowledge of the bcc line.

    Regarding the documentation;
    you call this type of addressing Variable Envelope Return Path (VERP). VERP is actually a special application of sub-addressing, commonly used for mailing lists (or rather for bounces thereof).
    Replacing the "@" with "=" seems to be an old qmail convention.

    I had a good read at Sieve Extensions and directly at the Postfix page recipient_delimiter and recipient_bcc_maps

    Finally, realizing that someone else might use sub-addressing for his own purpose, I changed the bcc_map:

    # /etc/postfix/bcc_map
    #
    /^([^+]+).*@(.*)$/      archive+$1=$2@myarchive.local
    
  23. Janos SUTO repo owner

    I think I'll fix the message processing flow not to return an error in case of a duplicate entry issue. Unfortunately it's not easy to prevent the duplicate error if you deliver the same actual message many times to piler.

  24. Janos SUTO repo owner

    Could you elaborate the problem? Which attached file is 'error'? Why is that? Where can you see the "error:conflicting types for ‘split’;" message?

  25. Thomas Czarnetzki

    Is there a config-option in piler.conf, which i must enable to use the feature "Sub-Addressing"?
    I have two problems:

    1. If i enable "archive_only_mydomains=1", then piler discard the emails
    2. If i disable "archive_only_mydomains=0", piler will accept this message. But the email will not assign to the user with the bcc-address. So they cant view the email in die WebUI.
  26. Janos SUTO repo owner

    Please elaborate the problem, perhaps you may give a detailed example, and what you mean by 'sub-addressing'.

  27. Thomas Czarnetzki

    i mean this feature, was discuss in this thread ;-)
    If i send a mail, who has user@domain.de in BCC, it will send by postfix to archive+user=domain.de@host.example.com.
    But piler discard this mail, if i have the option "archive_only_mydomains=1" on. Allthough "domain.de" is under the Domains added.

    If i disable the option, then piler accept this email, but dont assign to the user "user@domain.de". So the User cant see this email in his WebUI-Account.

  28. Mich80

    Hi Janos. One last thing about BCC.

    MAIL FROM:archive+sender=domain.com@myarchive.local

    got: RCPT TO:archive+to_user=domain.de@myarchive.local

    got: RCPT TO:archive+bcc_user=domain.co.uk@myarchive.local

    If I login as auditor I can see this email:

    MAIL FROM:sender@domain.com

    RCPT TO:to_user@domain.de

    and next to the recipient an icon (group of people). If I move the mouse on the icon I can see both rcpt and bcc email. Everything is correct

    But...
    if I login with sender account (regular user) I cannot see that group icon so the sender user doesn't know if there was a bcc user in the original email.
    However if I search bcc_user@domain.co.uk I can find the email with sender sender@domain.com and recipient to_user@domain.de.

    Is it possible to have that group icon even with regular user (or something that could show the bcc email)

    Thanks

  29. Janos SUTO repo owner

    hmm, it seems that the sender envelope address is also verp-like. However, piler reads the mail header From: line. What is it like?

  30. Mich80

    Hi. I think that the problem is that i login only as sender, not as sender@domain.com.
    I had to modify this file to view the emails(as seen in another post):

    /var/www/piler/model/user/auth.php

    //$emails = array($username);

    emails = array($username . '@domain.com');

  31. Thomas Czarnetzki

    Dear Janos,

    i have testet it again (with the parameter: process_rcpt_to_addresses=1), but nothing has changed.
    If my postfix send a mail in the format: "archive+to_user=domain.de@myarchive.local" to piler, AND archive_only_mydomains=1 then piler dont accept the mail:

    Dec 12 13:23:35 xxx piler[11519]: xxx: discarding: not on mydomains, from=xxx@xxx, message-id=xxx@xxx

    If i set "archive_only_mydomains=0", then it will be accepted.

  32. Janos SUTO repo owner

    It seems that 'mydomain' check is done before the verp address fix. I assume you use the prefork version of piler. What version is it?

    Btw. I've just noticed that when I rewrote piler using the epoll model I forgot to add the mydomain support. I'll add it again.

  33. Janos SUTO repo owner

    A question: where is the verp email address is specified? In the email headers or in the mail from or rcpt to (=envelope) addresses?

  34. Janos SUTO repo owner

    Anyway if the verp address is in the mail headers (To or Cc) try the following: edit src/parser.c, and find the last strtolower(puf); line (around the 731st line), and add the following line after it:

    if(strchr(puf, '@')) extract_verp_address(puf);
    

    Then recompile, and update the libpiler.so* files. Let me know how it goes.

  35. Thomas Czarnetzki

    I have modify the code and debug it (add a syslog line), but it does't work.
    I have send the following Mail: "To: address@domain1.tld, bcc: address@domain2.tld" Piler Is Running for domain2.tld

    Feb 20 06:53:38 mailarchive piler[26177]: puf is: address@domain1.tld
    Feb 20 06:53:38 mailarchive piler[26177]: ...: discarding: not on mydomains, from=xxx@xxx, message-id=xxx@xxx

    Of course "domain2.tld" is in mydomains. It looks like, that piler doen't search the email-address at the right place.

  36. Janos SUTO repo owner

    OK. Get the message from the archive, eg pilerget 4000000......xxxxx > /tmp/1.eml, then run pilertest on it, and verify that domain2.tld is recognized in the recipient list.

  37. Thomas Czarnetzki

    no, in the mail-source is the Address not included. Because it is a "bcc" Address. it will not show in the source.
    I think the verp-address-format (RCPT TO) will only be available within the smtp-communication.

  38. Janos SUTO repo owner

    It's odd: if process_rcpt_to_addresses=1 is set in piler.conf, then the parser should process the envelope addresses, and it should find the fixed address in domain2.tld.

  39. Janos SUTO repo owner

    OK, I think I've nailed it finally. Edit src/parser.c, then fix the parse_message() function as follows:

    There's an if condition around the 48th line:

    if(state.tolen < MAXBUFSIZE-len-1){
    

    Add the following before the above line:

    if(is_email_address_on_my_domains(puf, data) == 1) sdata->internal_recipient = 1;
    

    Then recompile, and check again.

  40. Thomas Czarnetzki

    first test looks fine! I will do our full checks asap, and give you a finally feedback shortly.
    btw: the first fix generate the follwing warning at compile:

    parser.c: In function ‘parse_line’:
    parser.c:734:10: warning: implicit declaration of function ‘extract_verp_address’ [-Wimplicit-function-declaration]
    if(strchr(puf, '@')) extract_verp_address(puf);

    Is the first fix also need for the bug, or only the second?

  41. Log in to comment