[Postfix related]: x-envelope-to reveals bcc

[Former user]

When using postfix, prepending the X-Envelope-To header will reveal all BCC recipients.

I might not have found the right place for prepending the header yet (smtpd_recipient_restrictions). Please correct me if there is another way!

As an alternative one could use recipient_bcc_maps and allow sub-addressing to take care of the original envelope recipient (which would be part of the sub-address).

E.g. my route to pile is [e-mail], sub-addressing of the original recipient [e-mail] would be archive+alex=[e-mail].

For this to work the piler parser needs to be aware of the type of sub-addressing used.

Alex

[Janos SUTO]

The purpose of the X-Envelope-To prepended header is to track all possible recipients of the message. You are right, that if X-Envelope-To is present, then it may reveal even bcc recipients as well. Unfortunately I can't think of another way to solve this issue, because without it, postfix won't copy the bcc addresses, and you have to rely on the mail header To/Cc lines. So yes, that's a trade-off.

Regarding the "sub-addressing" how would this work, and how do you configure it?

[Former user]

You would use "recipient_bcc_maps" instead of "always_bcc"

# /etc/postfix/main.cf
#
recipient_delimiter = +
recipient_bcc_maps = pcre:/etc/postfix/bcc_map

The map would look something like:

# /etc/postfix/bcc_map
#
/^(.*)@(.*)$/      archive+$1=$[e-mail]

(given that you have a transport for "myarchive.local")

Any given envelope address would rewrite to a bcc-recipient in the form of archive+<name>=<domain>@myarchive.local

For example an email delivered to "[e-mail]" would be bcc'ed to archive+test=[e-mail]

The "+" delimiter starts the sub-address and the "@" of the original recipient address is replaced by "=" as only one "@" is obviously allowed in email addresses.

Your parser could for instance know about the sub-address delimiter (config option!?) and about the "@" replacement character in order to form a valid address.

I found a statement more than a decade old by Wietse Venema about him wanting to address that issue - don't know what became of it. Old Post by Wietse

There might be another way by now that I don't know of.

[Janos SUTO]

Thanks for the hint, I'll check it out. However note that processing the same message several times (ie. one occasion for a single recipient) is a bit wasting of resources. With 10+ recipients much waste.

[Janos SUTO]

OK, check out the latest master branch, see the Download link above. Let me know how it works for you.

[Former user]

Wow, that was a prompt reply!

I just checked it, yet it doesn't quite work (yet)...

Jun 30 17:39:25 mail piler[21633]: 4000000053b184b635237a3c00a8dc181426: from=[e-mail], size=43726/6208, attachments=2, reference=, message-id=<[e-mail]>, retention=2557, delay=0.08, delays=0.07/0.00/0.01/0.00/0.00/0.00, status=stored
Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=[e-mail]>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0,status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <[e-mail]>)
Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=[e-mail]>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0, status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <[e-mail]>)
Jun 30 17:39:25 mail postfix/smtp[22211]: C0FFA8400A5: to=<archiv+testuser=[e-mail]>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.51, delays=0.1/0.01/0/0.4, dsn=2.0.0, status=sent (250 Ok 4000000053b184b635237a3c00a8dc181426 <[e-mail]>)

The message I sent:

• From: [e-mail]
• To: [e-mail].
• BCC: [e-mail], [e-mail], testuser@lötzimmer.org

In Piler I cannot see any BCC information, I only see a single mail from [e-mail] to [e-mail]

Piler did get the message but instead of finding the additional Information in the X-Envelope-to header, it would have to add the recipient information passed by the envelope address.

[Janos SUTO]

OK, try again. I think I've nailed it this time.

[Former user]

That must've been quite some changes in your commit...

We are almost there - it works fine but now I always get the archive-mail address ([e-mail]) as the first recipient in piler (even when using X-Envelope-To).

[Janos SUTO]

What's the "hostid" in piler.conf?

[Former user]

It was not "myarchive.local" ;)

Changed it, and it works now! Let me test for another week and if I don't find any problems you might consider this an alternative to the current postfix header prepending way.

[Janos SUTO]

OK, take your time. I hope it will be ok.

[Former user]

Strangest things are happening... After having been away for a few days I checked my queue and noticed quite a few mails stuck, not being delivered to piler.

Most of them end as below (piler in debug mode):

Jul  7 15:17:51 mail postfix/qmgr[24140]: 69EE58400EB: from=<[e-mail]>, size=12569, nrcpt=2 (queue active)
Jul  7 15:17:51 mail piler[24041]: connection from 1.2.3.4
Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: sent: 220 myarchive.local ESMTP#015
Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: got: EHLO mx.example.com
Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: sent: 250-myarchive.local#015#012250-PIPELINING#015#012250-SIZE#015#012250 8BITMIME#015
Jul  7 15:17:51 mail piler[24041]: 4000000053ba9e0923f715ac00d11c661cae: got: MAIL FROM:<[e-mail]> SIZE=12569 BODY=7BIT
Jul  7 15:17:51 mail piler[24222]: forked a child (pid: 24222)
Jul  7 15:17:51 mail piler[24222]: child (pid: 24222) started main()
Jul  7 15:17:51 mail postfix/smtp[24172]: 69EE58400EB: to=<archiv+test.user=[e-mail]>, relay=127.0.0.1[127.0.0.1]:10026, delay=319443, delays=319443/0/0/0, dsn=4.4.2, status=deferred (lost connection with 127.0.0.1[127.0.0.1] while sending MAIL FROM)

Another thing I noticed is the following (piler in normal mode):

Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: mysql_stmt_execute error: *Duplicate entry '[e-mail]' for key 'id'*
Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: from=[e-mail], size=42616/6960, attachments=1, reference=<[e-mail]>, message-id=<[e-mail]>, retention=2557, delay=0.08, delays=0.06/0.00/0.02/0.00/0.00/0.00, status=stored
Jul  1 16:35:00 mail postfix/smtp[9098]: 460968400C1: to=<archiv+test.user=[e-mail]>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.3, delays=0.05/0/0/0.25, dsn=2.0.0, status=sent (250 Ok 4000000053b2c71e13f0ce1400f725273a52 <[e-mail]>)
Jul  1 16:35:00 mail postfix/qmgr[9096]: 460968400C1: removed

[Janos SUTO]

Yes, I made a text processing error. Please update src/misc.c with the attached file, and recompile piler. It should fix the issue.

[Former user]

Again, thanks for the amazingly quick response!

I recompiled yesterday night and the interruption of the mail delivery has vanished.

Yet, I still get the second error about mail duplication. I have the feeling those messages do not show up in the archive:

Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: mysql_stmt_execute error: *Duplicate entry '[e-mail]' for key 'id'*
Jul  1 16:35:00 mail piler[9033]: 4000000053b2c71e13f0ce1400f725273a52: from=[e-mail], size=42616/6960, attachments=1, reference=<[e-mail]>, message-id=<[e-mail]>, retention=2557, delay=0.08, delays=0.06/0.00/0.02/0.00/0.00/0.00, status=stored
Jul  1 16:35:00 mail postfix/smtp[9098]: 460968400C1: to=<archiv+test.user=[e-mail]>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.3, delays=0.05/0/0/0.25, dsn=2.0.0, status=sent (250 Ok 4000000053b2c71e13f0ce1400f725273a52 <[e-mail]>)
Jul  1 16:35:00 mail postfix/qmgr[9096]: 460968400C1: removed

[Janos SUTO]

Please check if you can retrieve the message with "pilerget 4000000053b2c71e13f0ce1400f725273a52". Also I'd like to see "grep [e-mail] /var/log/maillog" output to see how many emails with the given message-id you received. And finally check the rcpt table in the piler database whether it has all recipients. To do this execute the following query:

mysql> select * from rcpt where id=(select id from metadata where piler_id='4000000053b2c71e13f0ce1400f725273a52');

I suspect that [e-mail] was added earlier.

[Former user]

My observation is that this error only appears after applying the VERP-Patches last week.

Above mail was received once. And the query returns exactly the only recipient. I learned to use "pilerget" (perfekt!), and the mail was actually saved in piler.

While using the web-frontend, these mails do not show up - in fact I can only see two mails received today.

Btw, what is the point of the "rcpt" table? Is it supposed to add a recipient for each mail? (Except for the ID, most of the other information is quite redundant.) No criticism intended - just wondering... ;)

[Janos SUTO]

The purpose of the rcpt table is to store the recipients of the email, including bcc stuff. Since an email may have several recipients, I decided to store them in a separate table, rather to add a blob(8192) not null type column to metadata.

Please do the following. Save the message with pilerget, eg. pilerget 40000.... > /tmp/1.eml Then run pilertest to parse it, eg. pilertest /tmp/1.eml. It should parse the email, and check the 'to:' line if it seems correct, and includes all recipients' emails.

Btw. do you have other, recent "duplicate entry" error emails? The verp stuff should not cause any trouble like that, since it merely adds another email address (coming from the mail envelope).

[Former user]

My rcpt table looks sth. like:

id         |   to                                 |  domain
--------------------------------------------------------------
29001  | [e-mail] | example.org
29002  | [e-mail] | example.org
29003  | [e-mail] | example.org
29004  | [e-mail]      | example.org

I was wondering if the redundant data was intended, but if ID is the internal message id, I see what you are getting at.

pilertest does not report any error. Here is the to: line:

to: *user test  [e-mail] test user example org  (example.org )*

I get this "duplicate entry" error on most of the mails now. I don't see anything obvious about them, e.g. sometimes those mails have multiple recipients, sometimes there's only one. I do notice, that while the mails are stored in piler (I know to use pilerget now), they do not show up in the web interface.

[Janos SUTO]

I've just checked the demo site, and indeed, it also produces the "Duplicate entry" error messages for some messages, however those messages are spam with invalid or missing message-id fields. Other than that I can't see an error, though those emails don't use verp addressing, just plain old [e-mail] like addresses. I'll add some messages with verp addressing as well. We'll see.

For the web gui, check if you have /var/piler/sphinx files, and they are several MBs large (or even bigger).

[Former user]

Alright, I switched back to "always_bcc" and I don't get any "Duplicate entry" error. Those mails also appear in the WebUI. (sphinx indexing seems to work)

1.) VERP addressing is responsible for the "Duplicate entry" error message (99,9% of all mails) 2.) in case the error is thrown, the respective message does not show in the WebUI

I hope this error description is of any help! Sorry for giving you a hard time ;)

[Janos SUTO]

I'll give it a deeper test, I think it can be fixed. I'll let you know when it's ok.

[Janos SUTO]

Please overwrite src/parser.c with the attached file, and recompile piler. I think it will solve the "duplicate entry" issue. If the message is actually archived, and you can retrieve it with pilerget, then we will access the message in the gui. But first, let's fix the "duplicate entry" bug.

[Former user]

That seems to have done the trick!

No more "duplicate entry" logs, and all new mails accessable in the webUI.

Any chance I can make those "inbetween" mails visible in the webUI as well? I tried pilerexport -> pilerimport but didn't get any changes as all messages had been archived already.

[Janos SUTO]

OK, 1 step closer. The cause of the duplicate entry issue was a minor bug that made the given email address to present twice in the "to" buffer, and trying to insert it to the rcpt table twice. However I don't think it prevents the gui to access the given email, because I think it's stored, archived by piler, and you could retrieve it by pilerget.

To check things, please get the serial id of the message:

mysql> select id from metadata where piler_id='4000....';

Now check if it's know by sphinx:

$ mysql -h 127.0.0.1 -P 9306 mysql> select * from main1,dailydelta1,delta1 where id=12345;

where 12345 is the number you got from the metadata table.

[Former user]

The metadata is filled with all information. The sphinx database does not have any knowledge of the mail-id; an empty set is returned.

If I understand the mechanism correctly, then there might have been an error preventing the data to be written into sph_index. Seems to be fixed now anyways.

I would delete all sphinx files and run the reindex utility?

[Janos SUTO]

That's exactly what happened. Try the following: instead reindexing everything, reindex only the problematic messages:

cd /tmp reindex -f 12345 -t 54321

where 12345 is the first affected id, and 54321 is the last affected id. After that you should be able to see all emails in the gui.

[Former user]

Alright, learned a lot!

There is one minor thing I am not sure whether it should be discarded; as you stated before, the use of VERP scheme requires each message to be delivered to piler as many times as there are recipients to the original mail. For all I can see this will continuously generate "Duplicate entry" errors as the second (third, etc.) recipient has already been logged in the "rcpt" table from parsing the first recipient's mail. This of course is not the case if we're talking about proper bcc recipients. Then the original mail should have no knowledge of the bcc line.

Regarding the documentation; you call this type of addressing Variable Envelope Return Path (VERP). VERP is actually a special application of sub-addressing, commonly used for mailing lists (or rather for bounces thereof). Replacing the "@" with "=" seems to be an old qmail convention.

I had a good read at Sieve Extensions and directly at the Postfix page recipient_delimiter and recipient_bcc_maps

Finally, realizing that someone else might use sub-addressing for his own purpose, I changed the bcc_map:

# /etc/postfix/bcc_map
#
/^([^+]+).*@(.*)$/      archive+$1=$[e-mail]

[Janos SUTO]

I think I'll fix the message processing flow not to return an error in case of a duplicate entry issue. Unfortunately it's not easy to prevent the duplicate error if you deliver the same actual message many times to piler.

[Janos SUTO]

I think the issue is resolved.

[达 文]

hello.the attached file is error.

error:conflicting types for ‘split’;

thank.

[Janos SUTO]

Could you elaborate the problem? Which attached file is 'error'? Why is that? Where can you see the "error:conflicting types for ‘split’;" message?

[达 文]

parser.c

line 638,

An error occurred when running 'make'

[Thomas Czarnetzki]

Is there a config-option in piler.conf, which i must enable to use the feature "Sub-Addressing"? I have two problems:

1. If i enable "archive_only_mydomains=1", then piler discard the emails
2. If i disable "archive_only_mydomains=0", piler will accept this message. But the email will not assign to the user with the bcc-address. So they cant view the email in die WebUI.

[Janos SUTO]

Please elaborate the problem, perhaps you may give a detailed example, and what you mean by 'sub-addressing'.

[Thomas Czarnetzki]

i mean this feature, was discuss in this thread ;-) If i send a mail, who has [e-mail] in BCC, it will send by postfix to archive+user=[e-mail]. But piler discard this mail, if i have the option "archive_only_mydomains=1" on. Allthough "domain.de" is under the Domains added.

If i disable the option, then piler accept this email, but dont assign to the user "[e-mail]". So the User cant see this email in his WebUI-Account.

[Janos SUTO]

What piler version do you have?

[Thomas Czarnetzki]

1.3.0

[Thomas Czarnetzki]

Do you need any more Informations from me? Can you replicate my problem?

[Janos SUTO]

I couldn't investigate the issue yet. I need more time.

[Mich80]

Hello. I have the same problem as Thomas Czarnetzki.

Piler v 1.3.1

I can see "to" and "bcc" email going towards mailpiler in this form

MAIL FROM:archive+sender=[e-mail]

got: RCPT TO:archive+to_user=[e-mail]

got: RCPT TO:archive+bcc_user=[e-mail]

but i cannot see the email in [e-mail], only [e-mail]

mailpiler logs "added 1 recipients". Shouldn't it be 2?

Thanks

[Janos SUTO]

Try setting process_rcpt_to_addresses=1 in piler.conf

[Mich80]

Thanks Janos. It's working!

[Janos SUTO]

Great news!

[Janos SUTO]

I conclude that the issue is solved. If you think otherwise, Thomas, please reopen it.

[Mich80]

Hi Janos. One last thing about BCC.

MAIL FROM:archive+sender=[e-mail]

got: RCPT TO:archive+to_user=[e-mail]

got: RCPT TO:archive+bcc_user=[e-mail]

If I login as auditor I can see this email:

MAIL FROM:[e-mail]

RCPT TO:[e-mail]

and next to the recipient an icon (group of people). If I move the mouse on the icon I can see both rcpt and bcc email. Everything is correct

But... if I login with sender account (regular user) I cannot see that group icon so the sender user doesn't know if there was a bcc user in the original email. However if I search [e-mail] I can find the email with sender [e-mail] and recipient [e-mail].

Is it possible to have that group icon even with regular user (or something that could show the bcc email)

Thanks

[Janos SUTO]

hmm, it seems that the sender envelope address is also verp-like. However, piler reads the mail header From: line. What is it like?

[Mich80]

Hi. I think that the problem is that i login only as sender, not as [e-mail]. I had to modify this file to view the emails(as seen in another post):

/var/www/piler/model/user/auth.php

//$emails = array($username);

emails = array($username . '@domain.com');

[Janos SUTO]

OK, great.

[Thomas Czarnetzki]

Dear Janos,

i have testet it again (with the parameter: process_rcpt_to_addresses=1), but nothing has changed. If my postfix send a mail in the format: "archive+to_user=[e-mail]" to piler, AND archive_only_mydomains=1 then piler dont accept the mail:

Dec 12 13:23:35 xxx piler[11519]: xxx: discarding: not on mydomains, from=xxx@xxx, message-id=xxx@xxx

If i set "archive_only_mydomains=0", then it will be accepted.

[Janos SUTO]

It seems that 'mydomain' check is done before the verp address fix. I assume you use the prefork version of piler. What version is it?

Btw. I've just noticed that when I rewrote piler using the epoll model I forgot to add the mydomain support. I'll add it again.

[Thomas Czarnetzki]

[root@piler]# piler -v 1.3.0 build 955

[Thomas Czarnetzki]

any update about this issue?

[Janos SUTO]

Not yet. I'll try to make it in a few days.

[Janos SUTO]

A question: where is the verp email address is specified? In the email headers or in the mail from or rcpt to (=envelope) addresses?

[Janos SUTO]

Anyway if the verp address is in the mail headers (To or Cc) try the following: edit src/parser.c, and find the last strtolower(puf); line (around the 731st line), and add the following line after it:

if(strchr(puf, '@')) extract_verp_address(puf);

Then recompile, and update the libpiler.so* files. Let me know how it goes.

[Thomas Czarnetzki]

I have modify the code and debug it (add a syslog line), but it does't work. I have send the following Mail: "To: [e-mail], bcc: [e-mail]" Piler Is Running for domain2.tld

Feb 20 06:53:38 mailarchive piler[26177]: puf is: [e-mail] Feb 20 06:53:38 mailarchive piler[26177]: ...: discarding: not on mydomains, from=xxx@xxx, message-id=xxx@xxx

Of course "domain2.tld" is in mydomains. It looks like, that piler doen't search the email-address at the right place.

[Thomas Czarnetzki]

Additional Info:

Feb 20 07:10:54 mailarchive piler[26844]: xxx: got: RCPT TO:archive+address=[e-mail]

so the right place to search is "RCPT TO"

[Janos SUTO]

OK. Get the message from the archive, eg pilerget 4000000......xxxxx > /tmp/1.eml, then run pilertest on it, and verify that domain2.tld is recognized in the recipient list.

[Thomas Czarnetzki]

no, in the mail-source is the Address not included. Because it is a "bcc" Address. it will not show in the source. I think the verp-address-format (RCPT TO) will only be available within the smtp-communication.

[Janos SUTO]

It's odd: if process_rcpt_to_addresses=1 is set in piler.conf, then the parser should process the envelope addresses, and it should find the fixed address in domain2.tld.

[Thomas Czarnetzki]

i have set this parameter allready: "process_rcpt_to_addresses=1"

[Janos SUTO]

OK, I think I've nailed it finally. Edit src/parser.c, then fix the parse_message() function as follows:

There's an if condition around the 48th line:

if(state.tolen < MAXBUFSIZE-len-1){

Add the following before the above line:

if(is_email_address_on_my_domains(puf, data) == 1) sdata->internal_recipient = 1;

Then recompile, and check again.

[Thomas Czarnetzki]

first test looks fine! I will do our full checks asap, and give you a finally feedback shortly. btw: the first fix generate the follwing warning at compile:

parser.c: In function ‘parse_line’: parser.c:734:10: warning: implicit declaration of function ‘extract_verp_address’ [-Wimplicit-function-declaration] if(strchr(puf, '@')) extract_verp_address(puf);

Is the first fix also need for the bug, or only the second?

[Janos SUTO]

Only the second.

[Janos SUTO]

I conclude the issue resolved.

[TM]

Hi! Apparantly this is missing from the documentation: https://www.mailpiler.org/wiki/current:index

[Janos SUTO]

Thanks for noticing. I’ve updated https://www.mailpiler.org/wiki/current:installation. Let me know if it needs more clarification.