postfix alias handling

[d tbsky]

Hi: postfix 'always_bcc' or 'recipient_bcc_maps' only bcc mails when email arrived. but the internal alias/forward won't trigger bcc anymore(even it triggers, I wonder if piler would handle these duplicate mails)

there is a great web ui function "CUSTOM_EMAIL_QUERY_FUNCTION", but maybe the similar thing can be done in archive level. eg: when piler receive an email, query a script and get some email addresses then add them to rcpt table.

there are some benefits about it:

1. the alias/forward relationship maybe change at later time, so previous emails belongs to a user maybe hidden even with "CUSTOM_EMAIL_QUERY_FUNCTION" web ui.
2. pilerexport can restore user emails more correctly. I think pilerexport won't think about "CUSTOM_EMAIL_QUERY_FUNCTION".

thanks a lot for help!!

[d tbsky]

Hi: I found with postfix "recipient_bcc_maps" and piler.conf "process_rcpt_to_addresses=1", piler will accept duplicated mails. so if I wrote a script for sending a copy to piler after postfix local delivery, the extra rcpt to will be recorded too?

I wonder what message to copy, is the header with "message-id" and mail to sub-addressing (archive+user=[e-mail].) enough?

[Janos SUTO]

I think the extra recipients will be recorded. In theory, the header would be sufficient, however I suggest to test it with a single email if it works properly.

[d tbsky]

hi: just finish the script. but my testing was failed. the extra mail was considered duplicate and not add to rcpt table. may I ask the correct parameters for piler to add rcpt table? according to discussions, I need

process_rcpt_to_addresses=1

hostid=/the same fqdn which mail server send to piler/ (is this necessary?)

do I miss other parameters?

[d tbsky]

hi: piler 1.3.4 is not working, but trunk works. now try to figure out what commit and what parameters make it work.

[d tbsky]

piler 1.3.4 + patch at commit 3ce983b works fine. now try to figure out what parameters are working...

[d tbsky]

"process_rcpt_to_addresses=1" seems the only parameter really necessary. "hostid=something" => sometimes it is necessary sometime not. I am not sure why.

there is one thing I am confused. can I send multiple mails in one smtp session? I try to send one full mail and two headers in one smtp session wth correct order (full mail first), then sometimes strange things happen. (the rcpt table is not added correctly)

should I send only one mail in one smtp session, or there is no limitation?

[d tbsky]

sorry. after more testing, "hostid=correct fqdn or ip" is necessary. or piler will record useless information to rcpt table.

however piler 1.3.4 gui seems can not handle rcpt table correctly. when I login with "auditor@local" I can see the mail and their rcpt users. but when I login with individual user, I can not see the mail.

[Janos SUTO]

You should check the sphinx query, and verify that the given user has all his email addresses in the query.

[d tbsky]

could you give me some hint how to check sphinx query? I can see the email address in the mysql rcpt table, and auditor gui.

[Janos SUTO]

Check the mail log, piler gui syslogs the sphinx queries.

[d tbsky]

thanks for the hint. after checking, the sphinx query didn't give me the result. the usrename is at mysql rcpt table, but it didn't go to sphinx database. so it can only show when I use auditor@local to login. is this the expected result?

[Janos SUTO]

It doesn't work that way, only the emails in the email itself (and the envelope addresses if enabled) go to the sphinx database. The problem might be with gui setup: I suspect that when the user logs in the gui doesn't assign all the user's email addresses that's why he can't see the given email.

[d tbsky]

sorry I am really confused now. I suppose the "process_rcpt_to_addresses" is to record extra receiver information. so if a mail which mail-to user1, bcc to user2, then both user1 and user2 would see the mail in piler. and if auditor search "mail-to" user2, he can see that mail. if not, then what's the usage of extra rcpt information?

[Janos SUTO]

Don't be stuck with the rcpt addresses, they work properly. However the gui shows the given message to a user only if one of his email addresses (known by the gui) macthes either the from or rcpt addresses recorded in the piler database. So as I said check the sphinx query if it lists any email address in the from or rcpt addresses.

So the bottomline is that the gui applies a filter for the non auditor users to restrict them to their own emails only.

[d tbsky]

I am sorry that I don't understand how sphinx works. when I use mysql client to connect port 9306, I can see the same rcpt data as mysql. but according to piler log, the gui sphinx query is

  sphinx query: 'SELECT id FROM main1,dailydelta1,delta1 WHERE        MATCH(' (@from test1XtestXexampleXcom | @to test1XtestXexampleXcom) ') ORDER BY `sent` DESC LIMIT 0,20 OPTION max_matches=1000' in 0.51 s, 14 hits

I don't understand the connection. it seems rcpt data didn't transfer to sphinx "to".

[Janos SUTO]

OK, then I need one more detail: the from and rcpt addresses for the email that the user with [e-mail] address cannot see.

[d tbsky]

hi: from [e-mail] to [e-mail]

I don't know if my piler is too old. the version is 1.34 + commit 3ce983b . (can not make gui working with trunk version)

full detail below:

[e-mail] -> [e-mail] (postfix bcc to piler at the stage)

(postfix internal) [e-mail] -> [e-mail], [e-mail] (script bcc to piler after some delay, and make sure original mail go to piler first)

after above, piler mysql rcpt table have 3 users (testalias,test1,test2).

when I use auditor to login piler, I see the mail, but it shows "to [e-mail]" (should be to testalias), and there is an icon, which shows all three receivers if I move mouse over the icon.

when I use [e-mail] to login piler, I can not see the mail.

[Janos SUTO]

OK, I see it by now. Well, you have a tricky setup, and it's difficult to solve. The problem is that when you forward the email, the messages is already processed for indexing with testalias@ address. I think the easiest solution could be adding testalias@ address for test1@ user.

[d tbsky]

sorry I don't really understand. my idea comes from issue #335

that discussion is focus on bcc, but the situation is the same. basically there will be duplicate mails for piler to handle. I also tried bcc method at #335, the problem is the same.

as you said, add testalias for test1 at gui will be another way. but I also need to have the ability to export mails belongs to test1, as I know, there are no dynamic filter like gui when running pilerexport. so it is easier to store all the information at beginning. I can re-write my script to combine those mails and "add X-Envelope-To:". but the method at issue #355 is better if it can work.

[Janos SUTO]

Well, if the solution in #335 is fine, then feel free to use it. Could this issue be closed, then?

[d tbsky]

sorry my solution is base on issue #335. so issue #335 is not working. have you try the setup the config as issue #335 recently?

[Janos SUTO]

No, I don't have such a setup. However, if you already know that you will forward the given message to email1@ and email2@, then your best bet is to prepend those two addresses as an X- header, eg. X-Envelope-To:, then enable the processing of that field in piler.conf, and you should be good to go.

[d tbsky]

actually I don't know the email address in current situation. so I hope the "process_rcpt_to_address" will do the job for me.

if "process_rcpt_to_address" can only do half-job( eg: only update mysql but not sphinx). then I need to rearrange the whole thing and pre-analyze the email address so I can add correct "X-Envelope-To:" header.

[Janos SUTO]

The process_rcpt_to_address option makes piler to read the envelope recipients, and in 1.3.4 the smtp handling is done by piler-smtp, so it adds an X-Piler-Envelope-To: header. So I'd suggest to test and decide what works for you.

[d tbsky]

ok. I will do further testing and report again.

[Janos SUTO]

Until then the issue is on hold. be sure to set it to open when you have further input.