Outbound issue - No From address and Email displays as Journal Report

[Stephen Risinger]

Inbound and internal emails are imported and display in piler correctly. Outbound emails do not show a From address and display the message as X-MS-Journal-Report. I have Exchange Standard 2007.

I attached an example of what one of these outbound emails looks like.

(I hope I have my Exchange server set up correctly! I followed the directions.)

[Janos SUTO]

Is it a complete email? I mean, some obfuscation is OK, but aren't there any more headers before

X-MS-Journal-Report: 
MIME-Version: 1.0 

[Stephen Risinger]

Actually, no. When I click on View Headers, all I see is

#!

X-MS-Journal-Report:
MIME-Version: 1.0

[Janos SUTO]

Well, I meant a complete pilerexport output of the email.

[Stephen Risinger]

Janos, I just emailed you a full eml export from piler.

[Janos SUTO]

Thanks, I've just checked it. So just to confirm it, you used pilerexport (and not the gui to retrieve the message), right? Because it starts with an empty line, which is odd.

[Stephen Risinger]

No, that was from the gui. Let me pull that via CLI and re-send...

[Stephen Risinger]

Janos, did you receive the corrected eml export?

[Janos SUTO]

Yes, I got it, just I was distracted. I'll check it soon.

[Stephen Risinger]

Janos, I found the problem, but I still think this is something Piler should be able to work with.

One of the previous SysAdmins set our Exchange 2007 server to remove the Receive header. Due to this missing, it appears Piler doesn't know how to parse out the rest of the message.

Is this an easy modification/fix? If so, awesome! If not, I will permanently disable this rule.

[eXtremeSHOK]

It is not recommended and not common to remove received headers.

I only know of cases where it is desirable to remove the headers in an attempt to prevent leakage of your network configuration, this generally is not successful.

The received headers are invaluable in tracing error conditions and the "flow" of the email.

im referring to these: Received: from mxfront8j.mail.yandex.net ([127.0.0.1]) by mxfront8j.mail.yandex.net with LMTP id J27O5Fv0 for [e-mail]; Thu, 27 Nov 2014 20:19:02 +0300 Received: from mx-01.example.net (mx-01.example.net [x.x.x.x]) by mxfront8j.mail.yandex.net (nwsmtp/Yandex) with ESMTPS id kPMcIMmunV-J168XFhY; Thu, 27 Nov 2014 20:19:01 +0300 (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (Client certificate not present)

[Janos SUTO]

Both the parser and the gui searches for the Received: lines. The reason is simple: it's the easiest to detect the (real) header. I may use another line, eg. From: or Date:, however they can be just in any order while Received: is always in the first place, so it's an obvious choice. So I recommend you to stop removing the Received: lines. I can't think of a scenario which would require such a practice. Perhaps the previous admin has a good explanation. If he does, then I'm interested in his answer.

Edit: perhaps he's concerned about the 'leaking' of some internal host names, addresses, etc. If this is the case, then please find a way to remove Received: lines only from messages meant for the outer world, but keep it for piler.

[eXtremeSHOK]

From is easy to fake and would cause issues with proving legitimacy of the email.

Daily we have faked from addresses caused by domains which do not have spf/dkim enabled.

It would be very dangerous practice to have this used in an archive.

[Stephen Risinger]

I'm not sure how to remove Received: lines only from messages meant for the outer world, as piler is added as a BCC for the journaling (as I understand it).

Janos, I am okay with closing this request as invalid (?), as it was an issue on our end, not with Piler, unless you see an opportunity (and have the time) to modify the parser and gui searches.

[Stephen Risinger]

Odd side question, do you know of a way to add the Received: lines (all the same) to the email messages already in the piler database?

[eXtremeSHOK]

If you can get exchange to re-add the headers and then purge the invalid emails and re-import.

Not much good news..as your archive emails which have missing headers could be considered garbage.. They cannot be used for legal purposes and since they have been altered there is also no way to prove when or where they originated, this violates data retention laws.

If your company falls under any legal requirements for data retention, these requirements have been broken.

[Janos SUTO]

Unfortunately there's no easy way to fix emails already archived, since it's a core feature to make any tampering with messages difficult*, even this one. As extremeshok suggested it's possible to purge these emails from the archive, then trying to reimport them if you know when the change happened. I think there might be many messages making this unpleasant.

However you may not need to edit the stored emails, since the missing From entries come from the metadata table, so the task is to select entries where the from column is empty, then retrieve the message from the archive with pilerget, get the From: line, and write it back to from and from_domain columns in metadata.

Eg. using some sort of pseudo code:

a) get affected ids:

select piler_id from metadata where `from`='';

b) do the dirty work:

foreach piler_id from 'idlist' {
   pilerget piler_id > tempfile
   sender=grep "^From:" tempfile | head -1 | cut -f2 -d ':'
   update metadata set `from`='$sender' where piler_id='$piler_id'
}

c) reindex the affected messages:

reindex -f 12000000 -t 15000000

[Janos SUTO]

I assume it's solved by now. Also I've fixed the parser (in the master branch) regarding the extraction of the from address.