SMTP Code 554 5.3.190 restoring emails to O365

[David Requena]

I’m having trouble restoring email messages to O365 mailboxes.

Our domain MX is configured as piler’s smart host (<customer-doman>.mail.protection.outlook.com)

Piler version is 1.3.11, as downloaded a few days back, on ubuntu 20.04. Freshly installed,

‌

ngrep shows piler actually opens a connection an proceeds to send an email, which is finally rejected with:

554 5.3.190 Journaling on-premises messages to cloud is misconfigured or not supported. For more information please go to http://go.microsoft.com/fwlink/?LinkId=526653. [AS8PR09MB4566.eurprd09.prod.outlook.com] [DB6PR0501CA0015.eurprd05.prod.outlook.com] [DB3EUR04FT050.eop-eur04.prod.protection.outlook.com]

Upon further investigation, this seems to imply the transmission has been identified as an on-premises exchange server attempting to journal to an O365 mailbox. Such a configuration is not supported by Microsoft, hence the transmission is rejected.

I’m having the same issue both with $config['REWRITE_MESSAGE_ID'] set to 0 and 1.

Is there a way to avoid such a problem? Surely many people are using piler with O365.

‌

On the other hand, I seem to remember reading somewhere that piler would use IMAP restoration when IMAP authentication was in use. That does not seem to be the case with us.

‌

‌

[Janos SUTO]

BY checking the suggested url it says that

A journaling rule is configured in the organization's on-premises environment to journal on-premises messages to Microsoft 365 or Office 365, but Journaling Archive is disabled. For this scenario to work, the organization's Office 365 administrator should either enable Journaling Archive or change the journaling rule to journal messages to a different location.

‌

Can you verify this settings and fix if necessary?

[David Requena]

EDIT: Forgot to mentio that I’ve been able to restore mail to this very same O365 tennant from another piler install a few days ago.

What piler.conf or config-site.php settings could possibly affect how mail restoration is performed? I could not find a comprehensive list/description out of example.conf and config.php, both lacking documentation for some settings

.Can you verify this settings and fix if necessary?

I' already read the error description which elsewhere states this is disabled “or unsupported”. There’s no M365 administrator accessible setting for Journaling Archive that I can find. I dumped the whole organization settings to inspect them. I even spent a couple days googling around and found no reference except for this one on the NDR codes description we’re discussing.

On the other hand, this is from Exchange’s Journaling documentation at https://docs.microsoft.com/en-us/exchange/security-and-compliance/journaling/journaling (emphasis mine)

You can't designate an Exchange Online mailbox as a journaling mailbox. You can deliver journal reports to an on-premises archiving system or a third-party archiving service. If you're running an Exchange hybrid deployment with your mailboxes split between on-premises servers and Exchange Online, you can designate an on-premises mailbox as the journaling mailbox for your Exchange Online and on-premises mailboxes.

So there must be some piler setting I must have messed up. Maybe along the lines of: for O365 to recognize these emails as journal messages, at least they must contain the original email as an attachment. Is there some piler setting controlling this?

‌

[Janos SUTO]

By default piler restores the email as it is, not changing it, except it prepends 2 lines

"Received: by piler" and “X-piler-id: 50000…..”

If you want to restore it as an attachment, then set $config['RESTORE_EMAILS_AS_ATTACHMENT'] = 1; in config-site.php.

Also let me know which variables you feel under documented and want to know more.

[David Requena]

If you want to restore it as an attachment, then set $config['RESTORE_EMAILS_AS_ATTACHMENT'] = 1; in config-site.php.

I do not want this specifically, users would see a different restored email. Anyway I’ve tested this setting now and restoration emails go through indeed, although without the original email attached, so no actual restoration.

A few days back I had another testing piler installation and that one was restoring to the very same O365 tenant without issue. Actual O365 org configuration has not been touched in the meantime so I think something must be wrong with this new install (same piler version).

There used to be a users mailing list at sourceforge and another one later at list.acts.hu, none alive today as is seems. Is there a current one somewhere? Probably that’d a better forum for this matter that than a bug report.

Also let me know which variables you feel under documented and want to know more

I’ll try to allocate some time to contribute a document with info on all settings I know of and placeholders for those I don’t. Then I’ll open a new issue with that doc to be filled in and kept with the rest of piler’s documentation. Would that be OK on your side?

[Janos SUTO]

Well, if the piler version is the same on both hosts, then try to copy the settings from config-site.php as much as possible. Also check with ngrep -X port 25 when you try to restore over smtp on both hosts, and see if there’s a difference. The restore process is basically trying to send the retrieved message over smtp. Not sure why o365 thinks it’s an onpremise exchange server which piler is not. I guess it checks the message headers, and perhaps the journaling headers should be removed. Though it wouldn’t explain why it works on the other host. Also such fix would alter the message that I believe the archive shouldn’t do. Perhaps it would be a good idea if you could contact o365 support, explain them the situation and ask for a solution or clues what to fix.

Regarding the docs, I’m OK with your proposition.

[David Requena]

Unfortunately it was a prior install on the same host. I re-did it because I had activated the folders feature and could no make any sense of it. I understand buckets are Enterprise only, right?

I’ll try with a clean config-site.php and see what happens. If it works, I’ll work my way setting by setting.

Please, keep the issue open. I’ll post an update if/when the culprit is found.

Also, no user mailing list then?

[Janos SUTO]

Well, the mailing list should be there. The archive is at http://www.mail-archive.com/[e-mail]/ and the list should work as well. Did you try to send something that failed?

[David Requena]

OK, found the culprit, I think.

Most of this might well be related to a bug I believe I’ve found in function removeJournal() in mime.php. I’m opening a separate issue for this one as I'm no PHP expert by any measure.

Anyway..

When restoring with $config['RESTORE_EMAILS_AS_ATTACHMENT'] = 0; piler sends email including the header

X-MS-Journal-Report:

There are several other journaling related headers but it’s this one that tiggers O365 flagging this transmission a as an attempt to journal external email to one of its mailboxes, so rejecting it with a 554 5.3.190 NDR code.

As far as I can tell, what piler is trying to restore is is the whole journal email, not the original archived one which, in O365’s case, was attached to it. As it is, this is just a journal email sent to O365, which in turn forbids being used as a journaling destination.

Removing said header, allows for the restoration to complete. The end result is a journal email in the users mailbox. This journal has the originally archived email attached as an EML attached anyways.

---

So, ending with an attached email all the same, why not go the config['RESTORE_EMAILS_AS_ATTACHMENT'] = 1; route? Well, I tried but here I'm facing a different issue:

O365 accepts the restoration email but never delivers it to the user mailbox. A few seconds after reception O365 sends an NDR back which, for the life of me, I cannot get at. Piler discards it because ‘not on mydomains’ even when I have archive_only_mydomains=0 in piler.conf. This being a TLS transmission, ngrep is of no help.

Why is piler rejecting this NDR I couldn’t tell. Used to have archive_only_mydomains=1 but it’s since been changed to 0 and piler restarted several times afterwards. Is there something else I must do to get piler back to archive anything received?

Thanks for your patience

EDIT: just got a little further intelligence on this.

Restoration is going through indeed. Result with config['RESTORE_EMAILS_AS_ATTACHMENT'] = 1; is a regular email with the journal email attached as an EML, which in turn has the originally archived email attached as an EML. This seems to be congruent with behavior shown when using config['RESTORE_EMAILS_AS_ATTACHMENT'] = 0;: just one level more of attachmentness.

Will try some dumping from c code to see whats going on with those discarded messages

[David Requena]

Resolved as a side effect of resolution of issue #1211