pilerimport rejects messages with invalid message, hdr_len: 0

[Bernhard]

pilerimport recjects messages "invalid_message, hdr_len: 0.

[root@triton piler]# piler -V piler 1.2.0, build 952, Janos SUTO [e-mail]

Build Date: Wed Jun 14 11:46:14 UTC 2017 ldd version: ldd (GNU libc) 2.17 gcc version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) OS: Linux triton 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Configure command: ./configure --localstatedir=/var --with-database=mysql --enable-tcpwrappers MySQL client library version: 5.5.52-MariaDB Extractors: /usr/bin/pdftotext /usr/bin/catdoc /usr/bin/catppt /usr/bin/xls2csv /usr/local/bin/ppthtml /usr/bin/unrtf /usr/bin/tnef

[root@triton piler]# pilerimport -e email_bug/3241.emlx email_bug/3241.emlx: invalid message, hdr_len: 0

[Janos SUTO]

Can I see this file: email_bug/3241.emlx?

[Bernhard]

Here is the header part of the elmx File

Received: from BUGSERVER.BUG.local (192.168.51.200) by BUGSERVER.BUG.local (192.168.51.200) with Microsoft SMTP Server (TLS) id 15.0.847.32 via Mailbox Transport; Tue, 26 Sep 2017 08:53:13 +0200 Received: from BUGSERVER.BUG.local (192.168.51.115) by BUGSERVER.BUG.local (192.168.51.115) with Microsoft SMTP Server (TLS) id 15.0.847.32; Tue, 26 Sep 2017 08:52:52 +0200 Received: from BUGSERVER.BUG.local ([fe80::914f:f1a:8a22:51dd]) by BUGSERVER.BUG.local ([fe80::914f:f1a:8a22:51dd%12]) with mapi id 15.00.0847.030; Tue, 26 Sep 2017 08:52:34 +0200 From: BuG Sabina Salkanovic [e-mail] To: "BuG Markus Stieger" [e-mail] Subject: BVH Starkfriedgasse | Fuchs | Haupt-AN | Stiegenhaustreppe Thread-Topic: BVH Starkfriedgasse | Fuchs | Haupt-AN | Stiegenhaustreppe Thread-Index: AdM2k/EXk3IT1uEZTjiSSnlCtNz1Bw== Date: Tue, 26 Sep 2017 06:52:33 +0000 Message-ID: [e-mail] Accept-Language: de-AT, en-US Content-Language: de-DE X-MS-Exchange-Organization-AuthAs: Internal X-MS-Exchange-Organization-AuthMechanism: 03 X-MS-Exchange-Organization-AuthSource: BUGSERVER.BUG.local X-MS-Has-Attach: yes X-MS-Exchange-Organization-SCL: -1 X-MS-TNEF-Correlator: Content-Type: multipart/mixed; boundary="004_e80a3676f2754333a1f27fe5af7aad06BUGSERVERBUGlocal" MIME-Version: 1.0

--004_e80a3676f2754333a1f27fe5af7aad06BUGSERVERBUGlocal Content-Type: multipart/alternative; boundary="000_e80a3676f2754333a1f27fe5af7aad06BUGSERVERBUGlocal"

[Janos SUTO]

No good, I can't use this text, I need the email file intact. If it's not (too) sensitive, please either send it to my address in a zip, or perhaps even better if you could put it to somewhere I can download the complete email. If it's sensitive, then run pilertest on it, and check its output. If it's not good, then download the master branch, compile it (but don't install it or overwrite anything! It's best done on another host), and run the latest version's pilertest on it.

[Bernhard]

ok, here you can find the complte mailsource: http://www.openit.at/download/3241.emlx

and here is the pilertest output:

pilertest email_bug/3241.emlx locale: C build: 952 parsing... post parsing... message-id: null / 74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b from: * () to: * () reference: subject: body: ** sent: 1506504711, delivered-date: 0 hdr len: 0 body digest: rules check: (null) folder: 0 retention period: 1727429511 attachments: direction: 0 spam: 0 NOT IN mydomains

[Janos SUTO]

Thanks for the file, you may remove it from your download area. I think user piler has no read access on the given file.

[Bernhard]

Ok, I have changed the permissions of the emlx files and now the import is running with no problems, but in the piler webinterface I see emails which are empty. I have attached 2 screenshots. Not all of the imported emails but a lot of them.

Is there a possibilty to force a reimport?

[Janos SUTO]

I can't see any screenshots. Reimport would not work well, because the message-id is recorded to the piler database that prevents the same message to be archived again. If you have all the original emails, then I recommend you the following to try:

• create a new database for piler, eg. piler2
• create a new store dir for piler, eg. /var/piler/store2
• update piler.conf to point to the new database and store dir, eg.

mysqldb=piler2
queuedir=/var/piler/store2

• update config-site.php to use the new database

Then run the import procedure again, and check the results.

[Bernhard]

ok, I did not upload the screeshots; here they are:

http://www.openit.at/download/1.jpg http://www.openit.at/download/2.jpg

[Janos SUTO]

OK, I see. Please try the method I suggested.

[Bernhard]

I have done your suggested method:

1. I cloned the complete machine (xenserver virtual machine)

2. reset of piler (FAQ: How can I delete everything from the archive?)

3. import of the emlx files.

and I see the same as before on all emported Emails!

[Janos SUTO]

Do you mean that all emails are empty? Any error during the import? Any errors in the mail logs? Does user piler have proper permissions on the to be imported emails?

[Bernhard]

The permissions on the directory where the emlx files are located has 755; the emlx files have 666.

This is the output during the import process:

[root@larissa piler]# pilerimport -d email_bug duplicate: email_bug/3320.emlx (duplicate id: 6) Syntax Warning: Invalid Font Weight Syntax Warning: Invalid Font Weight Syntax Warning: Invalid Font Weight Syntax Error: Unknown character collection 'PDFAUTOCAD-Indentity0' duplicate: email_bug/3370.emlx (duplicate id: 33) duplicate: email_bug/3357.emlx (duplicate id: 35) duplicate: email_bug/3336.emlx (duplicate id: 25) duplicate: email_bug/3349.emlx (duplicate id: 10) duplicate: email_bug/3309.emlx (duplicate id: 44) duplicate: email_bug/3340.emlx (duplicate id: 60) duplicate: email_bug/3343.emlx (duplicate id: 39) duplicate: email_bug/3269.emlx (duplicate id: 37) duplicate: email_bug/3299.emlx (duplicate id: 27) duplicate: email_bug/3325.emlx (duplicate id: 2) Syntax Error: Unknown character collection 'PDFAUTOCAD-Indentity0' duplicate: email_bug/3363.emlx (duplicate id: 28) duplicate: email_bug/3249.emlx (duplicate id: 96) duplicate: email_bug/3372.emlx (duplicate id: 1) duplicate: email_bug/3244.emlx (duplicate id: 5) duplicate: email_bug/3245.emlx (duplicate id: 11) duplicate: email_bug/3254.emlx (duplicate id: 43) duplicate: email_bug/1506595197.M618504P12322.io.openit.at,S=47151,W=48047:2,RS (duplicate id: 26) processed: 132

tar -f /var/log/maillog

Oct 2 14:59:00 larissa pilerget[12487]: /var/piler/store/00/59d/99/db/4000000059d2255b1e31aed4007ffe5299db.m: cannot open() Oct 2 14:59:00 larissa pilerget[12490]: /var/piler/store/00/59d/99/db/4000000059d2255b1e31aed4007ffe5299db.m: cannot open()

ls -l /var/piler/store/00/59d/3e/34/4000000059d225473b1aed94009c949a3e34.m

[Janos SUTO]

Can I access this cloned environment to see what went wrong?

[Bernhard]

here is a quick update to my problem. If you download the stored email from piler webinterface, then you get back the imported email and it is identically to the original emlx file.

[Janos SUTO]

Please remove the comment containing the password asap.

Janos

[Janos SUTO]

Piler couldn't create its temp files during parsing the emails.