I cannot see attachments in e-mails / they are still in the EML file

[Jose M. Albarran]

Hi

I don't know if this is the same than #704.

Piler version: 1.2.0-master build 945 Ubuntu 16.04.1 / MySQL 5.7 / PHP 7

I have checked several old e-mail imported with readpst / pilerimport.

I (auditor) don't see the attachments in the GUI.

But, if I download the email, I see the attachment embedded. It's a base64 encoded zip file. The boundary exists and seems to be correct.

--=-vnG+PdvbGusWwtziozLw Content-Disposition: attachment; filename="wz020700.zip" Content-Transfer-Encoding: base64 Content-Type: application/octet-stream

I will send the sample direcly by e-mail

[Janos SUTO]

Thanks for the test email. However it looks good testing with the master branch: it can detect the attached zip file properly, and stores it accordingly. Run pilertest against this message, and verify that it can see the attachment. If you download the email, then the attachment should be embedded just as it's in the original email. And you should see a link to the attachment as well.

[Jose M. Albarran]

The message was loaded with pilerimport. I will retest with this tomorrow.

Thanks!

[Janos SUTO]

Let me know, how it goes.

[Janos SUTO]

No news is good news.

[Jeremiah Poling]

I'm having this issue as well. I'm using the OVA from here: http://download.mailpiler.org/piler-build-935.ova and downloading emails using pilerimport from office365. Some, but not all emails with attachments are not showing the attachments in the webGUI, but they are there when I open the eml file from piler.

[Janos SUTO]

That's a pretty old build by now, I'm not sure if I'll upgrade it or provide a docker image instead.

Anyway, check if the problematic email is stored properly, ie. it has all the attachments in the attachment table (check first with the metadata table, and see the attachments column). If it seems OK, then it's a gui bug.

[Jeremiah Poling]

Are there instructions anywhere to upgrade from Build 935 to the one you just released?

[Janos SUTO]

Well, you should update the database schema using the sql script in util/ directory. Then compile the new version, install it, overwrite the gui, and you are done. Since you have a virtualized environment, I suggest to use snapshots allowing you to roll back in case of a problem, or if you have the required resources, you may copy the VM, and perform the upgrade on the copy to see if it goes smoothly. Anyway, be sure to backup the mysql database, and all piler related config files, keys, etc.

[Jeremiah Poling]

ok, I upgraded the virtual machine to the newest everything that apt-get offered, then I upgraded to the 1.2.0 release of piler, and then I did reindex -a. These emails are still not showing their attachments in the UI or the metadata table. Any further things I can try?

[Janos SUTO]

Show me the metadata entries for such an email. I'm interested in the id, piler_id, attachments columns. Then we'll see if the email is stored properly, ie. recognized the attachments or not.

[Jeremiah Poling]

SELECT * FROM piler.metadata order by size desc limit 10;

http://www.sharecsv.com/s/59e01730fcb2faa3ac55cfa26f3eb520/test.csv

[Janos SUTO]

Which email from this list doesn't look right in the gui?

[Jeremiah Poling]

The second one for sure, but I think all of them that don't have attachments listed. These are all 30+ megabyte eml files. I don't see how they could not have attachments.

Thanks, Jeremiah

[Janos SUTO]

I see that. Run the following commands, and let me know what pilertest reports. I'm interested in only the attachment related output.

cd /tmp
pilerget 400000005805644e10b8097400ee25930101 > 1.eml
pilertest 1.eml

[Jeremiah Poling]

body digest: 97430db0ee3eea1d5ccc7b9382b2f5c9ae18604f5ad00ea54b30b82c4ddac7f1
rules check: (null)
folder: 0
retention period: 1698934310
i:1, name=*20140929_160740.pdf*, type: *application/pdf*, size: 34637082, int.name: 1.eml.a1, digest: c815e1bdd5081b5fa0dd3adb9e8b842e2ffd6d384f4f6c0df02336dd9b8ee4a1
attachments:pdf,
direction: 1
spam: 0

[Janos SUTO]

OK, the parser knows correctly that it has a large pdf attached. Is it possible that these "0 attachment" files were archived with the older version, and the pilertest output came from version 1.2.0?

[Jeremiah Poling]

Yes, that's the situation.

Is there any way to reparse everything? I was hoping that the reindex -a would reparse, but I guess not.

Thanks, Jeremiah

[Jeremiah Poling]

Any more thoughts on this? Thanks!

[Janos SUTO]

Well, there's no way to reparse an already archived email, sorry. Technically it's possible to fix a message after it's stored, but there's no tool for that.

[eXtremeSHOK]

BTW, The tools do exist. As they are non-free you will need to contact us.

Only thing is one would need to recreate the archive, pretty much xshok-piler-archive-recovery and then xshok-piler-import-dir will have the entire archive reconstructed with the attachments imported and everything corrected.

[Jeremiah Poling]

Are those tools available somewhere that I would have access to? and would I need to wipe out the current database before running them? Or, I guess a better question would be: Is there a step by step guide? Thanks again!

[Janos SUTO]

The official piler repo doesn't have such a tool. Keep me posted how it goes.

[eXtremeSHOK]

They are non-free , https://bitbucket.org/snippets/extremeshok/dddX9/contact-info

[Jeremiah Poling]

I appreciate you bringing these tools to my attention, but this is just a minor annoyance to me, not a showstopper. so I'll pass this time. Thank You!

[Janos SUTO]

OK, then case is closed.