Some attachments in Piler are corrupted

[Christian]

Piler has been running for a while now and done a great service, so far.

But recently I discovered that occasionally (like 60 % of) the attachments are corrupted.
Meaning sometimes after downloading they have a 0 Byte file size and sometimes the file is corrupted and can’t be opened.

One of my questions is, if I would do a pilerimport and import all the mails i still have in my IMAP would this overwrite the corrupted attachments? Or would Piler just leave the corrupted attachments there because they already exist?

[Janos SUTO]

Piler wouldn’t overwrite existing emails archived. You need some debugging to figure out what’s wrong. Select a message on the gui that has a corrupted attachment. Then try downloading the message as an eml file. Then open it with some text editor, and check the attachments in the eml file. Also be sure to check the mail log if it syslogs any issue when you download the message.

[Christian]

Thank you for your answer.

I just did what you proposed and when I download the .eml file and extract the attachment it suddenly has 23 KB compared to 0 B when downloaded directly. But the file is still corrupted (even with 23 KB more) and cannot be opened. No messages in mail.log.

[Janos SUTO]

If there’s such a not sensitive email, please attach it to the issue.

[Christian]

This one has a corrupted pdf file attached. but the size is the same direct download or .eml extract.

[Janos SUTO]

Indeed, the attachment part looks odd. What version of piler do you have? Run piler -V

[Christian]

piler 1.3.1-epoll, build 980, Janos SUTO [e-mail]

Build Date: Sat Oct 7 16:20:27 CEST 2017
ldd version: ldd (Debian GLIBC 2.24-11+deb9u1) 2.24
gcc version: gcc version 6.3.0 20170516 (Debian 6.3.0-18)
OS: Linux tascam 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux
Configure command: ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-database=mariadb
MySQL client library version: 5.5.1
Extractors: /usr/bin/pdftotext /usr/bin/catdoc /usr/bin/catppt /usr/bin/xls2csv /usr/bin/unrtf /usr/bin/tnef

[Janos SUTO]

It’s possible that a bug caused improper writing of attachments in some cases. I’d like you to get the master branch (see the download section in bitbucket), and upgrade piler.

[Christian]

I’m already in the progress to upgrade to a containerized (docker) Piler (i use https://bitbucket.org/jsuto/piler/downloads/piler-1.3.7.tar.gz; i assume that is the master branch).
Is there any way to fix the corrupted attachments? For example i could take the fresh install of piler 1.3.7, then run pilerimport on the imap server and then … i don’t know how, use pilerimport to import from the old piler.
If it is as you say that pilerimport won’t overwrite existing e-mails or attachments… that’s how i could ensure as many as possible intact attachments to be in Piler.

I also seems to me that older messages (before 2017) are not as affected by this problem as recent (2 or 3 years old) ones.

Thanks for your help.

[Janos SUTO]

Yes, 1.3.7 should be fine. Unfortunately there;s no way fixing already archived emails. If you have all your emails still on the mail server, then I suggest to start a new archive, import them all, then discard the legacy archive.

[Christian]

What if i wrote some script to download all e-mails from the old piler as .eml file into a folder.
Then run pilerimport on the IMAP server on the new piler (1.3.7)
and then run “pilerimport -d”, it would not overwrite existing (intact) e-mails, just add old ones that are no longer stored in the IMAP.

What do you think? Could that work?

[Janos SUTO]

Yes, it should work.

[Christian]

Thanks for your help.