Message failed verification
Hello! I have successfully installed piler. I managed to import mails using imap, when e-mails appeared in piler search I noticed that a lot of mails(mostly with attachments), show "Message failed verification" and is missing attachments.
Comments (71)
-
repo owner -
reporter So, pilerget output: From: XXX xxx@xxx.xx User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130221 Thunderbird/17.0.3 MIME-Version: 1.0 To: yyy@xxx.xx Subject: =?UTF-8?B?0KHRhdC10LzQsA==?= Content-Type: multipart/mixed; boundary="------------090504060504000705030004"
This is a multi-part message in MIME format. --------------090504060504000705030004 Content-Type: multipart/related; boundary="------------080004050200050508040005"
--------------080004050200050508040005 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
--------------080004050200050508040005 Content-Type: image/jpeg; name="Picture 1" Content-Transfer-Encoding: base64 Content-ID: part3.09080000.03070908@xxx.xx Content-Disposition: inline; filename="Picture 1"
ATTACHMENT_POINTER_400000005243d96b3155f77c0083324d3753.a1_XXX_PILER--------------080004050200050508040005--
--------------090504060504000705030004 Content-Type: application/vnd.ms-visio.viewer; name="Network Actual(03.2013).vsd" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Network Actual(03.2013).vsd"
Metadata: | 820 | xxx@xxx.xx | xxx.xx | Схема | 0 | 1380178273 | 1362497358 | 1583422158 | 0 | 6126916 | 381 | 1 | 2 | 400000005243d96b3155f77c0083324d3753 | 51360F4E.4020606@xxx.xx | | f2ade7c2e4c7950392486d6f3b84301411751345b8c342a9d5a1b9b4da062d39 | 6f19b2ed065abad4c7f430dd7db63a4cc448aef0df8ef3b60d1f00592c7509c5 | da459520defdf350f06357dff55dab6a1776936963c670e98f92b04b4733ae44 |
-
repo owner ok, thanks. You clearly have some issue with the attachments. This email has two attachments. Now let's check the attachment table:
select * from attachment where piler_id='400000005243d96b3155f77c0083324d3753';
Also please show me: ls -la /var/piler/store/00/524/37/53
-
reporter mysql> select * from attachment where piler_id='400000005243d96b3155f77c0083324d3753'; Empty set (0.00 sec)
root@piler:~# ls -la /var/piler/store/00/524/37/53 total 2976 drwxr-x--- 2 piler piler 4096 Sep 26 09:51 . drwxr-x--- 6 piler piler 4096 Sep 26 09:52 .. -rw-r----- 1 piler piler 3034512 Sep 26 09:51 400000005243d96b3155f77c0083324d3753.a2 -rw-r----- 1 piler piler 1760 Sep 26 09:51 400000005243d96b3155f77c0083324d3753.m
-
repo owner You have the attachments on disk, however they are missing from the attachment table. This table is used to assemble the original message from the separated mime parts.
Do you have anything in the attachment table? It seems that it's corrupted. Btw. Please show me the log entries related to 400000005243d96b3155f77c0083324d3753:
grep 400000005243d96b3155f77c0083324d3753 /var/log/mail.log
-
reporter Yes I do have records in attachments table, some messages in piler are verified, thought most part "failed verification". /var/log/mail.log contains nothing regarding this id.
-
repo owner Please try searching in the older logs, eg. /var/log/mail.log.1, /var/log/mail.log*.gz I'd like to know what piler thought about the message when it was received.
-
reporter Already tried, it seems they not there because I imported all the mail by means of: pilerimport -i through console
-
repo owner How many emails are in the archive? Are they all from pilerimport?
-
reporter Over 1k messages, all from pilerimport.
-
repo owner Is it possible to drop the piler database and recreate it, and dropping everything under /var/piler/store/00/? If you have the messages, then please do so, and reimport again. I have little data to explain why some messages have entries in the attachment table and others why not.
-
reporter I've dropped database, run install once again. After installation imported 1 mail(from those with problem), this time not through imap but from eml file. This time piler shows all the attachments, but still shows "Message failed verification".
-
repo owner Do you get the message properly when you download it? Is it the complete message?
-
reporter From piler interface I can download all attachments. When I restore it to mailbox, or download in eml it opens without attachments:
In that message I noticed that first letter of attachments filename is missing. For example: filename=atalina.out.2013-08-22" - should be "catalina.out.2013-08-22"
-
repo owner Do you have any of these problematic messages with attachments which is not sensitive? If so can you send me one or two messages to test with? See my address in piler -V
-
reporter I'll search for suitable messages. For now, during "pilerimport -i" I can see a lot of following messages:
Format a4 is redefined
-
repo owner It's a warning from one of the attachment helper utilities (eg. pdftotext or another).
-
reporter Sent you couple of mails that did not passed verification in piler.
-
reporter Also noticed that pair of mails, completely identical except address they were sent to, one is verified - another failed verification.
-
repo owner Thanks, I got them. My personal archive took them and processed without problems, and they are verified, have the attachments. Then I copied them to the archive manually, changed the message-id, then imported them manually by pilerimport, and those 4 messages are all OK.
I know you mentioned before, but what OS and piler version do you use?
-
reporter Ubuntu 12.04.3 LTS
piler 0.1.25-master-branch, build 844, Janos SUTO <sj@acts.hu> Build Date: Wed Sep 25 17:14:35 EEST 2013 ldd version: ldd (Ubuntu EGLIBC 2.15-0ubuntu10.4) 2.15 gcc version: gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) Configure command: ./configure --localstatedir=/var --with-database=mysql --enable-starttls --enable-tcpwrappers
-
repo owner OK. Please download http://www.mailpiler.org/download/piler-binaries-amd64-master-branch-build-845.tar.gz
It's a 64-bit compilation of piler build 845. Please overwrite your binaries, drop your current database and /var/piler/store/00/* again (sorry), and let's try another round of importing.
Make sure you have the catdoc, tnef, unrtf and poppler-utils packages installed.
Other ubuntu users also have a hard time, and I'm still trying to solve your and their problem. If the debian compiled version works for you, then I'll provide a deb package.
I also created piler_0.1.25-master-branch-build-845_amd64.deb in the download directory of the above link, however it has piler.conf too, so if you prefer this, then make sure you don't overwrite your current piler related config and key files.
Btw. are you familiar creating deb packages with a preinstall and postinstall script? I created this deb package with checkinstall, but it doesn't offer such a feature.
-
reporter So I've installed few more packages: tnef poppler-utils libzip-dev ppthtml. Unfortunately I'm not familiar with creating deb packages. Latest master branch make failed:
gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c dirs.c -o dirs.o dirs.c: In function ‘createdir’: dirs.c:59:18: warning: ignoring return value of ‘chown’, declared with attribute warn_unused_result [-Wunused-result] gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c base64.c -o base64.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c misc.c -o misc.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c counters.c -o counters.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c cfg.c -o cfg.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c sig.c -o sig.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c decoder.c -o decoder.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c hash.c -o hash.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c parser.c -o parser.o parser.c: In function ‘parse_line’: parser.c:217:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] parser.c:240:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] parser.c:289:24: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] parser.c:486:21: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c parser_utils.c -o parser_utils.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c rules.c -o rules.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c session.c -o session.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c message.c -o message.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c attachment.c -o attachment.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c digest.c -o digest.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c store.c -o store.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c archive.c -o archive.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c tai.c -o tai.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c import.c -o import.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c imap.c -o imap.o imap.c: In function ‘process_imap_folder’: imap.c:200:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] imap.c:210:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c pop3.c -o pop3.o pop3.c: In function ‘process_pop3_emails’: pop3.c:179:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] pop3.c:189:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c extract.c -o extract.o extract.c: In function ‘extract_tnef’: extract.c:169:10: warning: ignoring return value of ‘system’, declared with attribute warn_unused_result [-Wunused-result] extract.c: In function ‘unzip_file’: extract.c:120:24: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c mydomains.c -o mydomains.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c retr.c -o retr.o gcc -O2 -Wall -g -fPIC -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c mysql.c -o mysql.o ar cr libpiler.a dirs.o base64.o misc.o counters.o cfg.o sig.o decoder.o hash.o parser.o parser_utils.o rules.o session.o message.o attachment.o digest.o store.o archive.o tai.o import.o imap.o pop3.o extract.o mydomains.o retr.o mysql.o ranlib libpiler.a gcc -shared -o libpiler.so.0.1.1 dirs.o base64.o misc.o counters.o cfg.o sig.o decoder.o hash.o parser.o parser_utils.o rules.o session.o message.o attachment.o digest.o store.o archive.o tai.o import.o imap.o pop3.o extract.o mydomains.o retr.o mysql.o ln -sf libpiler.so.0.1.1 libpiler.so ln -sf libpiler.so.0.1.1 libpiler.so.0 gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o piler piler.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerconf pilerconf.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerget pilerget.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilergetd pilergetd.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pileraget pileraget.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerimport pilerimport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. gcc -O2 -Wall -g -I. -I.. -I/usr/include/mysql -DBIG_JOINS=1 -fno-strict-aliasing -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerpurge pilerpurge.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L. pilerpurge.c: In function ‘main’: pilerpurge.c:397:10: error: ‘dry_run’ undeclared (first use in this function) pilerpurge.c:397:10: note: each undeclared identifier is reported only once for each function it appears in make[1]: *** [pilerpurge] Error 1 make: *** [all-recursive] Error 1
-
repo owner Hmm, the very latest master branch compiles on my ubuntu 12. However please try my binary packages, too.
-
reporter Piler: cant connect to mysql server
-
repo owner I suspect that the deb package just overwrote your piler.conf. Please check the mysql section, and fix it if necessary.
-
reporter pilerimport -e b45624167bf7cfdc2bfdbd7767adcf6287b18c2f.eml error parsing the configfile: /usr/local/etc/piler.conf ERR: reading cipher key
-
repo owner Please create it:
dd if=/dev/urandom bs=56 count=1 of=/usr/local/etc/piler.key chmod 600 /usr/local/etc/piler.key
-
reporter Still same
-
repo owner Sorry: add 640 permissions on piler.key, and make it to group piler. Also please make sure that /usr/local/bin/pilerimport is owned by piler user and setuid.
-
reporter Once again messages corrupted - failed verification.
-
reporter One more experiment, configured on mail server always_bcc option in postfix. When receiving mails in piler got following(for any mail arriving) in syslog:
Oct 1 17:16:15 piler piler[24896]: 40000000524ad939134042c400c12596605f: mkdir /var/piler/store/00/524/60/5f: error=No such file or directory Oct 1 17:16:15 piler piler[24896]: 40000000524ad939134042c400c12596605f: cannot open: /var/piler/store/00/524/60/5f/40000000524ad939134042c400c12596605f.m Oct 1 17:16:15 piler piler[24896]: 40000000524ad939134042c400c12596605f: error storing message: 40000000524ad939134042c400c12596605f.m
Permissions seems to be ok:
ls -l /var/piler/ total 20 drwx------ 2 piler piler 4096 Oct 1 15:28 imap drwxr-xr-x 2 piler piler 4096 Oct 2 09:05 sphinx drwxr-xr-x 2 piler piler 4096 Oct 1 15:35 stat drwx------ 3 piler piler 4096 Oct 1 15:35 store drwx--x--x 2 piler piler 4096 Oct 2 08:31 tmp
-
repo owner OK, let me try importing the 2 emails you sent to a test ubuntu installation.
-
reporter Want me to send them one more time?
-
reporter Any news? Maybe for now it's best option to download OVA image and play around with it?
-
repo owner Not yet. You are right, it's a good idea to try the OVA image. On my test box I couldn't reproduce the issue. Need some more time to figure out what's the problem at you.
-
repo owner - attached 1.txt
contents of attachment table after importing the 2 emails you sent me for debugging.
-
repo owner As you can see for some reasons the piler_id field is empty (can you verify the same for these 2 emails?). On centos at least it's filled, and on the demo site (32 bit Slackware Linux) it's filled properly, and pilerget returns both messages correctly.
-
reporter Piler_id is filled for every record in attachment table
-
repo owner Ok, here's my last experiment on ubuntu 12.04. I started with a minimal install, and used http://www.mailpiler.org/download/piler_0.1.25-master-branch-build-845_amd64.deb and installed the required packages + had to configure a bunch of things manually (create piler config, key, database, etc.)
Then when it was ready, I started the piler daemon and sent 20-30 test emails (all with attachments, including your 2 test emails), and piler processed them properly, I was able to get both of your emails.
Finally I dropped the database, recreated it, and this time I used pilerimport. The results are the same: archived all properly, and managed to retrieve the 2 of your messages properly. usingi downloaded
-
repo owner Do you think it's worth to give you a shell account to this virtual machine to check it for yourself with some real emails? After you are done I'll drop the whole VM. (Note that only the piler binaries are installed, not gui isn't). If you are interested then drop me an email, and send you the details.
-
reporter Seems that there is something in my config went wrong. For now I switched to OVA image on debian, no success over there yet, pilerimport fails:
Oct 9 15:20:07 piler kernel: [ 2416.535416] pilerimport[3869]: segfault at 7fffdac02000 ip 00007f7324e7fc85 sp 00007fffdab853a8 error 6 in libpiler.so.0.1.1[7f7324e66000+21000]
It failed at some point running pilerimport -i, but still imported part messages successfully.
-
reporter On that point with my understanding of how piler works giving me shell is no use, on my ubuntu box everything looks good but not working as intended.
-
repo owner Is it possible to see this email causing segfault for pilerimport?
-
reporter Hard to tell:
processing folder: INBOX... found 181 messages duplicate: INBOX-146.txt (id: 40000000525661992013d48400469169fa0d) duplicate: INBOX-148.txt (id: 4000000052566199215030cc00b94ff89934) duplicate: INBOX-155.txt (id: 400000005256619a07cb431c0097184deccc) Segmentation fault
Oct 10 10:13:04 piler kernel: [ 237.096530] pilerimport[2947]: segfault at 7fffb0378000 ip 00007fb538a0dc85 sp 00007fffb02fac38 error 6 in libpiler.so.0.1.1[7fb5389f4000+21000]
-
repo owner Do you have an INBOX-156.txt file? If so then please run pilertest on it, eg. pilertest /path/to/INBOX-156.txt. Pilertest is in the source tree of piler. If pilertest can show you some statistics about the message, then run it 20-30 times:
for i in
seq 1 20
; do ./pilertest /path/to/INBOX-156.txt; sleep 2; doneand see if it does the job properly all 20 times.
Btw. if you re-run the import, does it crashes again after the 155th message?
-
reporter It does crashes every time after 155th message. I don't know how to find that particular message, because I ran pilerimport -i from imap server.
-
repo owner pilerimport writes the downloaded email to a file, and then it processes it. So please look the current directory, and look for files named like INBOX-xxxx.txt where xxxx is a serial number indicating the currently imported message. There is a "duplicate: INBOX-155.txt" which means that the 155th message was imported. So look for the next, ie. INBOX-156.txt. Do you have it? If so than pilerimport managed to download the file. In this case check the file if it's a complete message.
-
reporter Found it:
ls -l /tmp/ | grep INBOX -rw------- 1 piler piler 490982 Oct 10 13:12 INBOX-156.txt -rw------- 1 root root 2837 Oct 10 14:13 INBOX-156.txt.a1 -rw------- 1 root root 287297 Oct 10 14:13 INBOX-156.txt.a2 -rw-r--r-- 1 root root 209943 Oct 10 14:13 INBOX-156.txt.a2.bin -rw------- 1 root root 192025 Oct 10 14:13 INBOX-156.txt.a3 -rw-r--r-- 1 root root 140322 Oct 10 14:13 INBOX-156.txt.a3.bin -rw------- 1 root root 8973 Oct 10 14:13 INBOX-156.txt.m
Crashes every time:
./pilertest /tmp/INBOX-156.txt locale: en_US.UTF-8 build: 836 parsing... post parsing... Segmentation fault
-
repo owner OK, is it possible to see INBOX-156.txt?
-
reporter Weird thing is that I don't have that mail in my inbox :)
-
repo owner See, email archiving is how useful? :-)
-
reporter How can pilerimport try to archive message I already deleted?
-
repo owner It must have been left on the server, and now it presented the allegedly deleted email to pilerimport.
-
reporter Maybe there should be some kind of exception catcher, because of that segfault I cannot import rest of messages.
-
repo owner Well, it's not really an exception (C language doesn't support try {} catch {} blocks like c++ or java), but a fault in the program that aborts pilerimport. However I can't really help it unless I have the chance to see it, and figuring out where and why it actually crashes. If it's a sensitive email then we need a remote troubleshooting: I create a deb package for you with tons of debugging messages, then you run it again, and report back the messages printed out, then I try something, make a new deb, you install it, run it, report back, etc. until the cause is found and fixed. Expect to finish this iteration by Christmas of 2014 :-)
-
reporter That mail contained broken docx attachment, and it does contain sensitive information...
-
repo owner OK, no problem. Can you do the following? Copy INBOX-156.txt to another file, eg. mail1.eml, then edit mail1.eml and remove the attachment. Then run pilertest on mail1.eml to see if there's anything different this time.
-
repo owner And another idea: login with your email application, and move this problematic email to a different folder, then try running pilerimport again.
-
reporter Edited INBOX-156.txt, removed part with broken attachment and imported it successfully.
-
repo owner At least that's a good news. Will you try something else? Edit the original copy of INBOX-156.txt, and change the message-id (eg. change its last letter). Then change the attachment type of this broken one to something unsupported to prevent any external helper to interfere, and try to pilertest it again.
-
reporter Imported successfully with changed attachment type.
-
repo owner OK, that's a good news. Now please describe how broken that attachment? If you save the attachment, can you open it with its application, eg. microsoft office, whatever? Or the application says it's corrupted, etc.
-
reporter Some how file corrupted, Microsoft Word cannot open it(recover failed too).
-
repo owner That's good - at least from the point of fixing the bug. Is it possible to take INBOX-156.txt, and remove everything before and after this problematic attachment, just leave it with its mime parts, and passing it to me for analysis? I'd rather pursue this issue until it's fixed. Then you could be able to import the rest after this message, too.
-
reporter Sent it to you by mail.
-
repo owner Hmm, I got INBOX-155.txt, although pilertest can't process it, so perhaps it's the poblematic email. However I couldn't reproduce the segfault issue. I'll move this email to the ubuntu VM, and testing further.
-
repo owner Done with ubuntu. I used the deb package, and it archived the email, although the attachment is not searchable.
-
repo owner Just wondering if there's any news or update on the issue?
-
repo owner No news is good news. If you have any updates, then reopen the issue.
-
repo owner - changed status to resolved
- Log in to comment
please try to get a problematic message from the command line using the pilerget command. pilerget expects a piler id, eg. 4000000......484784abcdef. To get this id, just download the message, and its name will be the piler id. Also show me its metadata:
select * from metadata where piler_id='4000000......';