Message failed verification

Janos SUTO repo owner

please try to get a problematic message from the command line using the pilerget command. pilerget expects a piler id, eg. 4000000......484784abcdef. To get this id, just download the message, and its name will be the piler id. Also show me its metadata:

select * from metadata where piler_id='4000000......';

2013-09-26T12:50:42+00:00

fakegod reporter

So, pilerget output: From: XXX xxx@xxx.xx User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130221 Thunderbird/17.0.3 MIME-Version: 1.0 To: yyy@xxx.xx Subject: =?UTF-8?B?0KHRhdC10LzQsA==?= Content-Type: multipart/mixed; boundary="------------090504060504000705030004"

This is a multi-part message in MIME format. --------------090504060504000705030004 Content-Type: multipart/related; boundary="------------080004050200050508040005"

--------------080004050200050508040005 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

--------------080004050200050508040005 Content-Type: image/jpeg; name="Picture 1" Content-Transfer-Encoding: base64 Content-ID: part3.09080000.03070908@xxx.xx Content-Disposition: inline; filename="Picture 1"

ATTACHMENT_POINTER_400000005243d96b3155f77c0083324d3753.a1_XXX_PILER--------------080004050200050508040005--

--------------090504060504000705030004 Content-Type: application/vnd.ms-visio.viewer; name="Network Actual(03.2013).vsd" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Network Actual(03.2013).vsd"

Metadata: | 820 | xxx@xxx.xx | xxx.xx | Схема | 0 | 1380178273 | 1362497358 | 1583422158 | 0 | 6126916 | 381 | 1 | 2 | 400000005243d96b3155f77c0083324d3753 | 51360F4E.4020606@xxx.xx | | f2ade7c2e4c7950392486d6f3b84301411751345b8c342a9d5a1b9b4da062d39 | 6f19b2ed065abad4c7f430dd7db63a4cc448aef0df8ef3b60d1f00592c7509c5 | da459520defdf350f06357dff55dab6a1776936963c670e98f92b04b4733ae44 |

2013-09-26T13:28:17+00:00

Janos SUTO repo owner

ok, thanks. You clearly have some issue with the attachments. This email has two attachments. Now let's check the attachment table:

select * from attachment where piler_id='400000005243d96b3155f77c0083324d3753';

Also please show me: ls -la /var/piler/store/00/524/37/53

2013-09-26T19:59:20+00:00

fakegod reporter

mysql> select * from attachment where piler_id='400000005243d96b3155f77c0083324d3753'; Empty set (0.00 sec)

root@piler:~# ls -la /var/piler/store/00/524/37/53 total 2976 drwxr-x--- 2 piler piler 4096 Sep 26 09:51 . drwxr-x--- 6 piler piler 4096 Sep 26 09:52 .. -rw-r----- 1 piler piler 3034512 Sep 26 09:51 400000005243d96b3155f77c0083324d3753.a2 -rw-r----- 1 piler piler 1760 Sep 26 09:51 400000005243d96b3155f77c0083324d3753.m

2013-09-27T06:01:13+00:00

Janos SUTO repo owner

You have the attachments on disk, however they are missing from the attachment table. This table is used to assemble the original message from the separated mime parts.

Do you have anything in the attachment table? It seems that it's corrupted. Btw. Please show me the log entries related to 400000005243d96b3155f77c0083324d3753:

grep 400000005243d96b3155f77c0083324d3753 /var/log/mail.log

2013-09-27T07:40:04+00:00

fakegod reporter

Yes I do have records in attachments table, some messages in piler are verified, thought most part "failed verification". /var/log/mail.log contains nothing regarding this id.

2013-09-27T07:50:09+00:00

Janos SUTO repo owner

Please try searching in the older logs, eg. /var/log/mail.log.1, /var/log/mail.log*.gz I'd like to know what piler thought about the message when it was received.

2013-09-27T09:01:00+00:00

fakegod reporter

Already tried, it seems they not there because I imported all the mail by means of: pilerimport -i through console

2013-09-27T09:46:01+00:00

Janos SUTO repo owner

How many emails are in the archive? Are they all from pilerimport?

2013-09-27T13:03:07+00:00

fakegod reporter

Over 1k messages, all from pilerimport.

2013-09-27T13:28:41+00:00

Janos SUTO repo owner

Is it possible to drop the piler database and recreate it, and dropping everything under /var/piler/store/00/? If you have the messages, then please do so, and reimport again. I have little data to explain why some messages have entries in the attachment table and others why not.

2013-09-28T20:41:26+00:00

fakegod reporter

I've dropped database, run install once again. After installation imported 1 mail(from those with problem), this time not through imap but from eml file. This time piler shows all the attachments, but still shows "Message failed verification".

2013-09-30T06:57:56+00:00

Janos SUTO repo owner

Do you get the message properly when you download it? Is it the complete message?

2013-09-30T19:56:21+00:00

fakegod reporter

From piler interface I can download all attachments. When I restore it to mailbox, or download in eml it opens without attachments:

server_sched.log

ATTACHMENT_POINTER_4000000052491c4c319575640096171afdb0.a2_XXX_PILER--------------090108000308020304080109
Content-Type: text/x-log;
 name=erver.log"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
 filename=erver.log"

ATTACHMENT_POINTER_4000000052491c4c319575640096171afdb0.a3_XXX_PILER--------------090108000308020304080109
Content-Type: text/x-log;
 name=ntegra_web.log"
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment;
 filename=ntegra_web.log"

ATTACHMENT_POINTER_4000000052491c4c319575640096171afdb0.a4_XXX_PILER--------------090108000308020304080109
Content-Type: text/x-log;
 name=ntegra_jboss_sched.log"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
 filename=ntegra_jboss_sched.log"

ATTACHMENT_POINTER_4000000052491c4c319575640096171afdb0.a5_XXX_PILER--------------090108000308020304080109
Content-Type: text/x-log;
 name=ntegra_jboss.log"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
 filename=ntegra_jboss.log"

ATTACHMENT_POINTER_4000000052491c4c319575640096171afdb0.a6_XXX_PILER--------------090108000308020304080109
Content-Type: text/plain; charset=F-8;
 name=atalina.out.2013-08-22"
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment;
 filename=atalina.out.2013-08-22"

ATTACHMENT_POINTER_4000000052491c4c319575640096171afdb0.a7_XXX_PILER

In that message I noticed that first letter of attachments filename is missing. For example: filename=atalina.out.2013-08-22" - should be "catalina.out.2013-08-22"

2013-10-01T07:19:52+00:00

Janos SUTO repo owner

Do you have any of these problematic messages with attachments which is not sensitive? If so can you send me one or two messages to test with? See my address in piler -V

2013-10-01T08:20:45+00:00

fakegod reporter

I'll search for suitable messages. For now, during "pilerimport -i" I can see a lot of following messages:

Format a4 is redefined

2013-10-01T08:32:59+00:00

Janos SUTO repo owner

It's a warning from one of the attachment helper utilities (eg. pdftotext or another).

2013-10-01T08:36:56+00:00

fakegod reporter

Sent you couple of mails that did not passed verification in piler.

2013-10-01T08:49:04+00:00

fakegod reporter

Also noticed that pair of mails, completely identical except address they were sent to, one is verified - another failed verification.

2013-10-01T08:59:14+00:00

Janos SUTO repo owner

Thanks, I got them. My personal archive took them and processed without problems, and they are verified, have the attachments. Then I copied them to the archive manually, changed the message-id, then imported them manually by pilerimport, and those 4 messages are all OK.

I know you mentioned before, but what OS and piler version do you use?

2013-10-01T10:16:47+00:00

fakegod reporter

Ubuntu 12.04.3 LTS

piler 0.1.25-master-branch, build 844, Janos SUTO <sj@acts.hu>

Build Date: Wed Sep 25 17:14:35 EEST 2013
ldd version: ldd (Ubuntu EGLIBC 2.15-0ubuntu10.4) 2.15
gcc version: gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) 
Configure command: ./configure --localstatedir=/var --with-database=mysql --enable-starttls --enable-tcpwrappers

2013-10-01T11:16:24+00:00

Janos SUTO repo owner

OK. Please download http://www.mailpiler.org/download/piler-binaries-amd64-master-branch-build-845.tar.gz

It's a 64-bit compilation of piler build 845. Please overwrite your binaries, drop your current database and /var/piler/store/00/* again (sorry), and let's try another round of importing.

Make sure you have the catdoc, tnef, unrtf and poppler-utils packages installed.

Other ubuntu users also have a hard time, and I'm still trying to solve your and their problem. If the debian compiled version works for you, then I'll provide a deb package.

I also created piler_0.1.25-master-branch-build-845_amd64.deb in the download directory of the above link, however it has piler.conf too, so if you prefer this, then make sure you don't overwrite your current piler related config and key files.

Btw. are you familiar creating deb packages with a preinstall and postinstall script? I created this deb package with checkinstall, but it doesn't offer such a feature.

2013-10-01T11:30:07+00:00

fakegod reporter

So I've installed few more packages: tnef poppler-utils libzip-dev ppthtml. Unfortunately I'm not familiar with creating deb packages. Latest master branch make failed:

gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c dirs.c -o dirs.o
dirs.c: In function ‘createdir’:
dirs.c:59:18: warning: ignoring return value of ‘chown’, declared with attribute warn_unused_result [-Wunused-result]
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c base64.c -o base64.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c misc.c -o misc.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c counters.c -o counters.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c cfg.c -o cfg.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c sig.c -o sig.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c decoder.c -o decoder.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c hash.c -o hash.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c parser.c -o parser.o
parser.c: In function ‘parse_line’:
parser.c:217:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
parser.c:240:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
parser.c:289:24: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
parser.c:486:21: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c parser_utils.c -o parser_utils.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c rules.c -o rules.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c session.c -o session.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c message.c -o message.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c attachment.c -o attachment.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c digest.c -o digest.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c store.c -o store.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c archive.c -o archive.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c tai.c -o tai.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c import.c -o import.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c imap.c -o imap.o
imap.c: In function ‘process_imap_folder’:
imap.c:200:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
imap.c:210:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c pop3.c -o pop3.o
pop3.c: In function ‘process_pop3_emails’:
pop3.c:179:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
pop3.c:189:18: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c extract.c -o extract.o
extract.c: In function ‘extract_tnef’:
extract.c:169:10: warning: ignoring return value of ‘system’, declared with attribute warn_unused_result [-Wunused-result]
extract.c: In function ‘unzip_file’:
extract.c:120:24: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c mydomains.c -o mydomains.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c retr.c -o retr.o
gcc -O2 -Wall -g  -fPIC -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c mysql.c -o mysql.o
ar cr libpiler.a dirs.o base64.o misc.o counters.o cfg.o sig.o decoder.o hash.o parser.o parser_utils.o rules.o session.o message.o attachment.o digest.o store.o archive.o tai.o import.o imap.o pop3.o extract.o mydomains.o retr.o  mysql.o
ranlib libpiler.a
gcc -shared -o libpiler.so.0.1.1 dirs.o base64.o misc.o counters.o cfg.o sig.o decoder.o hash.o parser.o parser_utils.o rules.o session.o message.o attachment.o digest.o store.o archive.o tai.o import.o imap.o pop3.o extract.o mydomains.o retr.o  mysql.o
ln -sf libpiler.so.0.1.1 libpiler.so
ln -sf libpiler.so.0.1.1 libpiler.so.0
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o piler piler.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.    
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerconf pilerconf.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.  
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerget pilerget.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.  
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilergetd pilergetd.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.    
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pileraget pileraget.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.  
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerimport pilerimport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.  
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.  
gcc -O2 -Wall -g  -I. -I..  -I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing  -g -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerpurge pilerpurge.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -lzip   -L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -lrt -ldl -L.  
pilerpurge.c: In function ‘main’:
pilerpurge.c:397:10: error: ‘dry_run’ undeclared (first use in this function)
pilerpurge.c:397:10: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [pilerpurge] Error 1
make: *** [all-recursive] Error 1

2013-10-01T12:01:48+00:00

Janos SUTO repo owner

Hmm, the very latest master branch compiles on my ubuntu 12. However please try my binary packages, too.

2013-10-01T12:14:56+00:00

fakegod reporter

Piler: cant connect to mysql server

2013-10-01T12:37:39+00:00

Janos SUTO repo owner

I suspect that the deb package just overwrote your piler.conf. Please check the mysql section, and fix it if necessary.

2013-10-01T12:40:46+00:00

fakegod reporter

pilerimport -e b45624167bf7cfdc2bfdbd7767adcf6287b18c2f.eml error parsing the configfile: /usr/local/etc/piler.conf ERR: reading cipher key

2013-10-01T12:50:43+00:00

Janos SUTO repo owner

Please create it:

dd if=/dev/urandom bs=56 count=1 of=/usr/local/etc/piler.key chmod 600 /usr/local/etc/piler.key

2013-10-01T13:05:06+00:00

fakegod reporter

Still same

2013-10-01T13:11:33+00:00

Janos SUTO repo owner

Sorry: add 640 permissions on piler.key, and make it to group piler. Also please make sure that /usr/local/bin/pilerimport is owned by piler user and setuid.

2013-10-01T13:13:58+00:00

fakegod reporter

Once again messages corrupted - failed verification.

2013-10-01T13:29:24+00:00

fakegod reporter

One more experiment, configured on mail server always_bcc option in postfix. When receiving mails in piler got following(for any mail arriving) in syslog:

Oct  1 17:16:15 piler piler[24896]: 40000000524ad939134042c400c12596605f: mkdir /var/piler/store/00/524/60/5f: error=No such file or directory
Oct  1 17:16:15 piler piler[24896]: 40000000524ad939134042c400c12596605f: cannot open: /var/piler/store/00/524/60/5f/40000000524ad939134042c400c12596605f.m
Oct  1 17:16:15 piler piler[24896]: 40000000524ad939134042c400c12596605f: error storing message: 40000000524ad939134042c400c12596605f.m

Permissions seems to be ok:

ls -l /var/piler/
total 20
drwx------ 2 piler piler 4096 Oct  1 15:28 imap
drwxr-xr-x 2 piler piler 4096 Oct  2 09:05 sphinx
drwxr-xr-x 2 piler piler 4096 Oct  1 15:35 stat
drwx------ 3 piler piler 4096 Oct  1 15:35 store
drwx--x--x 2 piler piler 4096 Oct  2 08:31 tmp

2013-10-01T14:18:41+00:00

Janos SUTO repo owner

OK, let me try importing the 2 emails you sent to a test ubuntu installation.

2013-10-02T13:42:18+00:00

fakegod reporter

Want me to send them one more time?

2013-10-03T06:55:43+00:00

fakegod reporter

Any news? Maybe for now it's best option to download OVA image and play around with it?

2013-10-08T06:14:16+00:00

Janos SUTO repo owner

Not yet. You are right, it's a good idea to try the OVA image. On my test box I couldn't reproduce the issue. Need some more time to figure out what's the problem at you.

2013-10-08T07:17:34+00:00

Janos SUTO repo owner

attached 1.txt

contents of attachment table after importing the 2 emails you sent me for debugging.

2013-10-08T20:08:14+00:00

Janos SUTO repo owner

As you can see for some reasons the piler_id field is empty (can you verify the same for these 2 emails?). On centos at least it's filled, and on the demo site (32 bit Slackware Linux) it's filled properly, and pilerget returns both messages correctly.

2013-10-08T20:14:42+00:00

fakegod reporter

Piler_id is filled for every record in attachment table

2013-10-09T12:36:57+00:00

Janos SUTO repo owner

Ok, here's my last experiment on ubuntu 12.04. I started with a minimal install, and used http://www.mailpiler.org/download/piler_0.1.25-master-branch-build-845_amd64.deb and installed the required packages + had to configure a bunch of things manually (create piler config, key, database, etc.)

Then when it was ready, I started the piler daemon and sent 20-30 test emails (all with attachments, including your 2 test emails), and piler processed them properly, I was able to get both of your emails.

Finally I dropped the database, recreated it, and this time I used pilerimport. The results are the same: archived all properly, and managed to retrieve the 2 of your messages properly. usingi downloaded

2013-10-09T12:43:18+00:00

Janos SUTO repo owner

Do you think it's worth to give you a shell account to this virtual machine to check it for yourself with some real emails? After you are done I'll drop the whole VM. (Note that only the piler binaries are installed, not gui isn't). If you are interested then drop me an email, and send you the details.

2013-10-09T12:45:32+00:00

fakegod reporter

Seems that there is something in my config went wrong. For now I switched to OVA image on debian, no success over there yet, pilerimport fails:

Oct  9 15:20:07 piler kernel: [ 2416.535416] pilerimport[3869]: segfault at 7fffdac02000 ip 00007f7324e7fc85 sp 00007fffdab853a8 error 6 in libpiler.so.0.1.1[7f7324e66000+21000]

It failed at some point running pilerimport -i, but still imported part messages successfully.

2013-10-09T13:24:26+00:00

fakegod reporter

On that point with my understanding of how piler works giving me shell is no use, on my ubuntu box everything looks good but not working as intended.

2013-10-09T13:31:13+00:00

Janos SUTO repo owner

Is it possible to see this email causing segfault for pilerimport?

2013-10-09T13:42:18+00:00

fakegod reporter

Hard to tell:

processing folder: INBOX... found 181 messages
duplicate: INBOX-146.txt (id: 40000000525661992013d48400469169fa0d)
duplicate: INBOX-148.txt (id: 4000000052566199215030cc00b94ff89934)
duplicate: INBOX-155.txt (id: 400000005256619a07cb431c0097184deccc)
Segmentation fault

Oct 10 10:13:04 piler kernel: [  237.096530] pilerimport[2947]: segfault at 7fffb0378000 ip 00007fb538a0dc85 sp 00007fffb02fac38 error 6 in libpiler.so.0.1.1[7fb5389f4000+21000]

2013-10-10T08:13:49+00:00

Janos SUTO repo owner

Do you have an INBOX-156.txt file? If so then please run pilertest on it, eg. pilertest /path/to/INBOX-156.txt. Pilertest is in the source tree of piler. If pilertest can show you some statistics about the message, then run it 20-30 times:

for i in seq 1 20; do ./pilertest /path/to/INBOX-156.txt; sleep 2; done

and see if it does the job properly all 20 times.

Btw. if you re-run the import, does it crashes again after the 155th message?

2013-10-10T09:02:23+00:00

fakegod reporter

It does crashes every time after 155th message. I don't know how to find that particular message, because I ran pilerimport -i from imap server.

2013-10-10T11:14:11+00:00

Janos SUTO repo owner

pilerimport writes the downloaded email to a file, and then it processes it. So please look the current directory, and look for files named like INBOX-xxxx.txt where xxxx is a serial number indicating the currently imported message. There is a "duplicate: INBOX-155.txt" which means that the 155th message was imported. So look for the next, ie. INBOX-156.txt. Do you have it? If so than pilerimport managed to download the file. In this case check the file if it's a complete message.

2013-10-10T11:36:48+00:00

fakegod reporter

Found it:

 ls -l /tmp/ | grep INBOX
-rw------- 1 piler piler 490982 Oct 10 13:12 INBOX-156.txt
-rw------- 1 root  root    2837 Oct 10 14:13 INBOX-156.txt.a1
-rw------- 1 root  root  287297 Oct 10 14:13 INBOX-156.txt.a2
-rw-r--r-- 1 root  root  209943 Oct 10 14:13 INBOX-156.txt.a2.bin
-rw------- 1 root  root  192025 Oct 10 14:13 INBOX-156.txt.a3
-rw-r--r-- 1 root  root  140322 Oct 10 14:13 INBOX-156.txt.a3.bin
-rw------- 1 root  root    8973 Oct 10 14:13 INBOX-156.txt.m

Crashes every time:

 ./pilertest /tmp/INBOX-156.txt
locale: en_US.UTF-8
build: 836
parsing...
post parsing...
Segmentation fault

2013-10-10T12:15:02+00:00

Janos SUTO repo owner

OK, is it possible to see INBOX-156.txt?

2013-10-10T12:21:30+00:00

fakegod reporter

Weird thing is that I don't have that mail in my inbox :)

2013-10-10T12:21:32+00:00

Janos SUTO repo owner

See, email archiving is how useful? :-)

2013-10-10T12:34:38+00:00

fakegod reporter

How can pilerimport try to archive message I already deleted?

2013-10-10T12:45:54+00:00

Janos SUTO repo owner

It must have been left on the server, and now it presented the allegedly deleted email to pilerimport.

2013-10-10T12:48:08+00:00

fakegod reporter

Maybe there should be some kind of exception catcher, because of that segfault I cannot import rest of messages.

2013-10-10T12:58:04+00:00

Janos SUTO repo owner

Well, it's not really an exception (C language doesn't support try {} catch {} blocks like c++ or java), but a fault in the program that aborts pilerimport. However I can't really help it unless I have the chance to see it, and figuring out where and why it actually crashes. If it's a sensitive email then we need a remote troubleshooting: I create a deb package for you with tons of debugging messages, then you run it again, and report back the messages printed out, then I try something, make a new deb, you install it, run it, report back, etc. until the cause is found and fixed. Expect to finish this iteration by Christmas of 2014 :-)

2013-10-10T13:29:55+00:00

fakegod reporter

That mail contained broken docx attachment, and it does contain sensitive information...

2013-10-10T14:04:38+00:00

Janos SUTO repo owner

OK, no problem. Can you do the following? Copy INBOX-156.txt to another file, eg. mail1.eml, then edit mail1.eml and remove the attachment. Then run pilertest on mail1.eml to see if there's anything different this time.

2013-10-10T14:09:26+00:00

Janos SUTO repo owner

And another idea: login with your email application, and move this problematic email to a different folder, then try running pilerimport again.

2013-10-10T14:11:09+00:00

fakegod reporter

Edited INBOX-156.txt, removed part with broken attachment and imported it successfully.

2013-10-10T14:42:29+00:00

Janos SUTO repo owner

At least that's a good news. Will you try something else? Edit the original copy of INBOX-156.txt, and change the message-id (eg. change its last letter). Then change the attachment type of this broken one to something unsupported to prevent any external helper to interfere, and try to pilertest it again.

2013-10-10T15:21:29+00:00

fakegod reporter

Imported successfully with changed attachment type.

2013-10-11T08:14:50+00:00

Janos SUTO repo owner

OK, that's a good news. Now please describe how broken that attachment? If you save the attachment, can you open it with its application, eg. microsoft office, whatever? Or the application says it's corrupted, etc.

2013-10-11T08:19:09+00:00

fakegod reporter

Some how file corrupted, Microsoft Word cannot open it(recover failed too).

2013-10-11T08:49:54+00:00

Janos SUTO repo owner

That's good - at least from the point of fixing the bug. Is it possible to take INBOX-156.txt, and remove everything before and after this problematic attachment, just leave it with its mime parts, and passing it to me for analysis? I'd rather pursue this issue until it's fixed. Then you could be able to import the rest after this message, too.

2013-10-11T08:58:50+00:00

fakegod reporter

Sent it to you by mail.

2013-10-11T09:10:37+00:00

Janos SUTO repo owner

Hmm, I got INBOX-155.txt, although pilertest can't process it, so perhaps it's the poblematic email. However I couldn't reproduce the segfault issue. I'll move this email to the ubuntu VM, and testing further.