Segmentation fault reindex folder (-F option)

[Former user]

Hi, First, thank you for Piler and your work. It's a really good application. But i have got a problem when i launch reindex with folder option like that : /usr/local/bin/reindex -F test -a Resultat is : segmentation fault :(

~# grep reindex /var/log/kern.log Feb 12 16:48:36 store1 kernel: : [1217801.561172] reindex[17158]: segfault at 7fb08fe949a0 ip 00007fb08fe949a0 sp 00007fffa80aafb8 error 15 Feb 12 16:48:36 store1 kernel: : [1217801.769082] reindex[17159]: segfault at 7f34d69e49a0 ip 00007f34d69e49a0 sp 00007fff541a5618 error 15

MySQL version : ~# mysql -V mysql Ver 14.14 Distrib 5.5.35, for debian-linux-gnu (x86_64) using readline 6.2

Piler version : ~# piler -v piler 0.1.25-master-branch, build 861, Janos SUTO [e-mail] Build Date: Wed Feb 12 16:33:50 CET 2014 ldd version: ldd (Debian EGLIBC 2.13-38+deb7u1) 2.13 gcc version: gcc version 4.7.2 (Debian 4.7.2-5) Configure command: ./configure --localstatedir=/var --with-database=mysql --enable-starttls --enable-tcpwrappers

LDD result : ~# ldd /usr/local/bin/reindex linux-vdso.so.1 => (0x00007fff8bb5c000) libpiler.so => /usr/local/lib/libpiler.so (0x00007f7b09527000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7b09310000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7b0908d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7b08e89000) libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f7b08a92000) libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f7b08831000) libtre.so.5 => /usr/lib/x86_64-linux-gnu/libtre.so.5 (0x00007f7b08621000) libzip.so.2 => /usr/lib/libzip.so.2 (0x00007f7b08414000) libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007f7b08208000) libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f7b07ff0000) libmysqlclient.so.18 => /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18 (0x00007f7b07aa9000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7b0788c000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7b07684000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7b072f9000) /lib64/ld-linux-x86-64.so.2 (0x00007f7b09756000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7b06ff1000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7b06ddb000)

I've attached strace -ff -o reindex output.

Thanks.

[Janos SUTO]

Hmm, it seems odd that it segfault right after trying to read the encryption key. Do you have any archiving or retention rules? Can you rerun reindexing without the -F switch, eg. "reindex -a"? If it fails, then try reindex -f 1 -t 1

[Tim CLERC]

reindex -a or reindex -f 1 -t 1 work perfectly. It's a piler fresh installation and i have no archiving rules or retention rules. :( How can i help you ?

~# ls -l /usr/local/etc/piler.key -rw-r----- 1 piler piler 56 févr. 12 16:44 /usr/local/etc/piler.key

~# file /usr/local/etc/piler.key /usr/local/etc/piler.key: data

[Janos SUTO]

Do you have any stale files, after the segfault? The reindex utility creates some temp files. I suspect that it can't clean them up due to the error, and those files may help us. If no luck, then a remote session (eg. teamviewer) might help.

[Tim CLERC]

Thanks for your reply and i don't have any stale files... :( I'm not a developper but i debug program (by displaying strings) and reindex program failed when function get_folder_id is called at line 226 in src/reindex.c. Then, i edit function get_folder_id in src/import.c and i found than prepare_sql_statement failed at line 139...I edit function prepare_sql_statement in src/mysql.c and i found than mysql_stmt_prepare failed at line 216...I don't understand why.

Packages libmysql installed on system :

libmysql++-dev | version: 3.1.0-2+b1

libmysql++3 | version: 3.1.0-2+b1

libmysqlclient-dev | version: 5.5.35+dfsg-0+wheezy1

libmysqlclient18:amd64 | version: 5.5.35+dfsg-0+wheezy1

libmysqld-dev | version: 5.5.35+dfsg-0+wheezy1

[Janos SUTO]

You mentioned that reindex -f 1 -t 1 had worked properly. Now please try it with a larger set of emails, eg. reindex -f 1 -t 1000. If it works, then try reindex -a.

The folder feature is kinda experimental, it was designed for a specific user. Most users don't need this feature, so probably you will be fine without using -F flag. Anyway it's definitely a bug, and I'm going to fix it.

[Janos SUTO]

Just fixed it in the master branch.

[Tim CLERC]

Really thanks ! I have no more segmentation fault now :) But reindex folder doesn't work correctly. For my tests, i have two folders, directory1 (id 1) and directory2 (id 2). I execute reindex -a -F directory1 command and all messages, even those who are in directory2, are imported in sph_index table with id 1. Consequence, user who have access to directory1 can access to all messages of directory 2. Maybe because metadata table have not folder column ? I'm sorry to disturb you for that :(

[Janos SUTO]

Just a question to understand your needs better: why do you need the folder feature? For what purpose?

[Tim CLERC]

We need folder feature for three reasons :

Sometimes a user can have messages with no FROM field or no TO field and when in same message FROM field or TO field don't match with user email address, nobody can see these messages. This is why we need link messages to a folder.

We have many alias or redirections. Users receive many mails with TO field who don't match with user email. Then, nobody can see these messages in Piler.

When an employe leave company, how to delete all the messages which he has access? For that, i think folder feature is more easy, no ?

[Janos SUTO]

That's a bit unusual to get a legitimate email without From or To fields. I saw some spam emails like that.

Normally when a user logs in, the gui queries all his email addresses from ldap, AD, etc. Then the user can see an email sent to a distribution list even if his primary address is not present in the mail headers. So aliases and redirections shouldn't be a problem as long as the gui is able to figure out a person's all email addresses.

Regarding the message removal. The main purpose of the archive is to save emails no matter what, so there's no easy way to remove messages. Usually pilerpurge is to the rescue, as it can remove any message that its retention time is reached. Never tried to remove based on folder parameters. Feel free to experiment with it.

[Janos SUTO]

I have an idea for the missing From/To field. What if instead of specifying a folder you could add the recipient email address by specifying a commandline option? Then you wouldn't need the folder stuff.

[Tim CLERC]

Hi,

I'm sorry to answer so late...Since your ticket of 2014-02-20, i installed OpenLDAP on a test server and add users with alias attributes. That works great :) Thanks for your help ! Your idea is very interessting and but do you think this option breaks the limitation of BCC or emails from mailing list ?

[Janos SUTO]

What limitation? You said emails don't have From nor To lines.

[Tim CLERC]

Yes we have emails don't have From nor To lines but we have too emails from mailing lists, who are very important and emails with BCC field and nobody can see them. I'm really sorry for my english....

[Janos SUTO]

Hold it! How come that you have emails with no From/To, but you have emails with Bcc header line? It must not be present, and the first MTA should strip it AFAIK.

[Tim CLERC]

I am sorry, I did not mean that e-mails have fields BCC but only that we have many e-mails where the addressee is in BCC and it is very important for us that these e-mails can be seen and the option which you propose allows to solve that?

[Janos SUTO]

If an email has valid From/To/Cc addresses, then no problem. However what to do with emails without any email address in the header? Could they be post processed to add a recipient to the email? The piler permissions are based on the header email addresses.

[Tim CLERC]

Yes i understand and we can't use Piler at the moment because we have many important mails from external mailing lists or where recepients is in BCC. Do you think that will be possible to see those mails in Piler in the future ?

[Janos SUTO]

I'm sorry to hear that you have to postpone the deployment of piler, but I can only repeat myself: if the From/To/Cc headers in the given email are not correct (or missing) then they must be fixed first, because the permissions are based on these headers.

[Janos SUTO]

Anyway the segfault issue is fixed. If you want to discuss the permission issue further, please open a new issue.