Old mails not found after reindex

[Mirko K. Mucko]

Hi there,

recently, I updated Piler to 1.4.4 and discovered, that old mails are no longer searchable via web interface. So, I switched from sphinx 2 to manticore 6. After several other issues (and switched to sphinx 3 for test purpose, later back to manticore), I’m still not able to search for old mails.

All new mails are indexed well, all mails bevore that are not searchable, but they appear in metadata and are accessable via pilerget !

So I decided to erease old files for indices (both sphinx and manticore) and re-index:

‌

> indexer --all -c /etc/piler/manticore.conf
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

using config file '/etc/piler/manticore.conf'...
indexing table 'main1'...
collected 0 docs, 0.0 MB
creating secondary index
total 0 docs, 0 bytes
[...]
> reindex -a -p -c /etc/piler/piler.conf
put 86755 messages to sph_index table for reindexing

‌

No errors appear, all “new” mails are (again) there, all old mails are still vanished. So I started to survey the mysql database and I see exactly this behaviour in sph_index: all new mails are filled correctly, for all old mails only ID, arrived, sent are filled, from(domain) and to(domain) are not filled. this seems to be my problem and besides the question how it came, the more important question is: how to solve?

Kind regards

Mirko

[Janos SUTO]

Check the metadata table, and verify the first 10 rows. Then run reindex -f 1 -t 10, and check if data is in the spx_index table. Also pay attention regarding the update of the install paths of old and new versions: are they they same?

[Mirko K. Mucko]

Janos, thanks for your support!

I checked the metadatatables, it seems to be ok: all fields are filled, especially fromand fromdomainare filled correctly. There are no fields for toor todomain.

I ran reindex -f 1 -t 10 and thecked sph_index, and it is as I wrote above:

from, to, fromdomain, todomain, bodyare empty, sizeand directionare 0. Only ID, arrived and sent are filled. Two recently arrived mails with IDs > 87.000 are correctly filled and accessable via frontend search.

‌

Any ideas?

‌

Cu

Mirko

[Janos SUTO]

I suspect that piler can’t retrieve emails from its store. Check the mail logs for errors.

[Mirko K. Mucko]

headbanging… you are so right. Top level path was correctly piler:piler, but below, files where root:root due to restore. Damn…. but now the story starts again…

‌

piler@piler:~/imap$ reindex -f 1 -t 10
Segmentation fault

‌

/var/log/kern.log and /var/log/messages say

Oct 25 11:16:10 piler kernel: [1130183.601650] reindex[1424004]: segfault at 0 ip 0000000000000000 sp 00007ffe18fa1738 error 14 in reindex[56290a8a3000+2000] likely on CPU 0 (core 0, socket 0)
Oct 25 11:16:10 piler kernel: [1130183.601672] Code: Unable to access opcode bytes at 0xffffffffffffffd6.

‌

[Janos SUTO]

Can you compile piler from source, and see if it helps?

[Mirko K. Mucko]

piler is already compiled from source, I’m back on Issue #1319 …

[Janos SUTO]

Check out this issue, it might be relevant: https://bitbucket.org/jsuto/piler/issues/1284/reindexer-has-problems-in-digest_string

Let me know, how it goes.

[Mirko K. Mucko]

That`s it, yes! I added one line as in the other issue and reindex works much, much besser! The password issue below…well….I realdy dunno…

‌

piler@piler:~$ reindex -f 1 -t 1000 -p -c /etc/piler/piler.conf
Command Line Error: Incorrect password
....

Command Line Error: Incorrect password
Command Line Error: Incorrect password
processed: 805 [ 80%]
put 805 messages to sph_index table for reindexing

and, more important, I can see those mails in polers web interface! THANKS a lot!

‌

Mirko

[Janos SUTO]

You're welcome.