Reindex old data

[Former user]

Hi! I'm using vmware image. Everytime the server is rebooted, piler search does not return any results. Data is collected in Mysql database, but search returns only data which is collected after reboot. Sphinxsearch indexer runs on "sph_index" table then puts data in "attachment", "metadata", "rcpt" tables and deletes from "sph_index". After reboot /var/piler/sphinx contents are purged and piler search does not have results to return until it creates index files from current data in "sph_index" table. Maybe I'm doing something wrong. My piler server currently has 109151 email messages regarding to health monitor.

[Janos SUTO]

Hello!

I'd like to know a bit more about your setup. Please show me the output of the following commands:

piler -v
pilerconf (don't include any password)
ls -la /var/piler
ls -la /var/piler/sphinx
df -h
searchd -v|head -3

Do you use the stock sphinx.conf from the _PILER_ distribution without any modifications (except the passwords)? If so, please enable the logging (in the last section of sphinx.conf, searchd), then restart searchd, let it run for a while to process new messages, then let's reboot your virtual machine, and see

ls -la /var/piler/sphinx

and the logs.

If you have any additionally installed / configured / etc. something that might cause problems, please include it, too.

[Janos SUTO]

Some additional ideas. Before rebooting disable the rc.piler and rc.searchd scripts to prevent them starting both piler and searchd. Then check the /var/piler/sphinx directory that everything is fine. Then start piler, check /var/piler/sphinx. Finally start searchd and check it again.

I did a stock install in a vmware VM following the documentation, then processed a few hundred messages, then rebooted, and after coming back, no problem with the sphinx search data.

ps. were there any changes to the piler sql schema and in the sphinx.conf sql_query statements?

[Janos SUTO]

After carefully examining the problem, I couldn't reproduce the loss of sphinx index data. However I added a new utility called reindex which can regenerate sphinx indices using the metadata table and the stored encrypted files.

The reindex utility is available from version 0.1.18.