auditor returns zero search results

[Christian]

Hi,

I have tried to reindex witch “indexer --all” but it only seems to index the att1

indexing index 'main1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.003 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'main2'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.002 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'main3'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.002 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'main4'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.001 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'dailydelta1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.001 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'delta1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.004 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'tag1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.001 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'note1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.001 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'att1'...
collected 234356 docs, 24.4 MB
sorted 1.9 Mhits, 100.0% done
total 234356 docs, 24379339 bytes

what can i do to troubleshoot this?

[Janos SUTO]

Read the docs first. Really. indexer --all initializes or resets the index data. It’s not reindexing.

Now run reindex -a command, then run indexer.delta.sh as user piler. Then show me the output of ls -la /var/piler/sphinx

[Christian]

i did a

rm -R /var/piler/sphinx/*

before running indexer --all as suggested in Ticket Issue #535.

The output of reindex -a is

$ reindex -a
put 370882 messages to sph_index table for reindexing
you have mail

‌

[Christian]

it works now. Thanks.

[Christian]

The problem occurs again. I suppose it has to do with the cron jobs. Yesterday evening after "reindex -a" it worked.

[Christian]

I have the following cron jobs:

*/15 * * * * /usr/bin/indexer --quiet delta1 --rotate && sleep 2 && /usr/bin/indexer --quiet --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate

*/15 * * * * /usr/bin/indexer --quiet tag1 --rotate

*/15 * * * * /usr/bin/indexer --quiet note1 --rotate

could this be the reason for no results with the auditor account?

[Janos SUTO]

What piler and sphinx versions do you have? Where did these cron jobs come from? Also how do you start searchd?

[Christian]

I have a containerized (docker) piler 1.3.8 and sphinx version 2.2.11-id64-release. The cron jobs come from the manual post install page on mailpiler.org (http://www.mailpiler.org/wiki/current:post-install).
I start searchd via “service rc.searchd start”.

[Janos SUTO]

My bad. I’ve fixed that page. Be sure to update the cron jobs. Also make sure nobody ever runs indexer --all. Only user piler should touch the sphinx index files.

[Christian]

Thanks. I updated the cron jobs and ran a reindex -a.
Sadly it did not solve my problem.
There are still zero results in auditor account.
Then i purged the directory /var/piler/sphinx and ran indexer --all again. That did not help either. Now im running reindex -a again, this will take a few hours.
What can i do to troubleshoot this?

[Janos SUTO]

OK, let’s summarize it:

• don’t run indexer --all
• make sure nobody ever touches the sphinx index files except piler

After reindexing you should have all your indices back. Then do the following. Record the file sizes in /var/piler/sphinx

Then every 15 minutes do the same. At some point you may notice that your indexed data is gone, ie. the index files are small. Then figure out what’s going on at that time.

‌

[Christian]

No, that did not work. I get no results as auditor.
I get this output from reindex -a

...
failed to add to sph_index table: 370918.eml
failed to add to sph_index table: 370919.eml
failed to add to sph_index table: 370920.eml
failed to add to sph_index table: 370921.eml
failed to add to sph_index table: 370922.eml
failed to add to sph_index table: 370923.eml
processed:        0 [  0%]
put 0 messages to sph_index table for reindexing
piler@a5ca76dde0eb:/tmp$

The strange thing is last time it said processed 100%.

I attached a file with the output of ls -lah /var/piler/sphinx
I don’t know what is wrong, i did not change anything besides recreate the index and do reindex. Just the day before yesterday it worked (momentarily though, but still).

What can i do?

[Janos SUTO]

You got 0 processed, because the messages are already in the sph_index table. Run indexer.delta.sh script as user piler. It will read the sph_index table, and empty it, while updating the index files.

‌

[Christian]

I tried that. Still zero results in Auditor.

Might you be interested in trying to find the error via tmate? I would have permission to give you shell access to our server.

[Janos SUTO]

Yes, let me do an on site troubleshoot to figure out what’s wrong. Use piler -V to get my email address to discuss the details.

[Janos SUTO]

The on-site reindex has fixed the problem.