Migrated Server & Indexer (Sphinx > Manticore) - reindex problems

[Thomas Klosinsky]

So,

we’ve just more or lesss successfully migrated servers and are now using the piler docker version 1.4.1.

Piler is getting emails and filling metadata table. Pilerget also gets all the stored emails…

I am not sure, if the the rtindex is working since the table piler1 in Manticore database only shows 20 entries. Always. But since they are always different, I assume that something works at least.

So, since we switched indexers from Sphinx to Manticore, our Sphinx index either way is unusable for us.
I need to reindex. So I wrote a small script which always reindexes a batch of 1000 emails using the reindex command.

With all services running (searchd & piler, basically) I get this error for each email to be indexed:

failed to add to sph_index table: mailid.eml

How can I reindex all emails?

Maybe it makes sense to modify the documentation with this case.

Best regards!
Thomas

[Thomas Klosinsky]

When I run: /usr/libexec/piler/indexer.delta.sh
FATAL: no tables found in config file '/etc/piler/manticore.conf'
FATAL: no tables found in config file '/etc/piler/manticore.conf'

[Thomas Klosinsky]

OK, got it…. the reindexer is in deed working…. I started it once and up to that point the reindexer throws an error for the sph_index table… afterwards it’s working fine

‌

So this is not a bug, but a misleading error message.

[Janos SUTO]

Both sphinx and manticore displays 20 hits by default. You may fix the select query to ask for more. When you use RT index, you don’t need the index.*.sh script any longer.

If you set rtindex=1 in piler.conf, then both reindex and the piler daemon should write index data directly to manticore. When you get the “failed to add” error, then check if the given index id is in the piler1 manticore table.

[Thomas Klosinsky]

Thanks!

So it would make sense to clear the sph_index table from the piler database and generally remove the indexer scripts from cronjobs of user piler in the docker image, right?

[Janos SUTO]

That’s correct.