Reindex -a filled the Disk up to almost 100%

[MS]

Hi,

we had the problem that our Piler did not show emails which where newer than May this year. At this point we had >100GB left free. And the syslog showed that new emails where still received and stored (status=stored)but did not appear in the GUI. I read up through the issues here and thought that a full reindex would fix the problem.
So i did the following steps:
searchd --stop
rm /var/piler/sphinx/*
indexer --all
reindex -a
The reindexing took quite some time (we have ~4.7 Million Emails) and then the disk was almost full (only around 3GB left). I saw that the sph_index table was pretty big. In an attempt of making space again free i removed all entries of the sp_index table. But that did not free up any space.

Now the syslog shows for new mails coming in a status=duplicate.

And the reindex created a lof of folders in the /var/piler/store/00 folder and subfolders.

it seems as if the reindex -a tried to copy all the mails which are already in the store folder?

However I really would need help to get the system back on track, as I’m neither really proficient in Linux nor Piler.

Thanks!

[Janos SUTO]

Perhaps I will extend the reindex docs to explain what it does. In a nutshell it retrieves stored emails from disk, reads and them, the put the textual data to spx_index table. If you did a reindex -a, then it means all your ~5M emails will appear in this table making it big on disk. Now I assume you don’t have innodb files per table option turned on which would write sph_index table data to a separate file on disk, so it was written to ib_data or similar which is tricky to get shrinked. It’s out of the scope of this forum anyway. If this is the case, then look for mysql forums as well on the matter.

[MS]

Thanks. After adding additional diskspace we are now in the process of reindexing it in smaller packets.

And so far it seems to work fine.