Particular dates Mails are not shown in gui.

[B.JEGADEESAN]

We export the particular date mails from our mail server and import the .eml files to piler server all the .eml files are imported successfully.
In mysql we can see the metadata and we get the starting and ending ID for that dates.
Then we do re-indexing for that particular ID’s all the re-index process are completed.
And we can get mails with pilerget and pilerexport comment but that particular date mails are not shown in gui.

We had piler version 1.4.4 and sphinx version 3.3.1.

[Janos SUTO]

Check if those IDs are present in the index data, eg.

select * from main1,dailydelta1,delta1 where id=....;

‌

[B.JEGADEESAN]

We checked in index data the particular id is not shown in table.
And we checked in /var/piler/sphinx directory main1.tmp.* files are present (Size of the main1.tmp.spa is 45GB, main1.tmp.spd is 45GB and main1.tmp.spd is 25GB,) and all directory date is 28.11.2023.
Also we checked the dailydelta1 count in it's shown nearly 18Lakhs.
So please let us know if indexer.main.sh is properly running or not and if not can we run indexer.main.sh manually? I am attached screenshot here.

‌

[Janos SUTO]

It’s possible that the main1.* files have grown so large that indexer.main.sh has a hard time updating them. The problem is that when the indexer runs, it writes a new data file, and rotates it at the end. When you have 40+ GB data, and want to merge 50MB data, you still need to write 40 GB + 50 MB data, so as the archive grows the greater the burden is. This is why it’s worth to open a new main index after a while. So try editing indexer.main.sh, and replace “main1” with “main2”. Before doing so verify that main2.* files exist. Then edit config-site.php, and fix the index tables definition, eg.

$config['SPHINX_MAIN_INDEX'] = 'main1,main2,dailydelta1,delta1';

Then you need to reindex the missing block of emails again.

And yes, you may run indexer.main.sh manually as user piler.

[B.JEGADEESAN]

Okay if we change main1 to main2 indexer.main.sh file, then we start re-index for the missing block of emails.
After re-index completed that Main1.tmp.* files are merged and deleted right?
If not merged and deleted can we remove that files manually? And it cause any affect our existing main1 data's please let us know.

[Janos SUTO]

Yes, preferably the rotate should handle any tmp files. If they keep lingering then you should move them to a different directory, verify that the index data is still fine, and then you may remove them.

‌

[B.JEGADEESAN]

We take all the tmp.* files in another directory from current sphinx directory.
And changed “main1” with “main2” in indexer.main.sh file before that we checked in sphinx directory main2.* files are already present.
Then we set the sphinx_min_index configuration in config-site.php file.
Then we run the indexer.main.sh comment manually.
Now main2.tmp.* files are created in sphinx directory and we checked Before this dailydelta1 count is : 18lakhs and main1 is : 8crore in index data there main1, dailydelta1 count was not change after indexer.main.sh started.
Please let us know this process take some time for merging completed?
And we hope once merging completed then the main2.tmp.* files are deleted automatically right?

[Janos SUTO]

English, please.

[B.JEGADEESAN]

We moved all the tmp.* files from the current Sphinx directory to another directory. Before doing so, we replaced instances of "main1" with "main2" in the indexer.main.sh file. We confirmed that main2.* files were already present in the Sphinx directory. Subsequently, we configured the sphinx_min_index setting in the config-site.php file. Following this, we executed the indexer.main.sh command manually.

At this point, main2.tmp.* files were generated in the Sphinx directory. Before the indexer.main.sh process started, we verified that the dailydelta1 count was 18 lakhs and the main1 count was 8 crores in the index data. However, during the indexer.main.sh process, there was no change in the counts of main1 and dailydelta1.

We would like to understand whether the merging process takes some time to complete. Additionally, once the merging is finished, are the main2.tmp.* files expected to be deleted automatically?

[Janos SUTO]

Yes, the merging obviously takes some time, and the main2.tmp.* files are expected to replace the original main2.* files.

[B.JEGADEESAN]

It has been over 12 hours, but the main2.tmp.* files have not yet merged with main2. Upon checking the maillog during the initiation of the indexer.main.sh merging process, it was observed that the merging has not been completed thus far. Additionally, when attempting to execute indexer.delta.sh, an error was encountered. The error message start like as follows:

"ERROR: *** Oops, indexer crashed! Please send the following report to developers."

As a result, the main2.* index is not properly merged. We seek guidance on resolving these issues. And please let us know why this error occurred?

[Janos SUTO]

Please visit the sphinx forum (http://sphinxsearch.com/forum/forum.html?id=1) for support as it seems that it’s a bug in sphinx 3.3.1. The error message suggests that the output below should be sent to the sphinx developers. Unfortunately I cannot do much about issues in another software.