Specific date mails are not reindexing properly in Sphinx.
Issue #1340
closed
While migrating our Piler from old node to new node, we initiated the reindexing process for approximately 18 million emails.
However, we encountered an issue where certain date-specific emails failed to reindex.
Despite having previously faced a similar challenge and restarting the reindexing process from scratch, we find ourselves confronting the same problem once again, especially as reindexing is running concurrently through multiple processes.
Please let us know how to overcome the reindexing challenges and ensure the comprehensive completion of the process for all our data without loss.
We had a Piler version 1.4.4 and Sphinx version 3.3.1.
Comments (6)
-
repo owner
-
reporter
No we had a different sphinx version in old node we had 2.2.1 and new node had 3.3.1.
Copying the raw store file from old node to new node.To facilitate this process, we need clarification on whether to execute the following command for merging the .tmp files into the main files.
COMMAND: /usr/local/bin/indexer --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate
-
repo owner
Yes, the --rotate option is supposed to replace the original files with the *.tmp files.
-
reporter
We before run this command (/usr/local/bin/indexer --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate), We run the indexer.delta.sh cron manually but its showed the below error.
We had piler version 1.4.4 and Sphinx version 3.3.1.
ERROR:
WARNING: key 'log' is not multi-value; value in /usr/local/etc/piler/sphinx.conf line 218 will be ignor ed.
*** Oops, indexer crashed! Please send the following report to developers.
Sphinx 3.3.1 (commit b72d67b)
-------------- report begins here ---------------
Current document: docid=0, hits=0
Current batch: minid=0, maxid=0
Hit pool start: docid=0, hit=0
-------------- backtrace begins here ---------------
Program compiled with gcc 9
Configured with flags: '--enable-dl' '--with-mysql' '--with-pgsql' '--with-unixodbc' 'CXXFLAGS=-DSPHIN X_TAG= -DNDEBUG -O3 -g1' 'LDFLAGS=-static-libstdc++'
Host OS is Linux ubuntu16 4.4.0-116-generic#140
Stack bottom = 0x7ffeacbf63e0, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7ffeacc00000, stacksize=0x100000)
Trying system backtrace:
begin of system symbols:
indexer[0x64a983]
indexer[0x41db50]
/lib64/libpthread.so.0(+0x12cf0)[0x7f418f0c4cf0]
indexer[0x48e1d3]
indexer[0x4ab126]
indexer[0x4ab992]
indexer[0x41bde0]
/lib64/libc.so.6(__libc_start_main+0xe5)[0x7f418e78dd85]
indexer[0x41d9f9]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (http://sphinxsearch.com/bugs)) and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(/usr/share/doc/sphinx/sphinx.txt or http://sphinxsearch.com/docs/current.html#reporting-bugs))
--- BT to source lines (depth 9): ---
addr2line: 'indexer': No such file
--- BT to source lines finished ---So if we run this command (/usr/local/bin/indexer --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate) its help to fixed this indexing issue right?
-
repo owner
This is more a sphinx issue, than a piler issue. As I suggested before go to the sphinx forum, and ask for help (just as the above message suggests). And/or you may try a sphinx upgrade, they have 3.5.1 released this February. And yet another option is to switch to Manticore, though it requires a full reindex.
-
repo owner
- changed status to closed
- Log in to comment
Did you try simply copying the sphinx database files from old node to new node? If you have the same sphinx versions on both sites, then no need to reindex. Though I’m not sure how you migrated, eg. copying the raw files or export-import.