Oops, indexer crashed!

[Christian Vielhauer]

Hi, i am testing aound with the vbox image:

I import all 275912 Mails with pilerimport -d /mnt/oldinbox

At first everything looks good ... but after i try adding a new mail it was shown on health page as added but i can't search for it although i wait a whole night ...

after that night i commented out the crontab jobs of piler user (and checked there is no /etc/cron.d/sphinx cronfile)

run as user piler in /tmp: reindex -a checked for that special mail in spy_index db-table (YES it's there)

run as user piler in /tmp: /usr/local/bin/indexer --quiet delta1 --rotate

run as user piler in /tmp: /usr/local/bin/indexer --quiet --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate

... and get the following

Any ideas?

Thanks

*** Oops, indexer crashed! Please send the following report to developers. Sphinx 2.0.7-release (r3759) -------------- report begins here --------------- Current document: docid=0, hits=0 Current batch: minid=0, maxid=0 Hit pool start: docid=0, hit=0 -------------- backtrace begins here --------------- Program compiled with gcc 4.4.5 Host OS is Linux debian 2.6.32-5-686 #1 SMP Mon Feb 25 01:04:36 UTC 2013 i686 GNU/Linux Stack bottom = 0xffffffffbfc6fd54, thread stack size = 0x10000 begin of manual backtrace: Something wrong with thread stack, backtrace may be incorrect (fp=ffffffffbfc6cef8) Stack looks OK, attempting backtrace. 8099ef4 ffffffffb72cd871 811a555 815f2ee 80e124c 809c1d8 80a37f3 ffffffffb7274ca6 8099721 Something wrong in frame pointers, backtrace failed (fp=0) begin of system backtrace: begin of system symbols: /usr/local/bin/indexer(_Z12sphBacktraceib+0x27b)[0x8179ceb] /usr/local/bin/indexer(_Z7sigsegvi+0x104)[0x8099ef4] [0xb7719400] /lib/i686/cmov/libc.so.6(cfree+0x31)[0xb72cd871] /usr/local/bin/indexer(_ZN22DiskIndexQwordTraits_cD2Ev+0xb5)[0x811a555] /usr/local/bin/indexer(_ZN13CSphIndex_VLN10MergeWordsI16DiskIndexQword_cILb1ELb0ELb1EES2_EEbPS_P10ISphFilter+0x65e)[0x815f2ee] /usr/local/bin/indexer(_ZN13CSphIndex_VLN5MergeEP9CSphIndexR10CSphVectorI18CSphFilterSettings16CSphVectorPolicyIS3_EEb+0x278c)[0x80e124c] /usr/local/bin/indexer(_Z7DoMergeRK17CSphConfigSectionPKcS1_S3_R10CSphVectorI18CSphFilterSettings16CSphVectorPolicyIS5_EEbb+0x5a8)[0x809c1d8] /usr/local/bin/indexer(main+0xfb3)[0x80a37f3] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7274ca6] /usr/local/bin/indexer[0x8099721] -------------- backtrace ends here ---------------

[Janos SUTO]

Please try upgrading sphinx, currently it's 2.0.8. If it doesn't help don't be afraid to test it with the latest sphinx beta (provided that you are evaluating piler).

Today expect a new version of the virtual machine shipping the latest version of piler. I'll use 64-bit debian 7, and hopefully it will solve your problem.

I'll let you know when it's ready.

[Christian Vielhauer]

So please compile piler with bigmem kernel, your old version supports only 3GB Ram With this new Kernel and more Ram i am able to read more KWords than with 3GB ...

I will now look for a link to download and then compile sphinx 2.0.8 ...

[Janos SUTO]

on 64-bit you virtually have unlimited memory. However another hint: edit sphinx.conf, and increase the indexer memory. By default it's set to 64 MB, perhaps time increase it to 256MB or perhaps even more. Let me know how you are going.

[Christian Vielhauer]

Hi JSuto,

i compiled 2.0.8 -> not fix the problem i increase the memory to 512 in sphinx.conf -> reboot -> not fix the problem

now the output is any longer in 2.0.8

root@mailarchiv:~# su - piler $ /usr/local/bin/indexer --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate Sphinx 2.0.8-release (r3831) Copyright (c) 2001-2012, Andrew Aksyonoff Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'... merging index 'delta1' into index 'main1'... read 82.4 of 82.4 MB, 100.0% done *** Oops, indexer crashed! Please send the following report to developers. Sphinx 2.0.8-release (r3831) -------------- report begins here --------------- Current document: docid=0, hits=0 Current batch: minid=0, maxid=0 Hit pool start: docid=0, hit=0 -------------- backtrace begins here --------------- Program compiled with gcc 4.4.5 Host OS is Linux mailarchiv 2.6.32-5-686-bigmem #1 SMP Fri May 10 09:11:45 UTC 2013 i686 GNU/Linux Stack bottom = 0xffffffffbff927d4, thread stack size = 0x10000 begin of manual backtrace: Something wrong with thread stack, backtrace may be incorrect (fp=ffffffffbff8f998) Stack looks OK, attempting backtrace. 809a054 ffffffffb72d0871 811aa45 8127863 begin of system backtrace: begin of system symbols: /usr/local/bin/indexer(_Z12sphBacktraceib+0x96)[0x817a586] /usr/local/bin/indexer(_Z7sigsegvi+0x104)[0x809a054] [0xb771b400] /lib/i686/cmov/libc.so.6(cfree+0x31)[0xb72d0871] /usr/local/bin/indexer(_ZN22DiskIndexQwordTraits_cD2Ev+0xb5)[0x811aa45] /usr/local/bin/indexer(_ZN13CSphIndex_VLN10MergeWordsI16DiskIndexQword_cILb1ELb0ELb1EES2_EEbPS_P10ISphFilter+0x643)[0x8127863] /usr/local/bin/indexer(_ZN13CSphIndex_VLN5MergeEP9CSphIndexR10CSphVectorI18CSphFilterSettings16CSphVectorPolicyIS3_EEb+0x278c)[0x80fad7c] /usr/local/bin/indexer(_Z7DoMergeRK17CSphConfigSectionPKcS1_S3_R10CSphVectorI18CSphFilterSettings16CSphVectorPolicyIS5_EEbb+0x5a8)[0x809c338] /usr/local/bin/indexer(main+0xfb3)[0x80a3953] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7277ca6] /usr/local/bin/indexer[0x8099881] Backtrace looks OK. Now you have to do following steps: 1. Run the command over the crashed binary (for example, 'indexer'): nm -n indexer > indexer.sym 2. Attach the binary, generated .sym and the text of backtrace (see above) to the bug report. Also you can read the section about resolving backtraces in the documentation. -------------- backtrace ends here ---------------

i want to do so, but 'nm' command is not available and i cant find it in apt

[Christian Vielhauer]

I now try sphinx 2.1.1-beta

[Christian Vielhauer]

And the same problem again ... Always at same KWord ... eventually this is the last Kword, and the next step fails ..

I don't know :-(

[Christian Vielhauer]

Hey Janos,

i get it to work ... i moved the complete sphinx dir to sphinx.bak (evtl. you need some info about the files in it) then i go to piper src dir and run make install again to get a new sphinx dir and created the sphinx base files with 'indexer --all' as piler user

run again 'reindex -a' as piler user run again /usr/local/bin/indexer delta1 --rotate run again /usr/local/bin/indexer --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate

this time everything works well ..

all time before i run only the last indexer -- merge main1 ..... after recompile new sphinx version. eventually the main1 file was broken, why ever

if you want u can mark it as fixed or write me to make some tests with backuped sphinx files or sql dump

[Janos SUTO]

ok, then I set the status to resolved. Thanks for your efforts fixing it.