mmap cannot allocate memory

[Crzaccount]

Hi Janos,

I have 32GB installed machine. I am splitting the main indexes for certain count of mails. When i restart the searchd service am getting the error message for main8003

###Start WARNING: rotating index 'main8003': .new preload failed: mmap() failed: Cannot allocate memory (length=17053165) WARNING: rotating index 'main8003': .new preload failed; ROLLBACK FAILED; INDEX UNUSABLE WARNING: index 'main8003': preload: mmap() failed: Cannot allocate memory (length=16569819); NOT SERVING ###End

Once this error comes, it is am not able to rotate/merge the indexes. In sphinx.conf i have mention the entry like

###Start index main8004 { source = main8003 path = /var/piler/sphinx/main8003 docinfo = extern charset_type = utf-8 enable_star = 1 min_prefix_len = 6 min_word_len = 1 }

source main8003: base { sql_query_pre = SET NAMES utf8 sql_query = SELECT id, from, to, fromdomain, todomain, subject, arrived, sent, body, size, direction, folder, attachments, attachment_types FROM sph_index WHERE id=-1; } ###End

Pls find the piler -v output as follows

###Start piler 0.1.25-master-branch, build 865, Janos SUTO [e-mail]

Build Date: Mon Mar 10 11:05:24 IST 2014 ldd version: ldd (GNU libc) 2.5 gcc version: gcc version 4.1.2 20080704 (Red Hat 4.1.2-54) Configure command: ./configure --localstatedir=/var --with-database=mysql ###End

It seems there is some problem in memory handling. Please advise, how to fix this issue.

[Janos SUTO]

How many main indices do you have? How large are they? Show me ls -la /var/piler/sphinx output. Don't forget to use the formatting options of bitbucket!

[Crzaccount]

We r using approx 105 indexes. Once the metadata count reaches certain limit index will be automatically changed(merge) to another index. As of now around 25 indexes were used and the error is coming while restarting the searched. However specific date mails are not displaying in webui. pls find the op of ls -la /var/piler/sphinx in the attachment output.txt

[Janos SUTO]

Firstly, stop searchd, make it shutdown. Then fix the permissions: all files in the sphinx dir should be owned by piler. Then try starting searchd again. Based on the output ~3-4 GB is the memory requirement for sphinx. Let me know how it goes.

[Crzaccount]

We did as u said above(changed the ownership of files in /var/piler/sphinx/ as piler ) and while starting searchd again, we got the below warning & still mails are not showing properly in Webui.

WARNING: index 'main8003': preload: mmap() failed: Cannot allocate memory (length=9110523); NOT SERVING

[Janos SUTO]

Edit sphinx.conf, and comment the definition of index main8003 out, then restart searchd.

[Crzaccount]

If we are Commenting the definition of main8003 in sphinx.conf, the mails which is in the indexes other than main8003 are showing properly in Webui. Eventhough we did the reindex for the main8003 the mails in that particular index is not showing in the Webui.

[Janos SUTO]

Have you run the delta indexer as well?

[Crzaccount]

Yes. We ran already. Pls let me know, what is to be done further.

[Janos SUTO]

Then those emails should appear in the currently used main index (or if it's too early, then in the dailydelta index). Can you verify this?

[Crzaccount]

Those mails are not displaying. We did kernel fine tune to overcome this warning ( preload: mmap() failed: Cannot allocate memory). Now i'm doing the reindex of mails (around 1 lakh mails. Processing by 1000) & store it in main8005. When we run the indexer command to merge the index, it is running successfully but the mails not showing in webui.

I have noticed that after indexer runs the main8005 temporary index files are there. Pls find the details below

###Start -rw-r--r-- 1 piler piler 191K Sep 18 14:06 /var/piler/sphinx/main8005.new.spa -rw-r--r-- 1 piler piler 18M Sep 18 14:06 /var/piler/sphinx/main8005.new.spd -rw-r--r-- 1 piler piler 81K Sep 18 14:06 /var/piler/sphinx/main8005.new.spe -rw-r--r-- 1 piler piler 609 Sep 18 14:06 /var/piler/sphinx/main8005.new.sph -rw-r--r-- 1 piler piler 4.3M Sep 18 14:06 /var/piler/sphinx/main8005.new.spi -rw-r--r-- 1 piler piler 0 Sep 18 14:06 /var/piler/sphinx/main8005.new.spk -rw-r--r-- 1 piler piler 0 Sep 18 14:06 /var/piler/sphinx/main8005.new.spm -rw-r--r-- 1 piler piler 29M Sep 18 14:06 /var/piler/sphinx/main8005.new.spp -rw-r--r-- 1 piler piler 1 Sep 18 14:06 /var/piler/sphinx/main8005.new.sps -rw-r--r-- 1 piler piler 159K Sep 18 13:39 /var/piler/sphinx/main8005.spa -rw-r--r-- 1 piler piler 15M Sep 18 13:39 /var/piler/sphinx/main8005.spd -rw-r--r-- 1 piler piler 66K Sep 18 13:39 /var/piler/sphinx/main8005.spe -rw-r--r-- 1 piler piler 609 Sep 18 13:39 /var/piler/sphinx/main8005.sph -rw-r--r-- 1 piler piler 3.7M Sep 18 13:39 /var/piler/sphinx/main8005.spi -rw-r--r-- 1 piler piler 0 Sep 18 13:39 /var/piler/sphinx/main8005.spk -rw------- 1 piler piler 0 Sep 18 14:06 /var/piler/sphinx/main8005.spl -rw-r--r-- 1 piler piler 0 Sep 18 13:39 /var/piler/sphinx/main8005.spm -rw-r--r-- 1 piler piler 25M Sep 18 13:39 /var/piler/sphinx/main8005.spp -rw-r--r-- 1 piler piler 1 Sep 18 13:39 /var/piler/sphinx/main8005.sps ###End

   I hope, the temporary files (new) should get clear automatically once the indexer completes. Pls suggest what is to be done further.

[Crzaccount]

I have also noticed that when indexer runs mails are processing from sph_index & it is not storing in main8005. Also there was no error when indexer runs

[Janos SUTO]

Yes, the indexer script puts data from sph_index table to the main index defined in the script itself. If there were no errors, then perhaps you are looking at the wrong index . The copy file list is unreadable, please use the formatting options of bitbucket, don't expect me to torture my eyes.

[Crzaccount]

It seems there is some problem in memory handling while processing the huge data in 32 bit OS. So we are planning to upgrade the OS to 64 bit with piler data. We will update you once done. Thanks for your support.

[Janos SUTO]

OK, reopen if the problem persists.