Mail import display issue and MariaDB stuck

[vinay garg]

Hii gentleman, i have been installed piler-1.1.0 with MariaDb-5.5.40. yesterday i use pilerimport for archive mail. approx i import 30,000 mails. I have set SEARCH_MAX_LIMIT upto 50,000 in config-site.php and sphinx.conf. Before i have already imported 1,00,000 mail. Yesterday i tried to import 30,000 mailfor single user. When i try to login with username and password and searched mail then i can see only 6,332. i cant understand when all other mails get stuck.

One more issue i am facing, during pilerimport MariaDB get stuck, till i dont kill process id of mysql. When MariaDB stucked, pilerimport hang up. i tried to search what happen in both case.

For that i use top command, i find pilerimport uses 100% cpu. please check in screenshot. for that i have added /5 * * * LC_ALL=C mpstat | tail -1 | awk '{print $11}' > /var/piler/stat/cpu.stat in crontab.

Please help me for find the exect problem with pilerimport, mail and MariaDB.

[Janos SUTO]

It seems that pdftotext is running as well, so I think pilerimport found a broken pdf file, that pdftotext can't handle properly. Anyway you may recompiler piler with a timeout value, check out ./configure --help for the exact option. Or upgrade to the master branch, and this timeout value is set automatically.

Also note that having 50k hits from sphinx looks pretty unusual, and requires lots of resources. The mysqld issue may be related to too little resources, eg. memory, since sphinx may need more memory to serve 50k hits at once.

[vinay garg]

if i recompile piler, then chance of loss indexed data may more. have there any other option for display all imported mails??

Can you tell me exect reason why am i not able to find all imported(30,000) mails.

[Janos SUTO]

I'm not sure how (merely) recompiling piler increases the chance for a data loss. Unfortunately I can't tell you why you can't find all your 30k emails. First, check the metadata table: run "select max(id) from metadata;" to find out how many emails are archived already. Then login as auditor, and hit the search button, and watch the middle bar. You should see how many emails sphinx found inside ( and ).

[vinay garg]

Thanks for your reply, MariaDB [piler]> select max(id) from metadata; +---------+ | max(id) | +---------+ | 1090392 | +---------+

As i have checked in auditor it describe 1090392 mails search sphinx search.

if i want to check for one particular user's mail which was i import 30,000 mail, i can search only 6500 mail insted of 30,000 mails. during import i checked, about 15000 mail was duplicate id like (duplicate: 12464-imap-15467.txt (duplicate id: 172744).

i think piler are notable to show duplicate mail on gui. But why this happen???

I checked pdfoftext process is not running.

[Janos SUTO]

Try to find an email which belongs to this particular user, and check if it has a From/To/Cc matching this user.

Note that piler should store the give email in 1 copy only. If you can find the same message twice or more, then somethings is wrong with your emails. Check if they have a valid message-id header field.

[vinay garg]

All these solution i have checked but unsuccessful. have there other solution??

[eXtremeSHOK]

Re initialize the sphinx index and then run the Sphinx delta update script.

Also set your config to allow 100 000 results in the gui.

Clear the caches and try again

[vinay garg]

i have reinitilize using reindex -a. but i dont to lost my data. so can you update which script have to run for delta update??? i have little confussion for update script. please share the script name and path.

i have set config to allow 100000 still in gui. i have clear the cache.

[Janos SUTO]

I don't recommend to ask 100k hits from sphinx, since it requires pretty much resources (memory + cpu). If you want to reindex all, then follow the procedure described in the FAQ.

[vinay garg]

i think i have find the problem.. i will update you soonly..

[vinay garg]

i have reinstall piler and sphinx again but problem is still available.

i have import total 192000 mail. i have seen in admin health monitor.

if i want to see all these mail on auditor, i can search only 24000.

please suggest.

[eXtremeSHOK]

Make sure the delta index script is running and all the permissions are correct.

Would you like to test on a fresh vm of a hosted piler?

[vinay garg]

Can you mention all permission?? Because As well i am using correct permission, once i want to confirm all permission.

yes delta index script is running.

yes i have installed fresh. but still problem facing.

[Janos SUTO]

The problem is (as I described it before) that piler somehow is not aware of that ~20k messages belong to the same user who has 30k emails, but can see only ~6000. That's why I asked for an email which this (non auditor) user should see, but he can't. Also make sure when he logs in, then he can see all his emails addresses in the settings menu. I can't help you until you do the homework, and find me 1-2 messages I asked for.

[vinay garg]

Pls suggest a tested solution for below mentioned problem:--- We are having two domains for e.g. [e-mail] & [e-mail] we want to fetch both emails in single piler. that we had already done . now problem is coming that we are unable to search the emails of both domains in same instance in normal search result.

[Janos SUTO]

As I said before: check the settings for this particular user, and verify if all of his email addresses are present. Btw. what mail server do you have? Zimbra open source?

[vinay garg]

We are using postfix for temporary bases; As i asked you we have two virtual domain, if i logging with [e-mail], i search only @adc.co.in mails. i cant find @abc.com mails. can you suggest upon it?? how can i see other domain mail. i am using IMAP interfacing.

[Janos SUTO]

OK, then we are a bit closer. Do you have the latest master branch installed for the gui? If not, then do so, and I'll help you to fix it with a custom fix for you.

[Janos SUTO]

OK, then upgrade to the master branch.

[vinay garg]

can you share the procedure of upgradation???

[Janos SUTO]

The upgrade procedure is on the piler site, be sure to follow it. Btw. I didn't tell you to change any settings.

[vinay garg]

i have been upgrade piler-1.1.0 to master-branch jsuto-piler-daf6exxxxx.

Now please mention the precedure of search mail for two virtual domain at same instance.

[Janos SUTO]

You need a custom function to config-site.php to fix the email address list:

$config['CUSTOM_EMAIL_QUERY_FUNCTION'] = 'akakaak';

function akakaak($username = '') {
   $session = Registry::get('session');
   $data = $session->get("auth_data");

   $a = array();

   foreach($data['emails'] as $email) {
      $s = explode("@", $email);
      array_push($a, $s[0] . "@otherdomain.com");
   }

   $data['emails'] = array_merge($data['emails'] , $a);

   $session->set("auth_data", $data);
}

[vinay garg]

Thanks jsuto.. final its work.. but i have one quey.. if i want to add for multiple(two or more than two) virtual domain, will it work???

[Janos SUTO]

Yes, it will work. use the array_push() for your other domains as well.

[vinay garg]

Again issue occured, if i have one group [e-mail], user [e-mail] and [e-mail] are member of [e-mail]. If i am login with [e-mail] then mails are not visible for group ([e-mail]) on regular user search. So group mail are not available for regular user. But auditor can search group mail as well as user mail.

[Janos SUTO]

Read the docs, and use the group feature for that.

[vinay garg]

which feature you defined is not working.. because im using LDAP for [e-mail] and MYSQL for [e-mail].

[Janos SUTO]

4 days ago you said you use imap authentication. Now you use ldap. Fine, then make sure the ldap query returns all email addresses this user needs access to. I'm done with this issue, I can't see any "critical bug" in the software related to your problem. If you are still stuck, try with the piler mailing list, perhaps the others may help you.

[vinay garg]

Thanks