Mails are missing in the search console

[Kanthanathan S]

Hi

Thanks for a wonderful product.

I am trying to test an archival solution and found this extremely useful.

But I find the following issues in my archival setup. Can you please guide me through.

1. mails are getting archived to archive server and are visible in the search console for a particular user after some time. But every day morning I see only mails that have come only for that day (i.e., after Midnight 12.00) and all previous mails which I had seen previous day are missing in the search console. When I login to admin the mail counts are proper.

2. When a mail is sent from x to y, with cc to self, I get the mail in the console, but it is not getting listed with criteria such as "from" or "to". My assumption is that the search logic uses either "From" or "To" be self.

3. I mail received in my inbox because I am part of a distribution list, but it does not appear under my search. (I will revalidate this, because not sure if my postfix is actually forwarding my distribution list to archive server).

I installed the version 0.1.23 and then upgraded to master branch build 825. Installed on Ubuntu Linux 12.04, sphinx search 2.0.8.

Meanwhile, a few query though. Few days back when I logged into your demo site using admin user, I saw an option for LDAP configuration under administration pull down menu, which is missing both in my installation and the new demo site. Is this a feature disabled by purpose and can be enabled if necessary. I will find this feature more useful.

Thank you once again Kanthan

[Janos SUTO]

Hello Kanthanathan,

1. Did you try to specify the previous day, eg. 2013.08.01? Please note the sphinx query the gui issues (see maillog), then try to connect to searchd:

mysql -h 127.0.0.1 -P 9306

then execute the same query, and see what happens.

2. You are right. For regular users an email filter is appended to each search query to prevent them seeing others' emails. By default the gui thinks that your interested in the emails you received. So I recommend to put your address to either "from" or "to".

3. Also check the emails you can see in the settings menu after logging. You should see your distribution lists, too. If not then the gui is not aware of that you belong to that group, too.

The LDAP + customer settings are designed for service providers with several customers, and probably several mail servers (eg. Exchange, etc). The mentioned features are hidden unless you set $config['ENABLE_SAAS'] = 1; in config-site.php

If you use piler for a single company (perhaps with a single AD realm), then it's not necessary to enable this feature (although it won't hurt if you do so).

[Kanthanathan S]

Hello Janos

My setup is simple. I have configured my postfix to bcc all mails from/to my email address to archive@mailarchiverhost.

This has been happening for the last 5 days. Everyday when I log to my console, I am seeing mails only for that day. My admin portal shows received about 401 mails of which 58 are duplicate and total processed is 363 Mails. However in my auditor login or in my login I am seeing only 54 Mails which belongs to today (2013.8.3). I am not able to see any earlier mails which are archived even if I explicitly search for by giving date range.

I login to mysql as per your suggestion and ran the following command, it shows 363 rows.

mysql -h localhost -u piler -p piler -P 9306

select * from v_messages;

This also shows only 363 mails only. (some 750+ rows selected)

Any other settings do I need to check. Where do i see the search command the gui issues. (What is the maillog and how do i see that)

Thanks Kanthan

[Janos SUTO]

Hello Kanthan,

please execute the following command precisely (note that localhost and 127.0.0.1 are not the same for the mysql client):

mysql -h 127.0.0.1 -P 9306

Then please make a search for yesterdays emails, and copy the sphinx query to this mysql prompt, and execute it. Then please show me the results.

FYI: maillog is where your syslog daemon writes any mail related stuff, eg. /var/log/maillog or /var/log/mail.log, etc.

[Kanthanathan S]

Hello Janos

the query returns an empty set.

The query is listed below. The query is to search mail from 2013/8/1 to 2013/8/2

SELECT id FROM dailydelta1,main1 WHERE sent >= 1375295400 AND sent <= 1375468199 AND MATCH('(@from kanthanXdomainXext | @to kanthanXdomainXext)') ORDER BY sent DESC LIMIT 0,1000 OPTION max_matches=1000;

Any input on how to proceed further

[Janos SUTO]

OK, then try the following only, and show me what you get:

SELECT id FROM dailydelta1,main1 WHERE sent >= 1375295400 AND sent <= 1375468199 ORDER BY sent DESC LIMIT 0,1000 OPTION max_matches=1000;

[Kanthanathan S]

It also returns an empty set

[Kanthanathan S]

I had initially configured DB_HOSTNAME in config-site.php as localhost. But yesterday after your mail I had configured the same as 127.0.0.1. Do you think will that have any impact on the same? However even after reconfiguration, today also I am seeing mails that had come after 2013/8/5 12.58AM only (This is first mail in my inbox after 12 Midnight).

[Janos SUTO]

I believe that something executes the indexer to index the main1 set. If so then it practically resets it. So I'd like you to search for sphinx config files, and make sure only the piler shipped version runs. Please check for /etc/sphinx.conf /etc/sphinxsearch/sphinx.conf, etc. and even the crontabs.

For DB_HOSTNAME "localhost" is fine.

[Kanthanathan S]

I found a surprise element. My root crontab is empty. However /var/piler/sphinx has got main2, main3 and main4 with ownership as root and modified time as 00.00. This made me suspicious and tried investigating further. I found sphinxsearch to run as user root in /etc/cron.d. I had removed the file. I removed the files main2, main3 and main4 in the /var/piler/sphinx. Stopped the search service and ran the indexer command as piler user once ( indexer --all). I have left it as it is and will see tomorrow morning. I think this should solve my problem mostly. I will update you tomorrow .

Meanwhile, thank you for all your inputs.

Kanthan

[Kanthanathan S]

Hello Janos

The problem is resolved. Thanks for your inputs.

Regards Kanthan

[Janos SUTO]

OK, here's a tip to get the older emails back:

run the reindex utility in the /tmp directory. If you think that your first 1000 emails are disappeared, then execute: reindex -f 1 -t 1000.

This command will retrieve these emails from the archive, and puts them to the sph_index table for reindexing. After running the command (and waiting for ~30 mins) you should see your first 1000 emails again.

Btw. could you explain what you found in /etc/cron.d? I'd add it to the FAQ...

[Kanthanathan S]

I ran the above command from /tmp as user piler. I am getting Segmentation Fault. Should I stop the search service and then run the command?

Meanwhile, I found the following line was in /etc/cron.d/sphinxsearch with the comment that it rebuilds all the indexes daily.

@daily root . /etc/default/sphinxsearch && if [ "$START" = "yes" ] && [ -x /usr/bin/indexer ]; then /usr/bin/indexer --quiet --rotate --all; fi

I had commented the above line.

I had installed precompiled sphinxsearch 2.0.8 from their website (dpkg format). I assume that had installed this cron entry.

[Janos SUTO]

Thanks for the info. What if you just ran: reindex -f 1 -t 10?

[Kanthanathan S]

still segmentation fault

[Mark Clarke]

I can confirm the segmentation fault error. I am using Ubuntu 12.04.3 The log files says:

Sep 25 15:50:34 srv01 kernel: [ 6731.994791] reindex[4744]: segfault at 7fff00000000 ip 00007f5cddcc25be sp 00007fff5e7977f0 error 4 in libpiler.so.0.1.1[7f5cddcab000+22000]

[Mark Clarke]

Here is some additional information. Not sure if it will help:

piler 0.1.24-master-branch, build 836, Janos SUTO [e-mail]

Build Date: Tue Sep 17 13:32:14 SAST 2013 ldd version: ldd (Ubuntu EGLIBC 2.15-0ubuntu10.4) 2.15 gcc version: gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) Configure command: ./configure --localstatedir=/var --with-database=mysql --enable-starttls --enable-tcpwrappers