Imported messages but no search results

[Ilias Chasapakis]

Hi,

here I am bugging you again :-)

I'm the guy with piler 1.0.24 (ova deployment for vmware).

Well, I managed to make piler import messages from a Google Apps account.

Now dashboard shows 23000 messages have been processed.

I know I might be missing something elementary, but I get no search results in any way...

I created an auditor account (auditor@local) to test but I get nothing.

The google_imap table is of course correctly populated.

I'd appreciate a lot some help with this...

What relevant information should I provide you with so that you may come up with some idea?

Thanks in advance!

[Janos SUTO]

Hello, so I assume that you have tons of emails in the metadata table. Please check the count of records in the sph_index table (select count(*) from sph_index). Also check if you have data under /var/piler/sphinx.

[Ilias Chasapakis]

Hi, the query returns 23.601 records (as many as the emails are).

#!

mysql> select count(*) from sph_index;
+----------+
| count(*) |
+----------+
|    23601 |
+----------+

I have 164 files (168KB total size) data under /var/piler/sphinx

[Janos SUTO]

164 files? It seems a bit too many to me. Anyway I think all you need is only to process the parsed data, so make sure user piler has some indexer jobs, eg. indexer.delta.sh and indexer.main.sh.

To process these data manually, then su to user piler, then as user piler execure the following:

/usr/local/libexec/piler/indexer.delta.sh

This should empty the sph_index table, and increase the sizes of the files under /var/piler/sphinx, and finally you should see some emails in the gui.

[Ilias Chasapakis]

Hi, the indexer returns a mysql connect error.

#!

Access denied for user 'piler'@'localhost' (using password: YES) (DSN=mysql://piler:***@localhost:3306/piler

In fact I've changed the default password for user piler and I already changed that for import functions ecc.

Could you tell me where is the config to change the password for the indexer?

[Janos SUTO]

edit piler.conf, and located "mysqlpwd" parameter

[Ilias Chasapakis]

Password under mysqlpwd was already correct, I had put it in there. I thought there might be another conf for the indexer. It also otputs a

WARNING: IsEmpty != attribute size (56)

after the mysql connect error... Hmmm...

[Janos SUTO]

Have you changed the "source base" block in sphinx.conf? I mean the sql_attr_uint lines.

[Ilias Chasapakis]

Stupid me, I had a swap file in there, thought I'd saved it before... It's fine now (the warning "WARNING: IsEmpty != attribute size (56)" comes up again but everything seems to work)

I can see search results in the gui now.

Now that everything seems in place, I'll start placing the cron jobs.

Thanks for your help!

[Janos SUTO]

OK. And thanks for the donation :-)

[Ilias Chasapakis]

You're welcome. I'm sorry it's not much as the project deserves. If I should get any income out of email archiving, be sure I'll make further (and better) donations...

[Janos SUTO]

The amount you donated is absolutely ok, and I appreciate it. I mean it.

[Ilias Chasapakis]

Thanks :-)

I saw there are only 1000 messages out of 23000 in the archive gui, is this a settings issue? Is there a configurable limit?

[Janos SUTO]

Yes, it's a settings issue. By default searchd returns max. 1000 hits. You may increase this value, but sphinx recommends to keep it, since increasing it requires more resources. Their reasoning is that nobody pages further than the first 2-3-4 pages. Although this could be somewhat different with email archiving, but if you get 1000 hits, then it means 1000+, and you should fix the search terms to get more relevant data.

[Ilias Chasapakis]

Ok! I think it's a reasonable limit... Thanks. I'll go better through "components" documentation to avoid asking abount non strictly piler related issues...