Can only search today's mail

[Jack Zielke]

After upgrading to the latest version I can only search through recent email. I expected I did something wrong during the update so I disabled the indexer cron jobs, reindexed all email (took almost 18 hours) and then ran the two cron jobs. That worked, I could see mail from 2008, etc. I let everything run for a few days and now it is just like it was right after the upgrade. I think I can view dailydelta1 and that's it.

Looking at a backup it looks like main1 used to be almost 8 Gig (.spd + spp). Currently it is 40M. The only change I have made to /usr/local/etc/sphinx.conf was to raise the mem_limit of indexer to 128M.

Reindex said it added 1.38 million emails, so this is pretty similar to your demo site in size.

[Janos SUTO]

I bet that the default daily debian/ubuntu indexer job destroyed the index. Please show me /etc/default/sphinxsearch.

[Jack Zielke]

This is a piler VM download from mailpiler.org. I upgraded it to debian 7.2. It does not have a file with that name. I am not saying that something from the upgrade is not wiping the index daily, just that it is not in /etc/default/sphinxsearch. I did a "locate sphinx" to see if there were any system installed files and I don't see anything odd.

I see source code in the home directory, /var/piler/sphinx, /usr/local/etc/sphinx* and sphinx.php in /var/www

[Janos SUTO]

Let me download the ova myself, and check for any gotchas. I hate to disappoint, but I think you have to reindex again after finding the cause.

[Jack Zielke]

I appreciate that. I was counting on the reindex but since it will take so long between tests I was going to wait for your input before I start changing things =]

[Jack Zielke]

Looking back through my notes, I got an sql error creating an index. Since I do get useful search data for the recent past I expect that everything in working as it should up to that point.

db-upgrade-0.1.23-vs-0.1.24.sql

create unique index `entry` on retention_rule (`domain`,`from`,`to`,`subject`,`_size`,`size`,`attachment_type`,`_attachment_size`,`attachment_size`,`spam`);

gave: ERROR 1064 (42000) at line 2: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'create unique index entry on retention_rule (domain,from,to,subject,`_'

I changed that to

alter table retention_rule add unique index `entry` (`domain`,`from`,`to`,`subject`,`_size`,`size`,`attachment_type`,`_attachment_size`,`attachment_size`,`spam`);

I think that worked...

[Janos SUTO]

can we chat on skype? My id is janos.suto

[Janos SUTO]

Btw. do you still have more than 1 day data in the archive or now you have emails 2-3 days back?

[Jack Zielke]

It looks like I have October 24 through today. Maybe I should just reindex again?

[Jack Zielke]

I am going to stop the cron jobs, import 100 000, run indexer.delta, then indexer.main, then import the next 100 000 and see how that goes.

[Janos SUTO]

OK, I still can't explain why it happened. I assume you had the virtualbox image with piler 0.1.23, then you have upgraded. If I recall it correctly th vbox image contained a sphinxsearch installed from source.

I can't say anything better than to reindex it again, sorry. Let me know how it goes.

[Jack Zielke]

It is taking around 1.5 hours to pull in 100 000 messages. So far each import seems to be working. I bet I messed something up when adding the dailydelta1 index.

[Jack Zielke]

I finished the import. If everything is working fine on Monday I will close ticket, if that is ok.

[Janos SUTO]

Yes, that's absolutely OK, thanks.

[Jack Zielke]

Seems perfect. I am not sure what I did to erase the index, but everything is humming along just fine.

[Jack Zielke]

resolved.

[Janos SUTO]

Thanks Jack, I'm glad that it's back to normal again.