Email Missing

[Rizki Rivai]

I already tried to - reindex piler. - Reimport mail said duplicate. - Pilerget return nothing - using mysql 'select * from metadata where piler_id='<id>' return empty set. - in /var/log/mail.log i have this message: Aug 21 09:49:43 piler pilerget[17422]: /var/piler/store/00/87/c7/d0/4000000053f55e4717b6945c00d95387c7d0.m: cannot open() - under /var/piler/store/00 only has 4 directory which are 53c 53d 53e 53f.

Any hint to fix this?

[Janos SUTO]

The empty select result indicates that piler actually did not archive such a mail. If you have such an entry in maillog, what's that? If you tried to reimport a message, then piler would say it's a duplicate, and not store it, since it's already stored. Note that reindex is a troubleshooting tool: use it only if the sphinx indices are corrupted or lost.

[Rizki Rivai]

I found out many imported message went to non-existent directory. As I posted earlier, a message was store on directory /var/piler/store/00/87... while in fact store directory only have /var/piler/store/00/53c, /var/piler/store/00/53d,/var/piler/store/00/53e,/var/piler/store/00/53f.

It seem this problem started when i import more than 5000 eml mail from a directory.

[Janos SUTO]

No, nothing went to non existing directories. Earlier versions used the 00/87/... scheme, however newer versions use 00/53f/... structure. However (to support upgraded systems) if pilerget can't find the message under store/00/53f/..., then it tries to look up in the older path (for compatibility). Btw. what version do you use?

[Rizki Rivai]

piler 0.1.24

[Janos SUTO]

Well, then the next step would be to recall any pilerimport output or maillog about importing those 5000 emails. I suspect that something has gone wrong.

Btw. when we solve this issue, it's worth to upgrade to 1.1.0. It has lots of improvements and bugfixes. Or if the archive is relatively small, it may be feasible to drop everything, then reinstall from scratch.

[Rizki Rivai]

from maillog i get a lot of "cannot open frame file: ..." other than that i havent found any suspicious error message.

[Janos SUTO]

Well, this alone should be suspicious enough. Usually a sign that the parser can't create temp files. You should either give write permissions to piler user on that directory, or go to /tmp, then run pilerimport there - as it's written in the docs.

[Rizki Rivai]

you are correct on permission. As my script is creating new directory to extract mail from server.

[Rizki Rivai]

permission problem

[Ochieng Maxwell]

I seem to be missing some mails and when I look at the logs the missing emails shows "cannot open frame file" with a message-id=null. these are mails coming through piler MTA. error seems to appear for some mail and not all. Permissions seems fine since most mails are being archived. Kindly advice.