Cant import mails from /var/piler/error/

[Martin Nadvornik]

Hi,

As I didnt't get a reply in the mailing list I hope you can help me here. I have recently experienced a storage error where a disk mounted to /var/piler/store was located. This resulted in the filesystem getting read only. In the Piler log I found out that all messages during that time where written to /var/piler/error/. Beside /var/piler/store every directory including the mysql Database was still accessible. When I tried to reimport all 253.717 mails from /var/piler/error/ every single mail was shown as duplicate.

pilerimport -D -d /var/piler/error
...
duplicate: /var/piler/error/6OB9307T38G7AM5P (duplicate id: 5250570)
duplicate: /var/piler/error/FQHYZKBUD7AMHCTO (duplicate id: 5250571)
duplicate: /var/piler/error/A0EIUGIZOK83M7X5 (duplicate id: 5250572)
duplicate: /var/piler/error/6GJPHJPA4FBIJLZV (duplicate id: 5240120)
duplicate: /var/piler/error/KU01VYYWX9PDF4X0 (duplicate id: 5250573)
duplicate: /var/piler/error/BNYTLRCPZ38SNQ6A (duplicate id: 5250574)
duplicate: /var/piler/error/Y5FAMZ68W51076C2 (duplicate id: 5250575)
duplicate: /var/piler/error/JNH48F73K7G2D9RT (duplicate id: 5250576)
duplicate: /var/piler/error/JTBUP8NDSF6WHZDZ (duplicate id: 5250577)
duplicate: /var/piler/error/ABJ97AO6VLSRB2LW (duplicate id: 5250578)
duplicate: /var/piler/error/DYHHTQNY4YD2TYY0 (duplicate id: 5250579)
duplicate: /var/piler/error/O0IC5S6KRMIJ0QV9 (duplicate id: 5250580)
duplicate: /var/piler/error/0OV1YYMQTX92M8M8 (duplicate id: 5250581)
duplicate: /var/piler/error/CWG9CWJ96IYS3MNC (duplicate id: 5250582)
duplicate: /var/piler/error/8D9F4SG4WLN949HJ (duplicate id: 5250583)
duplicate: /var/piler/error/6L2TUYVNSWDOE1IU (duplicate id: 5250584)
duplicate: /var/piler/error/LVY3WWDCXSVJXRDK (duplicate id: 5250585)
...

It also seems like those messages have been written somewhere because I can download them via their id using https://archive/index.php?route=message/download&id=XXXXXXX However they are not shown in the web gui even as auditor. I also tried to reindex everything but this resultet in an segmentation fault. But I am not sure if this is because of a bug or something got currupted.

reindex -a
[1]    10189 segmentation fault  reindex -a
reindex -a  10183,19s user 1763,62s system 72% cpu 4:33:49,45 total

Is there anything I can check or do you think that I need to migrate all mails to a fresh piler installation?

[Janos SUTO]

Without proper access to the store directory piler moved everything to the error directory. It's possible that the metadata table accepted the new entry, which would explain why you see all emails in the error directory as duplicates (though they are not).

Recent piler builds feature a rollback() function (in src/message.c) which should remove all related sql entries in case of a failure. Perhaps your piler build doesn't have this feature.

The fix might be the following:

• stop piler daemon
• observe the first and last duplicate id: numbers, eg. 5250570 and xxxxxx. Then remove all entries from metadata, rcpt and sph_index tables between those id numbers
• try importing the error emails again, this time you should NOT see them as duplicates
• start piler daemon again
• you may need to reindex everything from 5250570

Let me know how it goes.

[Janos SUTO]

No news is good news.

[Martin Nadvornik]

Hi Janos,

sorry for the delay. We have tried your suggestion and they were indeed not seen as duplicate. However after a reindex there were still mails that have not been correctly imported. We wrote a script that looked at all mails that have been in the /var/piler/error and checked if they can be found via the webinterface. There were many mails that have seemingly been imported by pilerimport but can't be found. Since we also have experienced another isssue in the last couple of days (described in #944) we would consider reinstalling piler completely, create a new database and reimport all mails unless you think this is easily solvable.

Thanks.

[Janos SUTO]

An email gets to the error directory if it can't be processed for some reason. It also means that it won't be seen in the gui. Btw. if the piler daemon can't process it for some reason, then I don't think pilerimport could make it either as they both use the same underlying functions. However, the cause for that should be obvious from the logs. Based on my past experience with other users it's often the case that the email contains invalid utf-8 sequences, thus mysql won't accept it. But in your case it might be different.

I suggest you to deploy the latest master branch on a test host, and let's see how it goes. If there's a problem, then please open a new issue on bitbucket.