The piler process consumes 100% cpu while processing certain emails

[Andrey Khlebnikov]

As per previous issue I've just uploaded 70k+ of emails from MS Exchange into mailpiler archive. I'm using preinstalled VM (2012.09.18) from mailpiler.org site.

The uploading process is made by fetchmail, which takes emails via POP3 from MS Exchange archiving mailbox and forwards them to mailpiler's smtp port via 127.0.0.1

Almost all messages were uploaded normally, but while processing some of them piler process starts to consume 100% of one cpu core. As per default config there are ten piler processes. As one of these processes hangs it seems piler starts to receive incoming emails by another process. When all ten processes were hung the fetchmail said that there is no smtp server on the destination address.

When uploading process stopped by error I've checked /var/piler/tmp and found ten similar emails (an their extracted parts). All of these ten emails are from one address (one of our internal automated systems). All have similar subjects (in English, without cyrillic symbols), all have similar zip files attached (one zip attach per email). All zip files contain similar csv files (one csv per each zip). Csv sizes are varied from 180 kbytes to 2 Mbytes.

I suppose piler hangs because of these csv's.

[Janos SUTO]

Please download the latest master branch (https://bitbucket.org/jsuto/piler/get/master.tar.gz), and let's try it again.

Btw. the pilerimport utility can do the import on its own (this feature is not present in version 0.1.21 yet) via POP3. However, it's up to you which way you use.

If you can identify this problematic email, and you can have it on the piler box in EML format, then run "pilertest /path/to/the/message" to verify that piler can actually parse it. (use the latest master branch version). If not, then it's a parser error that should be fixed. If so, I'd be glad if you could allow me to take a look at that email to find the spot that makes piler hang.

So for starters, let's see whether the latest version can handle this email or not.

[Andrey Khlebnikov]

I've just tested one of the emails with pilertest and it also hung. Where can I send the file? It is 1.3M in size.

[Janos SUTO]

Please gzip it, then send it to my address. You can see it in the output of the "piler -V" command.

[Janos SUTO]

Thanks for the email. It's pretty weird when I try to unzip the contents of it, libzip says, it's length is -1. So the zip_fread function just can't finish, that's why it hangs.

Please apply the following workaround that checks for the size, but currently the contents of the 3 csv files are invisible for piler to hand them to index.

Perhaps I'll look for another zip library to use.

[Janos SUTO]

if it still hangs after the patch, then reopen the issue.