/var/piler/tmp folder is filled with not imported mails

[Dennis]

Hello Janos,

since 3 days and counting we have a problem with mails, which will not be imported. These mails stack up in the /var/piler/tmp folder and now the local storage is nearly full (95%). How can we handle these files?

Kind Regards Dennis

[Janos SUTO]

Hello Dennis,

it' possible that you have such an email that the parser couldn't process properly. To prove this theory right (or wrong), stop the piler daemon, then run md5sum on these files, and you'll probably see lots of files with the same checksum. If the above theory is right, then you should move the problematic email from tmp to somewhere. Then I suggest you to rename the tmp folder to tmp.orig, then mkdir tmp; chown piler:piler tmp, and start the piler daemon again. Then you should investigate why piler has a hard time processing such email(s). Btw. what version do you have? Recent piler builds should move an email like that automatically to /var/piler/error for later revision.

[TOmi]

I have the same problem , folder /var/piler/tmp is very big, I checked md5sym for two files from directory :

[root@vPiler x]# md5sum 400000005c51aa611661d9cc0063c8a5fe40.a1.bin
**a19b78c94fcfcb863fbf43b9ce11d85a**  400000005c51aa611661d9cc0063c8a5fe40.a1.bin
[root@vPiler x]# md5sum 400000005c51aa61184a393c006314a8c3e7.a1.bin
**a19b78c94fcfcb863fbf43b9ce11d85a**  400000005c51aa61184a393c006314a8c3e7.a1.bin

disk's i-nodes are full

Filesystem                Inodes   IUsed   IFree IUse% Mounted on

/dev/mapper/system-var   3604480 3600798    **3682  100%** /var

[root@vPiler piler]# piler -v 1.3.4 build 994

[Dennis]

Hey Janos,

thank you very much for your answer! There were indeed 7-9 emails that couldn't be imported - don't know why, but now it works! The piler runs on a very old VM using Ubuntu 14.04.5 LTS using the version "piler 1.1.1, build 904", so it is pretty outdated.

We are know thinking of doing a fresh installation on a new VM using Ubuntu 18.04 and the newest version of piler. Do you have any tips for migrating the old database on the new version?

[Janos SUTO]

Check out http://www.mailpiler.org/wiki/current:migration-to-new-host how to migrate to a new host.

[Janos SUTO]

@TOmi: The workaround should be the same for you as well. If your build doesn't have /var/piler/error dir, then try upgrading it to the latest master branch.

[TOmi]

Is it good idea to use developer version in main server ?
Folder /var/piler/error is empty. Maybe in my case where server works for 900 accounts number of worker process was not enought ? Yesterday evening I change number_of_worker_processes from 4 to 8. Which option can I change additionaly ? Now I'm checking server.

[Janos SUTO]

If you have /var/piler/error dir, then your build should be able to move any problematic email there. I'd stop the piler daemon, rename tmp to tmp.old, then create tmp dir as suggested above, then remove any duplicate files from tmp.old, and then start piler daemon. Also check the mail log if piler tries to reprocess the same message-id over and over again. That would explain why you have /var/piler/tmp filled at the moment. Unless they are old stale files. In such case feel free to remove them.

[TOmi]

I have /var/piler/error it was there from same beginning. I've do it as you say but something went wrong and tmp filder was very big it happened twice. My Piler was started 1 january 2019. No I have emergency script - if folder tmp is bigger then 1 GB it stop piler process , rename it , create new and start product. Then I can analise what happened and when, so I have to waite. Maby afrer changing count of process in will work corect.

[root@vPiler piler]# ls -l /var/piler
total 548
drwx--x--x.  2 piler piler   4096 Feb  1 10:01 error
drwx------.  2 piler piler   4096 Dec 14 08:26 imap
drwxr-xr-x   2 piler piler   4096 Feb  1 12:15 sphinx
drwxr-xr-x.  2 piler piler   4096 Dec 14 08:45 stat
drwxr-xr-x   5 piler piler 258048 Feb  1 07:42 store
drwxr-xr-x  10 piler piler 282624 Feb  1 12:25 tmp
drwxr-xr-x. 15 root  root    4096 Jan 28 13:43 www

[Janos SUTO]

OK, I hope your piler deployment will be fine.