Hundreds of thousands of emails in the /var/piler/tmp
Issue #1328
closed
Hello Janos,
We have a problem with hundreds of thousands of emails in the /var/piler/tmp (in our case /opt/piler/tmp) directory. We have quite an old version of piler, so we are planning to migrate to a completely new server using pilerget and pilerimport commands (we don't want to migrate database or sphinx from the old installation - just start fresh with new piler and import all messages), but the problem is we don't know what to do with emails from this tmp directory and why they started appearing there two days ago in such high numbers (145635 files created during last 2 days)? It looks like it all started 25th of October at 3:49am and as a result left space on the device dropped from about 350 GB to almost 0 (you can see attached image with the graph from zabbix). It looks like those messages are not in the mysql database, and when we try to open them with pilerget, for example
pilerget 40000000653877c92b1e5e440018376bc5da
we get
Oct 26 13:29:07 z-mailpiler pilerget[1431]: /opt/piler/store/00/653/c5/da/40000000653877c92b1e5e440018376bc5da.m: cannot open()
Oct 26 13:29:11 z-mailpiler pilerget[1450]: /opt/piler/store/00/653/c5/da/40000000653877c92b1e5e440018376bc5da.m: cannot open()
in /var/log/syslog.
root@z-mailpiler:/home/intalio# piler -v
1.3.4 build 994
root@z-mailpiler:/home/intalio# du -sh /opt/piler/tmp
381G /opt/piler/tmp
root@z-mailpiler:/home/intalio# ls -la /opt/piler/tmp |wc -l
164286
root@z-mailpiler:/home/intalio# find /opt/piler/tmp -type f -ctime -1 -exec ls -lah {} + |wc -l
126
root@z-mailpiler:/home/intalio# find /opt/piler/tmp -type f -ctime -2 -exec ls -lah {} + |wc -l
145635
root@z-mailpiler:/home/intalio# date
czw, 26 paź 2023, 14:25:35 CEST
just a few files from the directory (paź is Oct)
-rw------- 1 piler piler 8164 paź 25 03:49 400000006538743332365ccc002074a660e8.m
-rw------- 1 piler piler 4725618 paź 25 03:49 4000000065387433367f87cc0076ce5571a9.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 4000000065387433367f87cc0076ce5571a9.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 4000000065387433367f87cc0076ce5571a9.m
-rw------- 1 piler piler 4725618 paź 25 03:49 400000006538743337db7a0400db275e38ba.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 400000006538743337db7a0400db275e38ba.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 400000006538743337db7a0400db275e38ba.m
-rw------- 1 piler piler 4725618 paź 25 03:49 40000000653874333acc0a440057ec624888.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 40000000653874333acc0a440057ec624888.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 40000000653874333acc0a440057ec624888.m
-rw------- 1 piler piler 4725618 paź 25 03:49 40000000653874340069f30c00b2cfd349bf.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 40000000653874340069f30c00b2cfd349bf.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 40000000653874340069f30c00b2cfd349bf.m
-rw------- 1 piler piler 4725618 paź 25 03:49 4000000065387434018219cc0077ad4d2669.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 4000000065387434018219cc0077ad4d2669.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 4000000065387434018219cc0077ad4d2669.m
-rw------- 1 piler piler 4725618 paź 25 03:49 40000000653874340652a12400f302b6f797.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 40000000653874340652a12400f302b6f797.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 40000000653874340652a12400f302b6f797.m
-rw------- 1 piler piler 4725618 paź 25 03:49 4000000065387434070c34f400c5122ab437.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 4000000065387434070c34f400c5122ab437.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 4000000065387434070c34f400c5122ab437.m
-rw------- 1 piler piler 4725618 paź 25 03:49 400000006538743409b3b9fc00e5038b1124.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 400000006538743409b3b9fc00e5038b1124.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 400000006538743409b3b9fc00e5038b1124.m
-rw------- 1 piler piler 4725618 paź 25 03:49 40000000653874340c85f88400922cd678c1.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 40000000653874340c85f88400922cd678c1.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 40000000653874340c85f88400922cd678c1.m
...
-rw------- 1 piler piler 4725618 paź 25 03:49 400000006538741f18446cdc00557a5af45d.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 400000006538741f18446cdc00557a5af45d.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 400000006538741f18446cdc00557a5af45d.m
-rw------- 1 piler piler 8164 paź 25 03:49 400000006538741f17c5de6c001c8c11f872.m
-rw------- 1 piler piler 4725618 paź 25 03:49 400000006538741f03047bdc002e7aa7eb56.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 400000006538741f03047bdc002e7aa7eb56.a1.bin
-rw------- 1 piler piler 4725618 paź 25 03:49 400000006538741f01a27ec40020a456d16d.a1
-rw-r--r-- 1 piler piler 3453335 paź 25 03:49 400000006538741f01a27ec40020a456d16d.a1.bin
-rw------- 1 piler piler 8164 paź 25 03:49 400000006538741f03047bdc002e7aa7eb56.m
-rw------- 1 piler piler 8164 paź 25 03:49 400000006538741f01a27ec40020a456d16d.m
-rw-r----- 1 piler piler 1926 paź 25 03:48 3CLCYSLQ8WE84AY4
-rw-r----- 1 piler piler 1926 paź 25 03:47 GQHSJGXYVB9W31F6
-rw-r----- 1 piler piler 1926 paź 25 03:46 J0VDQJB6IDXXPECR
-rw-r----- 1 piler piler 1926 paź 25 03:45 XOQHGI3QUBWOMUNN
-rw-r----- 1 piler piler 1926 paź 25 03:44 QU45DW5FSGOFJ6XU
-rw-r----- 1 piler piler 1926 paź 25 03:43 MA4PZ603B9Y8NB01
Obviously we can't delete them, because they would be lost. Is pilerimport able to detect that these messages are split into text and attachments and reimport them? What should we do? Or what should we check to see why so much free space on the hard drive has filled up in such a short time?
Kind regards
Comments (2)
-
repo owner
-
repo owner
- changed status to closed
No news is good news.
- Log in to comment
Hello Krzysztof, I think there’s a bug in that piler release that prevents it to parse this email properly, so it’s stuck, and keeps writing those temp files over and over again.
I suggest to stop piler, rename /opt/piler/tmp to /opt/piler/tmp.old (then be sure to recreate /opt/piler/tmp with proper rights and ownership). If the sending mail server still has it in its queue, then it will start again, so in that case you need to isolate this problematic message, and don’t let it be sent to piler, try to process it on the new archive.