-
assigned issue to
pilerimport rejects messages with no reason
Version:
$ piler -v
1.2.0-master build 948
$ mysql -V ##
mysql Ver 14.14 Distrib 5.7.13, for Linux (x86_64) using EditLine wrapper
$ lsb_release -a ##
Distributor ID: Ubuntu Description: Ubuntu 16.04.1 LTS Release: 16.04
Codename: xenial
iNodes free:
df -i
S.ficheros Nodos-i NUsados NLibres NUso% Montado en
udev 495337 475 494862 1% /dev
tmpfs 500129 608 499521 1% /run
/dev/sda1 29605888 120031 29485857 1% /
tmpfs 500129 1 500128 1% /dev/shm
tmpfs 500129 5 500124 1% /run/lock
tmpfs 500129 16 500113 1% /sys/fs/cgroup
/dev/sdb1 15269888 307777 14962111 3% /mnt/sdb1
tmpfs 500129 4 500125 1% /run/user/1000
Issue
I have just reseted the installation, resetting indexes with indexer -a, and executed a massive pilerimport process from maildir directory format (readpst extracted)
nohup readpst -C windows-1252 -M -b -te -o ./B-THINKING/maildir/ ./B-THINKING.pst > B-THINKING/log/readpst_maildir.log &
nohup pilerimport -d ./B-THINKING/maildir/B-THINKING -f B-THINKING > B-THINKING/log/pilerimport_maildir.log &
The pilerimport reports:
4300 mails processed, with 13 duplicated
219 messages failed to import
212 invalid message: hdr_len: 0
If, after this report, I try to import one by one (with pilerimport -e), all the failed and invalid import correctly (tested with 5 messages of each case)
NOTE: For me this bug is a blocker. If I cannot trust in the import process, or I have to re-process everytime, I cannot continue (I have near 500.000 messages for archiving, in 15 separate environments (pst, maildir, mbox)
I attach:
-
pilerimport output
-
mail.log (filtered by pilerimport messages)
-
readpst output
NOTE: For resetting I mean:
# Stop searchd and piler:
sudo service cron stop
sudo /etc/init.d/rc.searchd stop
sudo /etc/init.d/rc.piler stop
# Remove stored emails:
sudo rm -rf /var/piler/store/00/*
# Drop and recreate the piler database:
mysql> drop database piler;
mysql> create database piler character set utf8mb4 collate utf8mb4_spanish_ci;
mysql -u piler -p piler < ./util/db-mysql.sql
# Reset the sphinx index data
sudo rm -rf /var/piler/sphinx/*
sudo su - piler
indexer --all
# Start searchd and piler:
sudo /etc/init.d/rc.searchd start
sudo /etc/init.d/rc.piler start
sudo service cron start
Comments (5)
-
reporter -
reporter - edited description
-
repo owner Hmm, that seems to be a tough one. I'd like you to reset the archive, and re-run the import 2-3 times to see if the issue is consistent. Also I'd like to see some of the emails reported with hdr_len=0. Also try the import without the -f option (I'm not sure if you actually need it). And thanks for the formatting, it makes the issue much easier to read.
-
repo owner Any update? Btw. I've tested the import with ~2400 messages, and found no issue with it.
-
repo owner - changed status to closed
No news is good news. Reopen the issue in case of an update.
- Log in to comment