Version:

$ piler -v

1.2.0-master build 948

$ mysql -V ##

mysql Ver 14.14 Distrib 5.7.13, for Linux (x86_64) using EditLine wrapper

$ lsb_release -a ##

Distributor ID: Ubuntu Description: Ubuntu 16.04.1 LTS Release: 16.04

Codename: xenial

iNodes free:

df -i

S.ficheros Nodos-i NUsados NLibres NUso% Montado en

udev 495337 475 494862 1% /dev

tmpfs 500129 608 499521 1% /run

/dev/sda1 29605888 120031 29485857 1% /

tmpfs 500129 1 500128 1% /dev/shm

tmpfs 500129 5 500124 1% /run/lock

tmpfs 500129 16 500113 1% /sys/fs/cgroup

/dev/sdb1 15269888 307777 14962111 3% /mnt/sdb1

tmpfs 500129 4 500125 1% /run/user/1000

Issue

I have just reseted the installation, resetting indexes with indexer -a, and executed a massive pilerimport process from maildir directory format (readpst extracted)

nohup readpst -C windows-1252 -M -b -te -o ./B-THINKING/maildir/ ./B-THINKING.pst > B-THINKING/log/readpst_maildir.log &
nohup pilerimport -d ./B-THINKING/maildir/B-THINKING -f B-THINKING > B-THINKING/log/pilerimport_maildir.log &

The pilerimport reports:

4300 mails processed, with 13 duplicated

219 messages failed to import

212 invalid message: hdr_len: 0

If, after this report, I try to import one by one (with pilerimport -e), all the failed and invalid import correctly (tested with 5 messages of each case)

NOTE: For me this bug is a blocker. If I cannot trust in the import process, or I have to re-process everytime, I cannot continue (I have near 500.000 messages for archiving, in 15 separate environments (pst, maildir, mbox)

I attach:

pilerimport output
mail.log (filtered by pilerimport messages)
readpst output

NOTE: For resetting I mean:

# Stop searchd and piler:

sudo service cron stop

sudo /etc/init.d/rc.searchd stop

sudo /etc/init.d/rc.piler stop

# Remove stored emails:

sudo rm -rf /var/piler/store/00/*

# Drop and recreate the piler database:

mysql> drop database piler;

mysql> create database piler character set utf8mb4 collate utf8mb4_spanish_ci;

mysql -u piler -p piler < ./util/db-mysql.sql

# Reset the sphinx index data

sudo rm -rf /var/piler/sphinx/*

sudo su - piler

indexer --all

# Start searchd and piler:

sudo /etc/init.d/rc.searchd start

sudo /etc/init.d/rc.piler start

sudo service cron start

pilerimport rejects messages with no reason