1. Janos SUTO Avatar Janos SUTO
  2. Piler
  3. piler
  4. Issues

pilerimport gets stuck doing IMAP import. It will not respect folder exclude also

Issue #925 resolved
Norbert Schmidt created an issue

Hello, I am trying to import mail from an IMAP-Server. I am using this command to import the mail: pilerimport -i email.xxx.de -u xxxx@xxx.de -p "secure" -P 993 This always gets stuck at 52% mail # 24595 of folder "Sent Items". To get around this problem I tried to skip the folder using this command:

pilerimport -i email.xxx.de -u xxxx@xxx.de -p "secure" -P 993 -x 'Sent Items,'

The process does not respect the exclude and again tries to import the mails within "Sent Items" folder getting stuck at the same place.

What can I do to get around the problem. It already would help to successfully exclude the "Sent Items" folder...

I've had a look into the files created within the temp folder. All of the created files are empty. The last created file always has the same counting number each time I restart the import. Thus I've now got 6 empty files named as follows: root@mailarchiv:/var/piler/tmp# ls -la|grep 24596 -rw------- 1 piler piler 0 Jun 25 21:45 22726-imap-24596.txt -rw------- 1 piler piler 0 Jun 25 22:34 24018-imap-24596.txt -rw------- 1 piler piler 0 Jun 25 23:53 25532-imap-24596.txt -rw------- 1 piler piler 0 Jun 26 11:55 4938-imap-24596.txt -rw------- 1 piler piler 0 Jun 26 12:03 5519-imap-24596.txt -rw------- 1 piler piler 0 Jun 26 12:11 6091-imap-24596.txt

Thanks Norbert

The piler-installation is a fresh install of the last current release. Here's the output of piler -V piler 1.3.4, build 994, Janos SUTO sj@acts.hu

Build Date: Fri Jun 8 16:14:08 UTC 2018 ldd version: ldd (Debian GLIBC 2.19-18+deb8u10) 2.19 gcc version: gcc version 4.9.2 (Debian 4.9.2-10+deb8u1) OS: Linux mailarchiv 3.16.0-5-amd64 #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) x86_64 GNU/Linux Configure command: ./configure --localstatedir=/var --with-database=mysql --enable-tcpwrappers MySQL client library version: 5.5.60 Extractors: /usr/bin/pdftotext /usr/bin/catdoc /usr/bin/catppt /usr/bin/xls2csv /usr/bin/unrtf /usr/bin/tnef

Comments (16)

  5. Janos SUTO repo owner

    Try to download only the emails. pilerimport has an option for this: -o

    If you can download them all, then it might be a parser issue. If the download fails at the same message, then it's a protocol error in pilerimport.

  6. Norbert Schmidt reporter

    I used option -o It now seems to skip the rest of the messages in the folder, here's the output:

    Unbenannt.png

    then it continues with the next folder... But get's stuck on a later folder again...

    What else to debug this? Thanks Norbert

  7. Janos SUTO repo owner

    Those "ERROR: select cmd error" messages concern me. What imap server do you have?

    Anyway try to get the emails with something other than pilerimport, perhaps use your email client to download all the messages, and export them as eml files. Then pilerimport may process the eml files.

    Note that the webui directory has some Zend mail libraries. I'll try to come up with a php script on the weekend that can download the emails from an imap server.

  8. Norbert Schmidt reporter

    It is a Kerio Connect Server. I could try to download the mails using "imapsync" or something similar. I'll have to look into this tool if it can download to eml files. I am looking forward to your php script...

    Thanks Norbert

  10. Janos SUTO repo owner

    I forgot that the tool was ready: check out util/download-imap.php. It doesn't offer anything fancy, it tries to download all messages to the current directory. Once it's done, run pilerimport with -r option which removes all successfully imported emails. Then you'll able to pinpoint the problematic message causing trouble for piler.

  11. Norbert Schmidt reporter

    I tried the tool, question is: "What's the final word of the tool?" It ended with "process killed". All the mails it downloaded got imported well with pilerimport but I am not sure wether it downloaded all mails..... Thanks Norbert

  12. Janos SUTO repo owner

    It's definitely not "process killed". It should return the prompt when finished. Try running it again, if it keeps failing then you have to get the emails an other way (perhaps you own the Kerio imap server, and can copy them to the piler host somehow). Also check the Kerio logs and look for anomalies during the message download.

  13. Norbert Schmidt reporter

    Ok, thank you. I've now taken the last full-backup of the Kerio server, which contained the mails in eml format. I've now fed these eml files into pilerimport and thus have imported 128000 mails last night so hopefully all mails are imported now. Thanks a lot for your support Norbert

  17. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
resolved
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!