Problem importing from Zimbra imap server

[Ludji]

Hello,

I have Zimbra 7.2.2 with it's integrated IMAP server running, and when I try to import mails from the imap server to piler using pilerimport, it fails.

the command I use :

pilerimport -i imapserverIP -u [e-mail] -p superpwd -P 993

The output I get :

server cert:
     subject: /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=mail.zimbra.fr
     issuer: /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=mail.zimbra.fr

login failed, server reponse: A

I'm sure my login/pwd combination is correct.

I've tried to disable SSL connection and enable cleartext logins on my IMAP server, then I run :

pilerimport -i imapserverIP -u [e-mail] -p superpwd

it works, messages are imported, but it suddenly blocks at INBOX-2605, here's the output :

duplicate: INBOX-2599.txt (id: 40000000535a2f852b972ff400f4e154ee77)
duplicate: INBOX-2600.txt (id: 40000000535a2f852f0b7ec400323e582a9c)
duplicate: INBOX-2601.txt (id: 40000000535a2f8537b1416c001408668e5e)
duplicate: INBOX-2602.txt (id: 40000000535a2f853a9d5154009d96d9cbb5)
duplicate: INBOX-2603.txt (id: 40000000535a2f8601fa205c0029b55a600a)
duplicate: INBOX-2604.txt (id: 40000000535a2f86030c8a5c003d2f0b7da2)
duplicate: INBOX-2605.txt (id: 40000000535a2f8605cb352c0076e0b2f70f)
processed:    2606

Also, I see processed : 2606, however in Piler they are only 2020 something messages imported, seems like some importation failed. (the duplicate is there because i've ran the command twice, to check if it would pass message 2606, but it didn't).

SSL imap connection without cleartext login isn't supported with pilerimport ?

Help would be great.

Regards,

Ludji.

[Janos SUTO]

pilerimport uses the LOGIN <username> <password> command. What login procedure you need for the SSL login to succeed?

Also what version of piler do you use? Since 0.1.24 the imap protocol handling is improved, so it may be worth to try the latest master branch.

[Ludji]

Well i've tried login in with openssl into my IMAP server, using the LOGIN <user> <pwd> command, and it works fine.

Command i run :

 openssl s_client -connect <zimbraImapIPADDR>:993

Guess it needs a special SSL client to read the encryption, whenever i try with telnet, the LOGIN <user> <pwd> fails.

My version :

piler 0.1.24-master-branch, build 836, Janos SUTO [e-mail]

Build Date: Fri Apr 18 10:48:07 CEST 2014 ldd version: ldd (Debian EGLIBC 2.13-38+deb7u1) 2.13 gcc version: gcc version 4.7.2 (Debian 4.7.2-5)

[Ludji]

Atfer upgrade, the login works but importing emails fails, here's the log :

error importing '4127-imap-1215.txt'
4127-imap-1216.txt: invalid message, hdr_len: 0
error importing '4127-imap-1216.txt'
4127-imap-1217.txt: invalid message, hdr_len: 0
error importing '4127-imap-1217.txt'
4127-imap-1218.txt: invalid message, hdr_len: 0
error importing '4127-imap-1218.txt'
4127-imap-1219.txt: invalid message, hdr_len: 0
error importing '4127-imap-1219.txt'
4127-imap-1220.txt: invalid message, hdr_len: 0
error importing '4127-imap-1220.txt'
4127-imap-1221.txt: invalid message, hdr_len: 0
error importing '4127-imap-1221.txt'
4127-imap-1222.txt: invalid message, hdr_len: 0
error importing '4127-imap-1222.txt'
4127-imap-1223.txt: invalid message, hdr_len: 0
error importing '4127-imap-1223.txt'
4127-imap-1224.txt: invalid message, hdr_len: 0
error importing '4127-imap-1224.txt'
4127-imap-1225.txt: invalid message, hdr_len: 0
error importing '4127-imap-1225.txt'

the importation always stops at message 1225 and for every single message, it displays this error, only 400 mails have been correctly imported to Piler.

Let me know if you need the full log.

[Janos SUTO]

How about a php utility that can download messages via imap? Then you'll have every email in the current directory, and you can run pilerimport -d /path/to/dir.

[Ludji]

I can only find utilities that can transfer emails from one IMAP account to another, not from an account to a folder.

Does the pilerimport -d support metadata ? I know a way to export emails from zimbra to directories, emails are in .eml files and their metadata are in .meta files, is this supported ?

[Janos SUTO]

Not sure. What's in a .meta file? And why is it needed for archiving the given email?

[Ludji]

I can attach you one if you want to, but I don't need them anymore, I thought they were holding some useful informations but all these informations are in the headers of a .eml file, so it's fine :) sorry for that.

I still have an issue with pilerimport.. Even the -d /path/to/dir suddently stops at a specific email, is there a way to see what's going on ? A log file or a debug mode of pilerimport ? I really need to know what's wrong and why does this command freezes, it imports 8400+ mails then freezes on the 8403... Maybe something wrong with this one email ?

[Ludji]

Seems to block on this specific .eml file, is there something wrong with it ? Edit : just imported this single mail with pilerimport -e file and it worked.

I really don't understand why the command freezes during its process..

[Janos SUTO]

Can you check if pilertest can process it? It merely parses the message, prints some statistics, etc, and is able prove any parser related issue.

[Janos SUTO]

When pilerimport fails, it prints or syslogs an error. Also make sure you run pilerimport in both cases from /tmp. If it's possible to check out pilerimport on your site via teamviewer, then let's discuss it on skype (janos.suto).

[Janos SUTO]

Anyway I copied this email to a test installation, and "pilerimport -e aaaa.eml" has archived this message without problem, and I could restore it from the archive.

[Janos SUTO]

I think we managed to solve this issue.