IMAP import - too short message!

[Pilcheck]

Hello,

Piler has been great since we started to archive emails but now its time to get old messages into the archive. We are attempting to import emails via IMAP from Zimbra 8.0.7 to Piler 0.1.25-rc2, build 871.

As we tried to import the web interface show messages being processed, the disk usage also increased an appropriate amount. When we search for older messages they can not be found. We are in /tmp and see files being created, but they are all empty.

How can we help debug this issue?

Here's what we're doing:

root@piler:/tmp# pilerimport -i mail.domain.com -u [e-mail] -p 'supersecret' -P 993

And the response:

server cert:
     subject: /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.hellolife.net
     issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=PositiveSSL CA 2

List of IMAP folders:
=> 'Adverts'
=> 'An Undeliverable List'
=> 'Chats'
=> 'Contacts'
=> 'Cron Daemon'
=> 'Daily Mails'
=> 'Drafts'
=> 'Emailed Contacts'
=> 'Good Stuff'
=> 'INBOX'
=> 'Junk'
=> 'LOPSA'
=> 'Low-Volume Lists'
=> 'NANOG'
=> 'Proxmox List'
=> 'Sent'
=> 'Trash'
=> 'zabbix'
processing folder: Daily Mails... found 287 messages
invalid reply: *1: too short message! *
23118-imap-1.txt: invalid message, hdr_len: 0
error importing '23118-imap-1.txt'
invalid reply: *3: too short message! *
23118-imap-3.txt: invalid message, hdr_len: 0
error importing '23118-imap-3.txt'
invalid reply: *5: too short message! *
23118-imap-5.txt: invalid message, hdr_len: 0
error importing '23118-imap-5.txt'
invalid reply: *7: too short message! *
23118-imap-7.txt: invalid message, hdr_len: 0
error importing '23118-imap-7.txt'
invalid reply: *9: too short message! *
23118-imap-9.txt: invalid message, hdr_len: 0
error importing '23118-imap-9.txt'
invalid reply: *11: too short message! *
23118-imap-11.txt: invalid message, hdr_len: 0

root@piler:/tmp# ls -halt
total 1.4M
drwxrwxrwt  2 root  root  1.3M Jul  1 14:51 .
-rw-------  1 piler piler  23K Jul  1 14:51 27032-imap-19.txt
-rw-------  1 piler piler    0 Jul  1 14:51 4000000053b303411a9d4d1400b5578af4d6.m
-rw-------  1 piler piler    0 Jul  1 14:51 27032-imap-18.txt
-rw-------  1 piler piler    0 Jul  1 14:51 4000000053b303411a1a03b4006fbe587c44.m

[Janos SUTO]

Can you show me the network dump (exclude any authentication related info)? An ngrep output would be great. Anyway it's possible that there's a bug in the imap protocol handling routine. We'll see from the network traffic.

Also there's an alternative method to import emails from an imap server. Use the util/download-imap.php script, and it will down exactly what its name suggests, then you may use pilerimport to process emails from the current directory.

[Pilcheck]

We decided to enable clear-text logins and the pilerimport works as exptected. I have emailed you the first 1000 lines of both port 143 and port 993 ngreps as well as pilerimport output. I hope that helps.

Also, is there a way to empty emails per user?
Since Piler sees these duplicates I do not believe it is inserting the missing data.

[Janos SUTO]

If you meant that how to remove emails after importing from an imap account, then use the -r switch to pilerimport.

However if the pilerimport output is what you sent me, then it's not ok, not good. In that case try to download messages with the util/download-imap.php script.

Anyway I'll setup a zimbra community edition, and try its imap stuff.

[Janos SUTO]

Please overwrite src/imap.c with the attached file, the recompile piler. Then try running pilerimport with dry run (add the -D option), and check both port 143 and 993 if pilerimport can process all folders and emails properly.

[Pilcheck]

After recompiling with your attached imap.c port 143 and 993 both work properly, thank you!

Also, just for your information, Zimbra 8 introduced imap_throttle controls which may halt imports and pilerimport does not exit.

Zimbra log shows this:

...
2014-07-09 13:08:05,950 INFO  [ImapSSLServer-3362] [name=[e-mail];mid=94;ip=10.0.12.61;] imap - FETCH elapsed=1
2014-07-09 13:08:05,951 WARN  [ImapSSLServer-3361] [name=[e-mail];mid=94;ip=10.0.12.61;] imap - too many IMAP requests from remote ip 10.0.12.61 dropping connection
2014-07-09 13:08:05,951 INFO  [ImapSSLServer-3361] [name=[e-mail];mid=94;ip=10.0.12.61;] imap - FETCH elapsed=0
2014-07-09 13:08:05,951 INFO  [ImapSSLServer-3361] [name=[e-mail];mid=94;ip=10.0.12.61;] imap - dropping connection for user [e-mail] (server-initiated)

Pilerimport displays:

processing folder: ToDo ... found 51 messages
processed:      51
processing folder: Contacts... found 2 messages
processed:      53
processing folder: Returns... found 200 messages
processed:     253
^C
root@piler:/tmp#

ngrep displays this:

T 10.0.12.51:143 -> 10.0.12.61:38414 [AP]
  A2026 OK FETCH completed..                                                                                                                                              
#
T 10.0.12.61:38414 -> 10.0.12.51:143 [AP]
  A2027 FETCH 999 (BODY.PEEK[])..                                                                                                                                         
#
T 10.0.12.51:143 -> 10.0.12.61:38414 [AP]
  * BYE zimbra8.domain.tld Zimbra IMAP4rev1 server closing connection..   

Maybe pilerimport can timeout or alert us after receiving a BYE.

Thanks for you attention.

[Janos SUTO]

Thanks for noticing. Can you show me where/how to set this variable? I'd test it for myself with my zimbra test installation.

[Pilcheck]

The defaul throttle settings can be found deep in Zimbra 8's release notes. Page 19: http://files.zimbra.com/website/docs/8.0/ZCS_807_OS_ReleaseNotes_UpgradeInst.pdf

To see our current throttle settings:
zimbra@zimbra8:~$ zmlocalconfig | grep imap_throttle

To change these:
root@zimbra8:/opt/zimbra# su zimbra 
zimbra@zimbra8:~$ zmlocalconfig -e imap_throttle_acct_limit=1000
zimbra@zimbra8:~$ zmlocalconfig -e imap_throttle_ip_limit=1000
zimbra@zimbra8:~$ zmlocalconfig -e imap_throttle_fetch=false

I have tried a cominbation of settings but I still get stuck on this ip_limit. The odd thing is, I can import all emails for admin accounts (25,000+ emails) but normal user accounts can not get past this ip_limit, and they all stop around 253 imports.

The piler host is in the mta trusted networks.

I also increased zimbraImapMaxConnections becasue I saw it recommended in my search, but this may not actually be required.

Get current setting:
zimbra@zimbra8:~$ zmprov gs zimbra8.domain.tld zimbraImapMaxConnections

Modify setting:
zimbra@zimbra8:~$ zmprov ms zimbra8.domain.tld zimbraImapMaxConnections 1000

We restart zimbra to update settings:
zimbra@zimbra8:~$ zmcontrol restart

Then we watch the log:

zimbra@zimbra8:~$ exit
root@zimbra8:/opt/zimbra# tail -f /opt/zimbra/log/mailbox.log | grep imap

[Janos SUTO]

I think the issue is fixed by now.