Dupilcate mails show up in gui

[Karthick jraj]

I have piler 1.1.1 with zimbra. A user from zimbra sent mail to multiple users. When i check in archive it shows multiple mail in gui. Both from user login and archive login it shows multiple mails.

Is there any option to ignore the duplicate not display from the gui. I checked message id is also same for both.

[Janos SUTO]

Show me the related logs to see that piler actually stores duplicates. I doubt that it happens.

[Karthick jraj]

Hi,

here is the log file,

May 6 09:31:04 sbademo piler[3902]: 40000000554992093462d5d400852389dd9d: from=[e-mail], size=343962/130944, attachments=0, reference=[e-mail], message-id=[e-mail], retention=14, delay=0.13, delays=0.11/0.01/0.00/0.01/0.00/0.00, status=stored May 6 09:31:04 sbademo piler[3907]: 400000005549920934e7c12c005b79b41509: from=[e-mail], size=343959/130944, attachments=0, reference=[e-mail], message-id=[e-mail], retention=14, delay=0.18, delays=0.15/0.01/0.00/0.01/0.00/0.00, status=stored May 6 09:31:04 sbademo piler[3903]: 400000005549920a1119ad8c00bfe4ca5879: mysql_stmt_execute error: Duplicate entry '[e-mail]' for key 'id' (errno: 1062) May 6 09:31:04 sbademo piler[3903]: 400000005549920a1119ad8c00bfe4ca5879: discarding: duplicate message, id: 2988, message-id: [e-mail] May 6 09:31:04 sbademo piler[3903]: 400000005549920a1119ad8c00bfe4ca5879: from=[e-mail], size=343962/0, attachments=0, reference=[e-mail], message-id=[e-mail], retention=0, delay=0.13, delays=0.12/0.01/0.00/0.00/0.00/0.00, status=duplicate May 6 09:31:04 sbademo piler[3908]: 400000005549920a11dd8e14005850de258c: mysql_stmt_execute error: Duplicate entry '[e-mail]' for key 'id' (errno: 1062) May 6 09:31:04 sbademo piler[3908]: 400000005549920a11dd8e14005850de258c: discarding: duplicate message, id: 2988, message-id: [e-mail]

[Janos SUTO]

Please format the log. You don't expect me to torture my eyes, do you?

[Karthick jraj]

Sorry for that...

May 6 09:31:04 sbademo piler[3902]: 40000000554992093462d5d400852389dd9d: from=[e-mail], size=343962/130944, attachments=0, reference=[e-mail], message-id=[e-mail], retention=14, delay=0.13, delays=0.11/0.01/0.00/0.01/0.00/0.00, status=stored

May 6 09:31:04 sbademo piler[3907]: 400000005549920934e7c12c005b79b41509: from=[e-mail], size=343959/130944, attachments=0, reference=[e-mail], message-id=[e-mail], retention=14, delay=0.18, delays=0.15/0.01/0.00/0.01/0.00/0.00, status=stored

The above is the log.

[Janos SUTO]

I added a debug code around the deduplication detection, we'll see what happens.

[Karthick jraj]

Still I see duplicate mails getting stored.

[Janos SUTO]

Can we see the logs?

[Janos SUTO]

Please download https://bitbucket.org/jsuto/piler/get/mmap_dedup.tar.gz, and install it. Make sure you set mmap_dedup_test=1 in piler.conf to enable the new feature.

[Janos SUTO]

Also please describe your environment. Is it a physical host or a virtual? What filesystem and mount options do you have for /var?

[Karthick jraj]

I have installed this fix. let me check for a day and will revert you back.

By the way the environment is as explained below, Physical server Centos 6.2 X86_64 /var is mount with ext4 filesystem as lvm partition.

[Karthick jraj]

Thanks this fix solved the duplicate mails issue.