1. Janos SUTO Avatar Janos SUTO
  2. Piler
  3. piler
  4. Issues

discarding duplicate message

Issue #1223 wontfix
Adiyal Ketan created an issue

I have checked the piler logs and showing lost of the email are discard. please check the below logs.

lot of email are not showing.

how to skip the discarding duplicate message.

Jan 13 11:17:32 njuser piler[48587]: 1/O6WUC188OANMN9V8: discarding: duplicate message, id: 0, message-id: 384717540.15.1642052713589.JavaMail.tomcat
Jan 13 11:17:32 njuser piler[65212]: 1/O6WUC188OANMN9V8: discarding: duplicate message, id: 0, message-id: 384717540.15.1642052713589.JavaMail.tomcat
Jan 13 11:17:32 njuser piler[74490]: 1/O6WUC188OANMN9V8: discarding: duplicate message, id: 0, message-id: 384717540.15.1642052713589.JavaMail.tomcat
Jan 13 11:17:32 njuser piler[56144]: 0/RHIZJIBIAQWAIAJA: discarding: duplicate message, id: 0, message-id: 827537948.117.1642052714091.JavaMail.tomcat

Comments (17)

  1. Janos SUTO repo owner

    I’m not sure what you mean by “skip the discarding duplicate message”. When piler discards a message, because it thinks it’s a duplicate, then it syslogs the fact, because it’s crucial to log every important decision about an email. Discarding an incoming message instead of archiving it is pretty important.

  2. Adiyal Ketan reporter

    yes, i want to skip the discarding duplicate message id.

    beacuse i have no idea but lot of email are not visable.

    so can you guide how we can do.

    ‌

  3. Janos SUTO repo owner

    Let me rephrase it: I won’t make such a change. You need to debug why you get the same discarded message several times. For starters, check if piler is started only once. Be sure to kill it until no piler daemon is running, then start it again. Another problem might be if piler can’t process a message, and it’s trying to re-process it over and over. Recent piler builds should move such email to /var/piler/error dir.

  5. Adiyal Ketan reporter

    I have stopped the piler service and start as per above troubleshoot point but same issue.

    latest arrive message showing duplicate message id.

    please suggest lot of email are received in piler but not visable after search in piler web pannel.

    /var/piler/error directory are empty.

    As per lot of search below solution are mentioned i have tried but not work.

    archive_emails_not_having_message_id=1 in piler.conf – not working solutution

    so please suggest.

    ‌

    Feb 1 17:32:30 njuser piler[56997]: 1/I14K594CNRCPK5YC: discarding: duplicate message, id: 0, message-id: 31334175.15327.1643716949173.JavaMail.tomcat
    Feb 1 17:32:30 njuser piler[98991]: 1/I14K594CNRCPK5YC: discarding: duplicate message, id: 0, message-id: 31334175.15327.1643716949173.JavaMail.tomcat
    Feb 1 17:34:58 njuser piler[121949]: 1/WZIA6DSAB878IEY1: discarding: duplicate message, id: 0, message-id: 1706671976.36084.1643717097239.JavaMail.tomcat
    Feb 1 17:34:59 njuser piler[111897]: 1/WZIA6DSAB878IEY1: discarding: duplicate message, id: 0, message-id: 1706671976.36084.1643717097239.JavaMail.tomcat
    Feb 1 17:34:59 njuser piler[98991]: 1/WZIA6DSAB878IEY1: discarding: duplicate message, id: 0, message-id: 1706671976.36084.1643717097239.JavaMail.tomcat
    Feb 1 17:34:59 njuser piler[111897]: 1/GDIW9HX9A2JKTMN5: discarding: duplicate message, id: 46560815, message-id: 2069438152.36087.1643717098177.JavaMail.tomcat
    Feb 1 17:34:59 njuser piler[98991]: 1/GDIW9HX9A2JKTMN5: discarding: duplicate message, id: 46560815, message-id: 2069438152.36087.1643717098177.JavaMail.tomcat
    Feb 1 17:34:59 njuser piler[32540]: 1/GDIW9HX9A2JKTMN5: discarding: duplicate message, id: 46560815, message-id: 2069438152.36087.1643717098177.JavaMail.tomcat

  6. Adiyal Ketan reporter

    Is there is any way to force Piler to accept duplicated messages and skip message ID checking ?

  7. Janos SUTO repo owner

    Something is wrong. How is it possible that 3 piler processes are trying to process the very same email?

    Show me “pilerconf -q number_of_worker_processes” and “ps uaxw|grep piler” outputs.

  8. Adiyal Ketan reporter

    root@njuser:~# ps -auxw | grep piler
    piler 1826 0.0 0.0 39624 1456 ? Ss 2021 3:12 /usr/sbin/piler -d
    piler 7640 0.0 0.0 39624 1572 ? Ss 2021 1:10 /usr/sbin/piler -d
    piler 9051 0.0 0.0 25036 508 ? S Jan12 0:00 searchd --config /etc/piler/sphinx.conf
    piler 9052 0.1 16.6 2151404 667004 ? Sl Jan12 51:32 searchd --config /etc/piler/sphinx.conf
    piler 21880 0.0 0.0 39624 1360 ? Ss 2021 3:19 /usr/sbin/piler -d
    piler 30983 0.5 0.5 59916 21960 ? D Jan31 8:48 /usr/sbin/piler -d
    piler 32540 0.4 0.4 54744 16804 ? D Jan31 8:18 /usr/sbin/piler -d
    piler 40793 0.1 8.0 361732 322612 ? S 2021 210:56 /usr/sbin/piler -d
    piler 42834 0.1 7.9 359176 320288 ? S 2021 209:46 /usr/sbin/piler -d
    piler 56997 0.2 0.6 62376 24552 ? D Jan28 14:54 /usr/sbin/piler -d
    piler 57139 0.0 0.2 47128 8964 ? Ss 2021 1:13 /usr/sbin/piler -d
    piler 59920 0.2 0.5 58268 20444 ? D Jan28 15:15 /usr/sbin/piler -d
    piler 72851 0.4 0.7 67524 29672 ? D Jan30 12:28 /usr/sbin/piler -d
    piler 94180 0.0 0.2 47596 9512 ? Ss 2021 3:26 /usr/sbin/piler -d
    piler 98991 0.4 0.7 68280 30452 ? D Jan30 11:20 /usr/sbin/piler -d
    piler 100467 2.7 0.2 107580 8532 ? Ds 17:56 0:08 /usr/sbin/piler-smtp -d
    piler 100471 0.0 0.2 47148 8164 ? Ss 17:56 0:00 /usr/sbin/piler -d
    piler 100472 1.2 0.4 55824 18476 ? D 17:56 0:04 /usr/sbin/piler -d
    piler 100473 1.0 0.3 47936 12404 ? D 17:56 0:03 /usr/sbin/piler -d
    root 101638 0.0 0.0 13144 1064 pts/0 S+ 18:01 0:00 grep piler
    piler 111897 0.6 0.9 74764 36824 ? D Jan24 75:08 /usr/sbin/piler -d
    piler 115751 0.0 0.0 39624 1672 ? Ss Jan18 0:17 /usr/sbin/piler -d
    piler 121949 0.6 0.6 63524 25584 ? D Jan24 74:01 /usr/sbin/piler -d
    piler 122889 0.6 0.9 77144 39204 ? R Jan24 75:56 /usr/sbin/piler -d
    piler 126765 0.6 0.7 66192 28252 ? D Jan24 74:51 /usr/sbin/piler -d
    root@njuser:~# pilerconf -q number_of_worker_processes”

  11. Janos SUTO repo owner

    Great! Now there should be 3 /usr/bin/piler processes running. However, as you can see, there are a lot more, so piler has been started multiple times which explains why several processes try to process the very same message. Find what starts piler, and be sure it’s started once only. Then your problems will be solved. For the next release I’ll add a safety check that piler can’t be started multiple times in parallel.

  12. Adiyal Ketan reporter

    Thanks for your support.

    after kill all piler process now all are message are stored and not showing discard message right now.

    but i haven’t idea why lot of piler process are start.

    ‌

  14. Adiyal Ketan reporter

    what’s issue in environment ?

    kindly let me know what we can check and what’s solution for that.

    some time we received discarding: duplicate message, id.

    after checking with above command below output.

    ‌

    root@njuser:~# pilerconf -q number_of_worker_processes
    number_of_worker_processes=2
    root@njuser:~# tail -f /var/log/mail.log | grep "discarding: duplicate message"
    ^C
    root@njuser:~# ps -aux | grep piler
    root 60732 0.0 0.0 13144 1048 pts/0 S+ 14:48 0:00 grep piler
    piler 102781 0.0 0.0 25036 500 ? S Feb01 0:00 searchd --config /etc/piler/sphinx.conf
    piler 102782 0.2 16.0 2098604 642804 ? Sl Feb01 8:40 searchd --config /etc/piler/sphinx.conf
    piler 103044 1.2 0.2 108644 8444 ? Ss Feb01 53:26 /usr/sbin/piler-smtp -d
    piler 103058 0.0 0.2 47148 9236 ? Ss Feb01 0:03 /usr/sbin/piler -d
    piler 103059 1.0 0.9 77188 38868 ? S Feb01 42:48 /usr/sbin/piler -d
    piler 103060 1.0 0.8 72868 34544 ? S Feb01 42:01 /usr/sbin/piler -d

  15. Janos SUTO repo owner

    The issue is what I already told you: you started piler multiple times. Make sure you don’t do that.

  16. Adiyal Ketan reporter

    how much piler process should run ?

    I have started piler only one time so please check above piler process list.

    root@njuser:~# ps -aux | grep piler
    root 72212 0.0 0.0 13144 1152 pts/0 S+ 17:34 0:00 grep piler
    piler 102781 0.0 0.0 25036 500 ? S Feb01 0:00 searchd --config /etc/piler/sphinx.conf
    piler 102782 0.2 16.4 2220360 658944 ? Sl Feb01 64:19 searchd --config /etc/piler/sphinx.conf
    piler 103044 1.4 0.1 107484 7632 ? Ss Feb01 427:14 /usr/sbin/piler-smtp -d
    piler 103058 0.0 0.2 47148 9232 ? Ss Feb01 0:24 /usr/sbin/piler -d
    piler 108365 0.9 1.0 81076 42764 ? S Feb15 91:15 /usr/sbin/piler -d
    piler 108388 0.9 1.0 81256 42960 ? S Feb15 90:44 /usr/sbin/piler -d
    root@njuser:~# pilerconf -q number_of_worker_processes
    number_of_worker_processes=2

    ‌

  18. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
wontfix
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!