1. Janos SUTO Avatar Janos SUTO
  2. Piler
  3. piler
  4. Issues

can't see emails from a date forward

Issue #1230 wontfix
Martin Manzanelli created an issue

hi, we have a piler running on docker.

we cant see message on de search tool since a date to now. its a case similar to that https://bitbucket.org/jsuto/piler/issues/950/not-able-to-find-mails-in-piler-webui.

can you help me to solve this issue?

Comments (9)

  2. Martin Manzanelli reporter

    Hi Janos.

    I culdn't see mails in search result since a power outage. But in the health page i saw messages stil arriving.
    Yesterday i follow the steps at those post https://bitbucket.org/jsuto/piler/issues/269/reindex-speed https://bitbucket.org/jsuto/piler/issues/1030/reindex-does-not-reindex-all-mails
    and the messages started showing at search page, with this i can understand the first problem was solved.

    Now i have a question. It's normal to see 87gb sph_index table? im having aboutn 8.000.000 of mails. The query select count(*) from sph_index; never ends.

    ‌

    wait your feedback.

    ‌

    thnks.

    ‌

  3. Martin Manzanelli reporter

    I can see that on tail /var/log/mail.log

    Feb 11 15:37:30 archivomail piler: INDEXER ERROR: delta indexing and merging is already running. It started at Fri Sep 3 15:35:07 CEST 2021
    Feb 11 15:37:37 archivomail piler: INDEXER INFO: merging to main started
    Feb 11 15:37:37 archivomail piler: INDEXER INFO: merging to main finished
    Feb 11 15:37:42 archivomail piler: INDEXER INFO: resetting daily delta started
    Feb 11 15:37:42 archivomail piler: INDEXER INFO: resetting daily delta finished

    ‌

    this is correct “It started at Fri Sep 3”? i have restarted the docker today

    ‌

    other info to add:

    the query “select count(*) from sph_index;” finished after 43minutes, ther result is 7979099

  4. Janos SUTO repo owner

    The sph_index table should up to 30 minutes of emails. Then the delta indexer should process the table, and empty it. So having ~8 million messages in the sph_index table is far from normal. The power outage may have introduced some corruption either in mysql or sphinx data. So try the following:

    • check if my mysql has enough resources allocated, especially the innodb variables. See the FAQ for a starting point. You may need much more based on email volume
    • either drop the sph_index table and recreate it or delete its contents
    • kill the indexer process running since last September
    • verify that new emails' contents are written to sph_index table, then check if the delta indexer is able to process those emails in a timely manner
    • check whether your current main1.* data files are under 10 GBs or so. If they are above, then it’s worth to start using main2 instead

    ‌

  5. Martin Manzanelli reporter

    Hi Janos

    i have been follow the steps below

    • check if my mysql has enough resources allocated, especially the innodb variables. See the FAQ for a starting point. You may need much more based on email volume

    i have tuned mysql innodb parameters.

    ‌

    • either drop the sph_index table and recreate it or delete its contents

    i have trucated table sph_index, now i have only new, after truncate.

    ‌

    • kill the indexer process running since last September

    ther are no index proccess running , may be its a flag some where?? can you help me to finding it?

    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    root 1 0.0 0.0 18124 2992 ? Ss 17:15 0:00 /bin/bash /usr/local/bin/start.sh
    root 41 0.0 0.0 180484 2056 ? Ssl 17:15 0:01 /usr/sbin/rsyslogd
    root 62 0.0 0.1 355104 14356 ? Ss 17:15 0:00 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
    www-data 63 0.0 0.1 357916 20172 ? S 17:15 0:00 php-fpm: pool www
    www-data 64 0.0 0.1 357916 20276 ? S 17:15 0:00 php-fpm: pool www
    piler 72 0.0 0.0 36340 4088 ? S 17:15 0:00 searchd --config /etc/piler/sphinx.conf
    piler 73 0.2 8.4 1030556 871808 ? Sl 17:15 0:04 searchd --config /etc/piler/sphinx.conf
    piler 81 0.0 0.0 52144 1116 ? Ss 17:15 0:00 /usr/sbin/piler -d
    piler 83 1.3 0.1 67820 18624 ? S 17:15 0:27 /usr/sbin/piler -d
    piler 84 1.5 0.1 65656 17124 ? S 17:15 0:31 /usr/sbin/piler -d
    piler 85 2.6 0.0 50536 5620 ? Ss 17:15 0:52 /usr/sbin/piler-smtp -d
    fetchma+ 107 0.0 0.0 44560 3056 ? Ss 17:15 0:00 /usr/bin/fetchmail -f /etc/fetchmailrc --pidfile /var/run/fetchmail/fetchmail.pid --syslog
    root 114 0.0 0.0 27996 2264 ? Ss 17:15 0:00 /usr/sbin/cron
    root 115 0.0 0.0 4236 740 ? S 17:15 0:00 tail -f /dev/null
    piler 204 0.0 0.0 18228 3272 pts/0 Ss+ 17:15 0:00 bash
    root 276 0.0 0.0 18252 3376 pts/1 Ss 17:16 0:00 bash
    root 1236 0.0 0.0 36640 2824 pts/1 R+ 17:48 0:00 ps -aux

    ‌

    • verify that new emails' contents are written to sph_index table, then check if the delta indexer is able to process those emails in a timely manne

    new mails are writed ok. the error message about de running proccess still apearing.
    "INDEXER ERROR: delta indexing and merging is already running. It started at Fri Sep 3 15:35:07 CEST 2021"

    ‌

    • check whether your current main1.* data files are under 10 GBs or so. If they are above, then it’s worth to start using main2 instead

    i haven’t got large main1.* files becouse i have never run a sucsess reindex.daleta.sh + reindes.main.sh after cleanup of /var/piler/sphinx (following this post https://bitbucket.org/jsuto/piler/issues/269/reindex-speed))

    ‌

    but i have got delta1.spd near 10GB . I paste an ls of /var/piler/sphinx

    -rw-r--r-- 1 piler piler 129M Feb 11 00:44 att1.spa
    -rw-r--r-- 1 piler piler 257M Feb 11 00:44 att1.spd
    -rw-r--r-- 1 piler piler 2.0M Feb 11 00:44 att1.spe
    -rw-r--r-- 1 piler piler 395 Feb 11 00:44 att1.sph
    -rw-r--r-- 1 piler piler 10M Feb 11 00:44 att1.spi
    -rw-r--r-- 1 piler piler 0 Feb 11 00:44 att1.spk
    -rw------- 1 piler piler 0 Feb 14 17:15 att1.spl
    -rw-r--r-- 1 piler piler 0 Feb 11 00:44 att1.spm
    -rw-r--r-- 1 piler piler 8.6M Feb 11 00:44 att1.spp
    -rw-r--r-- 1 piler piler 1 Feb 11 00:44 att1.sps
    -rw-r--r-- 1 piler piler 0 Feb 14 02:30 dailydelta1.spa
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 dailydelta1.spd
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 dailydelta1.spe
    -rw-r--r-- 1 piler piler 637 Feb 14 02:30 dailydelta1.sph
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 dailydelta1.spi
    -rw-r--r-- 1 piler piler 0 Feb 14 02:30 dailydelta1.spk
    -rw------- 1 piler piler 0 Feb 14 17:15 dailydelta1.spl
    -rw-r--r-- 1 piler piler 0 Feb 14 02:30 dailydelta1.spm
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 dailydelta1.spp
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 dailydelta1.sps
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 dailydelta1.tmp.tmps
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 dailydelta1.tmps
    -rw-r--r-- 1 piler piler 155M Feb 10 20:40 delta1.spa
    -rw-r--r-- 1 piler piler 9.7G Feb 10 22:36 delta1.spd
    -rw-r--r-- 1 piler piler 76M Feb 11 00:36 delta1.spe
    -rw-r--r-- 1 piler piler 637 Feb 10 22:36 delta1.sph
    -rw-r--r-- 1 piler piler 434M Feb 10 22:36 delta1.spi
    -rw-r--r-- 1 piler piler 0 Feb 10 20:40 delta1.spk
    -rw------- 1 piler piler 0 Feb 14 17:15 delta1.spl
    -rw-r--r-- 1 piler piler 0 Feb 10 20:40 delta1.spm
    -rw-r--r-- 1 piler piler 16G Feb 10 22:36 delta1.spp
    -rw-r--r-- 1 piler piler 1 Feb 11 00:36 delta1.sps
    -rw-r--r-- 1 piler piler 0 Feb 14 02:30 main1.spa
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 main1.spd
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 main1.spe
    -rw-r--r-- 1 piler piler 637 Feb 14 02:30 main1.sph
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 main1.spi
    -rw-r--r-- 1 piler piler 0 Feb 14 02:30 main1.spk
    -rw------- 1 piler piler 0 Feb 14 17:15 main1.spl
    -rw-r--r-- 1 piler piler 0 Feb 14 02:30 main1.spm
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 main1.spp
    -rw-r--r-- 1 piler piler 1 Feb 14 02:30 main1.sps
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main1.tmps
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main2.spa
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main2.spd
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main2.spe
    -rw-r--r-- 1 piler piler 637 Feb 10 19:26 main2.sph
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main2.spi
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main2.spk
    -rw------- 1 piler piler 0 Feb 14 17:15 main2.spl
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main2.spm
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main2.spp
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main2.sps
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main2.tmps
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main3.spa
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main3.spd
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main3.spe
    -rw-r--r-- 1 piler piler 637 Feb 10 19:26 main3.sph
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main3.spi
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main3.spk
    -rw------- 1 piler piler 0 Feb 14 17:15 main3.spl
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main3.spm
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main3.spp
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main3.sps
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main3.tmps
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main4.spa
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main4.spd
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main4.spe
    -rw-r--r-- 1 piler piler 637 Feb 10 19:26 main4.sph
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main4.spi
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main4.spk
    -rw------- 1 piler piler 0 Feb 14 17:15 main4.spl
    -rw-r--r-- 1 piler piler 0 Feb 10 19:26 main4.spm
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main4.spp
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main4.sps
    -rw-r--r-- 1 piler piler 1 Feb 10 19:26 main4.tmps
    -rw-r--r-- 1 piler piler 0 Feb 14 17:30 note1.spa
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 note1.spd
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 note1.spe
    -rw-r--r-- 1 piler piler 345 Feb 14 17:30 note1.sph
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 note1.spi
    -rw-r--r-- 1 piler piler 0 Feb 14 17:30 note1.spk
    -rw------- 1 piler piler 0 Feb 14 17:30 note1.spl
    -rw-r--r-- 1 piler piler 0 Feb 14 17:30 note1.spm
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 note1.spp
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 note1.sps
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 note1.tmp.tmps
    -rw-r--r-- 1 piler piler 1 Feb 11 00:36 note1.tmps
    -rw-r--r-- 1 piler piler 0 Feb 14 17:30 tag1.spa
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 tag1.spd
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 tag1.spe
    -rw-r--r-- 1 piler piler 344 Feb 14 17:30 tag1.sph
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 tag1.spi
    -rw-r--r-- 1 piler piler 0 Feb 14 17:30 tag1.spk
    -rw------- 1 piler piler 0 Feb 14 17:30 tag1.spl
    -rw-r--r-- 1 piler piler 0 Feb 14 17:30 tag1.spm
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 tag1.spp
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 tag1.sps
    -rw-r--r-- 1 piler piler 1 Feb 14 17:30 tag1.tmp.tmps
    -rw-r--r-- 1 piler piler 1 Feb 11 00:36 tag1.tmps

    ‌

    now i think the next step is find the gost proccess running. after that run 

    reindex -f i -t i+x -p
/usr/libexec/piler/indexer.delta.sh
/usr/libexec/piler/indexer.main.sh

    I wait your help to find this running procces and guide with the nexts steps.

    ‌

    Thanks

  6. Janos SUTO repo owner

    I’d like to see this host. If ssh access is find, then please send me the connection details. See piler -V for my email.

  7. Martin Manzanelli reporter

    hi Janos i cant share a direct ssh access. may be a webex meet and shared control, its ok??

  8. Janos SUTO repo owner

    If webex shared control worked in a browser, then it would be ok. If not, then good luck fixing the issue.

  10. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
wontfix
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!