1. Janos SUTO Avatar Janos SUTO
  2. Piler
  3. piler
  4. Issues

Unable to search GUI

Issue #1160 closed
Michael Nardone created an issue

We recently had a loss of power requiring that we restore our Database from backup, we have done this however are not seeing GUI search results.

[root@mailstore tmp]# crontab -l -u piler

PILERSTART

5,35 * * * * /usr/local/libexec/piler/indexer.delta.sh
30 2 * * * /usr/local/libexec/piler/indexer.main.sh
*/15 * * * * /usr/bin/indexer --quiet tag1 --rotate
*/15 * * * * /usr/bin/indexer --quiet note1 --rotate
30 6 * * * /usr/bin/php /usr/local/libexec/piler/generate_stats.php --webui /var/www/piler
*/5 * * * * /usr/bin/find /var/www/piler/tmp -type f -name i.* -exec rm -f {} ;

PILEREND

[root@mailstore tmp]# ls -la /var/piler/sphinx
total 38560824
drwxr-xr-x. 2 piler piler 4096 Mar 25 08:45 .
drwx------. 9 piler piler 4096 Mar 24 15:18 ..
-rw-r--r-- 1 piler piler 19800 Mar 25 08:35 dailydelta1.spa
-rw-r--r-- 1 piler piler 973065 Mar 25 08:35 dailydelta1.spd
-rw-r--r-- 1 piler piler 2000 Mar 25 08:35 dailydelta1.spe
-rw-r--r-- 1 piler piler 622 Mar 25 08:35 dailydelta1.sph
-rw-r--r-- 1 piler piler 269049 Mar 25 08:35 dailydelta1.spi
-rw-r--r-- 1 piler piler 0 Mar 25 08:35 dailydelta1.spk
-rw------- 1 piler piler 0 Mar 25 08:35 dailydelta1.spl
-rw-r--r-- 1 piler piler 0 Mar 25 08:35 dailydelta1.spm
-rw-r--r-- 1 piler piler 1172119 Mar 25 08:35 dailydelta1.spp
-rw-r--r-- 1 piler piler 1 Mar 25 08:35 dailydelta1.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 dailydelta1.tmps
-rw-r--r-- 1 piler piler 1 Mar 25 02:48 dailydelta1.tmp.tmps
-rw-r--r-- 1 piler piler 5800 Mar 25 08:35 delta1.spa
-rw-r--r-- 1 piler piler 233529 Mar 25 08:35 delta1.spd
-rw-r--r-- 1 piler piler 162 Mar 25 08:35 delta1.spe
-rw-r--r-- 1 piler piler 622 Mar 25 08:35 delta1.sph
-rw-r--r-- 1 piler piler 88898 Mar 25 08:35 delta1.spi
-rw-r--r-- 1 piler piler 47525488 Mar 25 08:35 delta1.spk
-rw------- 1 piler piler 0 Mar 25 08:35 delta1.spl
-rw-r--r-- 1 piler piler 0 Mar 25 08:35 delta1.spm
-rw-r--r-- 1 piler piler 237195 Mar 25 08:35 delta1.spp
-rw-r--r-- 1 piler piler 1 Mar 25 08:35 delta1.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 delta1.tmps
-rw-r--r-- 1 piler piler 184916920 Feb 17 02:30 main1.spa
-rw-r--r-- 1 piler piler 7384393811 Feb 17 08:45 main1.spd
-rw-r--r-- 1 piler piler 63929739 Feb 17 08:45 main1.spe
-rw-r--r-- 1 piler piler 622 Feb 17 08:45 main1.sph
-rw-r--r-- 1 piler piler 257381422 Feb 17 08:45 main1.spi
-rw-r--r-- 1 piler piler 0 Feb 17 08:44 main1.spk
-rw------- 1 piler piler 0 Mar 25 02:30 main1.spl
-rw-r--r-- 1 piler piler 0 Feb 17 02:30 main1.spm
-rw-r--r-- 1 piler piler 9499788227 Feb 17 08:45 main1.spp
-rw-r--r-- 1 piler piler 1 Feb 17 08:45 main1.sps
-rw-r--r-- 1 piler piler 336326579 Mar 25 02:45 main1.tmp8.spi
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main1.tmps
-rw-r--r-- 1 piler piler 184949960 Mar 25 02:30 main1.tmp.spa
-rw-r--r-- 1 piler piler 4756340736 Mar 25 02:48 main1.tmp.spd
-rw-r--r-- 1 piler piler 40370176 Mar 25 02:48 main1.tmp.spe
-rw-r--r-- 1 piler piler 0 Mar 25 02:30 main1.tmp.spi
-rw-r--r-- 1 piler piler 0 Mar 25 02:30 main1.tmp.spm
-rw-r--r-- 1 piler piler 16726884157 Mar 25 02:47 main1.tmp.spp
-rw-r--r-- 1 piler piler 0 Mar 25 02:30 main1.tmp.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main1.tmp.tmps
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main2.spa
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main2.spd
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main2.spe
-rw-r--r-- 1 piler piler 622 Jul 2 2015 main2.sph
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main2.spi
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main2.spk
-rw------- 1 piler piler 0 Mar 19 09:22 main2.spl
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main2.spm
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main2.spp
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main2.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main2.tmps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main2.tmp.tmps
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main3.spa
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main3.spd
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main3.spe
-rw-r--r-- 1 piler piler 622 Jul 2 2015 main3.sph
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main3.spi
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main3.spk
-rw------- 1 piler piler 0 Mar 19 09:22 main3.spl
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main3.spm
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main3.spp
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main3.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main3.tmps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main3.tmp.tmps
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main4.spa
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main4.spd
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main4.spe
-rw-r--r-- 1 piler piler 622 Jul 2 2015 main4.sph
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main4.spi
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main4.spk
-rw------- 1 piler piler 0 Mar 19 09:22 main4.spl
-rw-r--r-- 1 piler piler 0 Jul 2 2015 main4.spm
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main4.spp
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main4.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main4.tmps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 main4.tmp.tmps
-rw-r--r-- 1 piler piler 0 Mar 25 08:45 note1.spa
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 note1.spd
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 note1.spe
-rw-r--r-- 1 piler piler 330 Mar 25 08:45 note1.sph
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 note1.spi
-rw-r--r-- 1 piler piler 0 Mar 25 08:45 note1.spk
-rw------- 1 piler piler 0 Mar 25 08:45 note1.spl
-rw-r--r-- 1 piler piler 0 Mar 25 08:45 note1.spm
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 note1.spp
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 note1.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 note1.tmps
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 note1.tmp.tmps
-rw-r--r-- 1 piler piler 0 Mar 25 08:45 tag1.spa
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 tag1.spd
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 tag1.spe
-rw-r--r-- 1 piler piler 329 Mar 25 08:45 tag1.sph
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 tag1.spi
-rw-r--r-- 1 piler piler 0 Mar 25 08:45 tag1.spk
-rw------- 1 piler piler 0 Mar 25 08:45 tag1.spl
-rw-r--r-- 1 piler piler 0 Mar 25 08:45 tag1.spm
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 tag1.spp
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 tag1.sps
-rw-r--r-- 1 piler piler 1 Jul 2 2015 tag1.tmps
-rw-r--r-- 1 piler piler 1 Mar 25 08:45 tag1.tmp.tmps

Understand Spinx files are much larger than they should be, was hoping to just get it back online for now.

The indexer attempted to run last night and after it’s run we’re now seeing EXT4 corruption, wondering if this is due to some part of the indexer script timing out or hitting a memory limit?

Mar 24 06:30:02 mailstore php: processed 2021.03.23 to 2021.03.24: 0 records deleted, 0 records added
Mar 25 02:30:13 mailstore kernel: EXT4-fs error (device sdb1): ext4_mb_generate_buddy: EXT4-fs: group 6734: 15723 blocks in bitmap, 9579 in gd
Mar 25 02:30:15 mailstore kernel: JBD: Spotted dirty metadata buffer (dev = sdb1, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
Mar 25 02:30:15 mailstore kernel: EXT4-fs error (device sdb1): mb_free_blocks: double-free of inode 0's block 220690900(bit 31188 in group 6734)
Mar 25 02:30:15 mailstore kernel: EXT4-fs error (device sdb1): mb_free_blocks: double-free of inode 0's block 220690901(bit 31189 in group 6734)
Mar 25 02:30:15 mailstore kernel: EXT4-fs error (device sdb1): mb_free_blocks: double-free of inode 0's block 220690902(bit 31190 in group 6734)
Mar 25 02:30:15 mailstore kernel: EXT4-fs error (device sdb1): mb_free_blocks: double-free of inode 0's block 220690903(bit 31191 in group 6734)

I have recently changed retention to 365 days (It was previously multiple years) so we’re hoping to significantly cut down the overall DB size going forward.

Happy to split the indexes if that’s the best way forward however was hoping for some direction as we have already had to restore from backup multiple times due to issues we encountered previously.

Comments (4)

  1. Janos SUTO repo owner

    For starters, I’d check for filesystem errors, and be sure all of them are fixed or cleared.

    Disable all indexer processes in piler’s crontab, then stop and start searchd. Verify that it can read all index data at startup. Then check for index data, and the the results to me:

    \$ mysql -h 127.0.0.1 -P9306

    mysql> select * from main1;

    mysql> select * from main1 order by sent desc;

    ‌

    If all seems to be lost, you can still reindex the sphinx data, though it will take time. Perhaps it’s easier to get the last backup of the sphinx data, and try to restore based on the results of the above commands.

  2. Michael Nardone reporter

    Looks good from the results although the unix timestamp shows last email was February 16th.

    [root@mailstore piler]# /etc/init.d/rc.searchd stop
    stopping searchd
    [root@mailstore piler]# /etc/init.d/rc.searchd start
    starting searchd . . .
    Sphinx 2.2.9-id64-release (rel22-r5006)
    Copyright (c) 2001-2015, Andrew Aksyonoff
    Copyright (c) 2008-2015, Sphinx Technologies Inc (http://sphinxsearch.com))

    using config file '/usr/local/etc/piler/sphinx.conf'...
    listening on 127.0.0.1:9312
    listening on 127.0.0.1:9306
    precaching index 'main1'
    precaching index 'main2'
    precaching index 'main3'
    precaching index 'main4'
    precaching index 'dailydelta1'
    precaching index 'delta1'
    precaching index 'tag1'
    precaching index 'note1'
    precached 8 indexes in 8.784 sec

    MySQL [(none)]> select * from main1;
    +------+------------+------+-------------+
    | id | sent | size | attachments |
    +------+------------+------+-------------+
    | 1 | 1433403577 | 0 | 1 |
    | 2 | 1433404390 | 0 | 1 |
    | 3 | 1433404419 | 0 | 0 |
    | 4 | 1433404447 | 0 | 1 |
    | 5 | 1433401480 | 0 | 0 |
    | 6 | 1433404524 | 0 | 1 |
    | 7 | 1433404501 | 0 | 1 |
    | 8 | 1433404502 | 0 | 1 |
    | 9 | 1433404541 | 0 | 1 |
    | 10 | 1433404557 | 0 | 0 |
    | 11 | 1433404752 | 0 | 0 |
    | 12 | 1433404678 | 0 | 4 |
    | 13 | 1433404925 | 0 | 0 |
    | 14 | 1433404988 | 0 | 0 |
    | 15 | 1433405012 | 0 | 6 |
    | 16 | 1433405053 | 0 | 0 |
    | 17 | 1433405105 | 0 | 2 |
    | 18 | 1433405106 | 0 | 2 |
    | 19 | 1433405108 | 0 | 0 |
    | 20 | 1433402227 | 0 | 0 |
    +------+------------+------+-------------+
    20 rows in set (0.37 sec)

    MySQL [(none)]> select * from main1 order by sent desc;
    +---------+------------+--------+-------------+
    | id | sent | size | attachments |
    +---------+------------+--------+-------------+
    | 9181761 | 1613489399 | 25817 | 0 |
    | 9181760 | 1613488971 | 42353 | 0 |
    | 9181759 | 1613488965 | 34558 | 1 |
    | 9181758 | 1613488957 | 34901 | 1 |
    | 9181749 | 1613488956 | 34571 | 1 |
    | 9181750 | 1613488956 | 34545 | 1 |
    | 9181751 | 1613488956 | 34686 | 1 |
    | 9181752 | 1613488956 | 34454 | 1 |
    | 9181753 | 1613488956 | 34637 | 1 |
    | 9181754 | 1613488956 | 34593 | 1 |
    | 9181755 | 1613488956 | 34453 | 1 |
    | 9181756 | 1613488956 | 34451 | 1 |
    | 9181757 | 1613488956 | 34643 | 1 |
    | 9181748 | 1613488569 | 14556 | 0 |
    | 9181747 | 1613488424 | 88437 | 0 |
    | 9181746 | 1613488176 | 84172 | 0 |
    | 9181745 | 1613487817 | 645271 | 1 |
    | 9181744 | 1613487542 | 93020 | 8 |
    | 9181743 | 1613487541 | 27844 | 0 |
    | 9181742 | 1613487540 | 27812 | 0 |
    +---------+------------+--------+-------------+
    20 rows in set (3.91 sec)

  3. Janos SUTO repo owner

    OK, then reindex the rest, eg. reindex -f 9181761 -t xxxxxxxxxx. You can get the latter value from the piler mysql database metadata table, look at the id column.

  5. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
closed
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!