New docker setup - support for bind mounted dirs + separate containers for new services

[Leszek Piatek]

I’m trying to get new docker setup working - separate containers for each service, I’m “almost there”, but still have some glitches:

1. why do I need to run cron jobs of searchd indexer on searchd container? if not run there I see WARNING: process with PID X is not running and indexer is not run

2. I have troubles with getting results in webgui, I have entries in metadata:

MariaDB [piler]> select count(id) from metadata;
+-----------+
| count(id) |
+-----------+
|        95 |
+-----------+
1 row in set (0.001 sec)

But nothing at all in manticore/sphinx:

MySQL [(none)]> SELECT id FROM main1,dailydelta1,delta1 ORDER BY sent DESC;
Empty set (0.001 sec)

MySQL [(none)]> SHOW TABLES;
+-------------+-------+
| Index       | Type  |
+-------------+-------+
| dailydelta1 | local |
| delta1      | local |
| main1       | local |
| main2       | local |
| main3       | local |
| main4       | local |
| note1       | local |
| tag1        | local |
+-------------+-------+
8 rows in set (0.001 sec)

I run all indexer related jobs from cron on searchd container:

searchd_1     | WARNING: Load from config index main1 with type disk
searchd_1     | WARNING: keep existing index  main1
searchd_1     | WARNING: Keep existing before local index main1
searchd_1     | WARNING: Load from config index main2 with type disk
searchd_1     | WARNING: keep existing index  main2
searchd_1     | WARNING: Keep existing before local index main2
searchd_1     | WARNING: Load from config index main3 with type disk
searchd_1     | WARNING: keep existing index  main3
searchd_1     | WARNING: Keep existing before local index main3
searchd_1     | WARNING: Load from config index main4 with type disk
searchd_1     | WARNING: keep existing index  main4
searchd_1     | WARNING: Keep existing before local index main4
searchd_1     | WARNING: Load from config index dailydelta1 with type disk
searchd_1     | WARNING: add deferred index dailydelta1
searchd_1     | WARNING: keep existing index  dailydelta1
searchd_1     | WARNING: Keep existing before local index dailydelta1
searchd_1     | WARNING: Load from config index delta1 with type disk
searchd_1     | WARNING: add deferred index delta1
searchd_1     | WARNING: keep existing index  delta1
searchd_1     | WARNING: Keep existing before local index delta1
searchd_1     | WARNING: Load from config index tag1 with type disk
searchd_1     | WARNING: add deferred index tag1
searchd_1     | WARNING: keep existing index  tag1
searchd_1     | WARNING: Keep existing before local index tag1
searchd_1     | WARNING: Load from config index note1 with type disk
searchd_1     | WARNING: add deferred index note1
searchd_1     | WARNING: keep existing index  note1
searchd_1     | WARNING: Keep existing before local index note1
searchd_1     | WARNING: will rotate note1
searchd_1     | WARNING: will rotate tag1
searchd_1     | WARNING: will rotate delta1
searchd_1     | WARNING: will rotate dailydelta1
searchd_1     | WARNING: TaskRotation starts with 4 deferred indexes
searchd_1     | WARNING: seamless rotate local index note1
searchd_1     | rotating index 'note1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | WARNING: binlog: failed to unlink /var/piler/manticore/binlog.001: No such file or directory
searchd_1     | last message repeated 3 times
searchd_1     | rotating index 'note1': success
searchd_1     | WARNING: seamless rotate local index tag1
searchd_1     | rotating index 'tag1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'tag1': success
searchd_1     | WARNING: seamless rotate local index delta1
searchd_1     | rotating index 'delta1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'delta1': success
searchd_1     | WARNING: seamless rotate local index dailydelta1
searchd_1     | rotating index 'dailydelta1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'dailydelta1': success
searchd_1     | rotating index: all indexes done
searchd_1     | caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
searchd_1     | WARNING: keep existing index  main1
searchd_1     | WARNING: Keep existing before local index main1
searchd_1     | WARNING: keep existing index  main2
searchd_1     | WARNING: Keep existing before local index main2
searchd_1     | WARNING: keep existing index  main3
searchd_1     | WARNING: Keep existing before local index main3
searchd_1     | WARNING: keep existing index  main4
searchd_1     | WARNING: Keep existing before local index main4
searchd_1     | WARNING: keep existing index  dailydelta1
searchd_1     | WARNING: Keep existing before local index dailydelta1
searchd_1     | WARNING: keep existing index  delta1
searchd_1     | WARNING: Keep existing before local index delta1
searchd_1     | WARNING: keep existing index  tag1
searchd_1     | WARNING: Keep existing before local index tag1
searchd_1     | WARNING: add deferred index note1
searchd_1     | WARNING: keep existing index  note1
searchd_1     | WARNING: Keep existing before local index note1
searchd_1     | WARNING: will rotate note1
searchd_1     | WARNING: TaskRotation starts with 1 deferred indexes
searchd_1     | WARNING: seamless rotate local index note1
searchd_1     | rotating index 'note1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'note1': success
searchd_1     | rotating index: all indexes done
searchd_1     | caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
searchd_1     | WARNING: keep existing index  main1
searchd_1     | WARNING: Keep existing before local index main1
searchd_1     | WARNING: keep existing index  main2
searchd_1     | WARNING: Keep existing before local index main2
searchd_1     | WARNING: keep existing index  main3
searchd_1     | WARNING: Keep existing before local index main3
searchd_1     | WARNING: keep existing index  main4
searchd_1     | WARNING: Keep existing before local index main4
searchd_1     | WARNING: keep existing index  dailydelta1
searchd_1     | WARNING: Keep existing before local index dailydelta1
searchd_1     | WARNING: add deferred index delta1
searchd_1     | WARNING: keep existing index  delta1
searchd_1     | WARNING: Keep existing before local index delta1
searchd_1     | WARNING: keep existing index  tag1
searchd_1     | WARNING: Keep existing before local index tag1
searchd_1     | WARNING: keep existing index  note1
searchd_1     | WARNING: Keep existing before local index note1
searchd_1     | WARNING: will rotate delta1
searchd_1     | WARNING: TaskRotation starts with 1 deferred indexes
searchd_1     | WARNING: seamless rotate local index delta1
searchd_1     | rotating index 'delta1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'delta1': success
searchd_1     | rotating index: all indexes done
searchd_1     | caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
searchd_1     | WARNING: keep existing index  main1
searchd_1     | WARNING: Keep existing before local index main1
searchd_1     | WARNING: keep existing index  main2
searchd_1     | WARNING: Keep existing before local index main2
searchd_1     | WARNING: keep existing index  main3
searchd_1     | WARNING: Keep existing before local index main3
searchd_1     | WARNING: keep existing index  main4
searchd_1     | WARNING: Keep existing before local index main4
searchd_1     | WARNING: add deferred index dailydelta1
searchd_1     | WARNING: keep existing index  dailydelta1
searchd_1     | WARNING: Keep existing before local index dailydelta1
searchd_1     | WARNING: keep existing index  delta1
searchd_1     | WARNING: Keep existing before local index delta1
searchd_1     | WARNING: keep existing index  tag1
searchd_1     | WARNING: Keep existing before local index tag1
searchd_1     | WARNING: keep existing index  note1
searchd_1     | WARNING: Keep existing before local index note1
searchd_1     | WARNING: will rotate dailydelta1
searchd_1     | WARNING: TaskRotation starts with 1 deferred indexes
searchd_1     | WARNING: seamless rotate local index dailydelta1
searchd_1     | rotating index 'dailydelta1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'dailydelta1': success
searchd_1     | rotating index: all indexes done
searchd_1     | caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
searchd_1     | WARNING: add deferred index main1
searchd_1     | WARNING: keep existing index  main1
searchd_1     | WARNING: Keep existing before local index main1
searchd_1     | WARNING: keep existing index  main2
searchd_1     | WARNING: Keep existing before local index main2
searchd_1     | WARNING: keep existing index  main3
searchd_1     | WARNING: Keep existing before local index main3
searchd_1     | WARNING: keep existing index  main4
searchd_1     | WARNING: Keep existing before local index main4
searchd_1     | WARNING: keep existing index  dailydelta1
searchd_1     | WARNING: Keep existing before local index dailydelta1
searchd_1     | WARNING: keep existing index  delta1
searchd_1     | WARNING: Keep existing before local index delta1
searchd_1     | WARNING: keep existing index  tag1
searchd_1     | WARNING: Keep existing before local index tag1
searchd_1     | WARNING: keep existing index  note1
searchd_1     | WARNING: Keep existing before local index note1
searchd_1     | WARNING: will rotate main1
searchd_1     | WARNING: TaskRotation starts with 1 deferred indexes
searchd_1     | WARNING: seamless rotate local index main1
searchd_1     | rotating index 'main1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'main1': success
searchd_1     | rotating index: all indexes done
searchd_1     | caught SIGHUP (seamless=1, in_rotate=0, need_rotate=0)
searchd_1     | WARNING: keep existing index  main1
searchd_1     | WARNING: Keep existing before local index main1
searchd_1     | WARNING: keep existing index  main2
searchd_1     | WARNING: Keep existing before local index main2
searchd_1     | WARNING: keep existing index  main3
searchd_1     | WARNING: Keep existing before local index main3
searchd_1     | WARNING: keep existing index  main4
searchd_1     | WARNING: Keep existing before local index main4
searchd_1     | WARNING: add deferred index dailydelta1
searchd_1     | WARNING: keep existing index  dailydelta1
searchd_1     | WARNING: Keep existing before local index dailydelta1
searchd_1     | WARNING: keep existing index  delta1
searchd_1     | WARNING: Keep existing before local index delta1
searchd_1     | WARNING: keep existing index  tag1
searchd_1     | WARNING: Keep existing before local index tag1
searchd_1     | WARNING: keep existing index  note1
searchd_1     | WARNING: Keep existing before local index note1
searchd_1     | WARNING: will rotate dailydelta1
searchd_1     | WARNING: TaskRotation starts with 1 deferred indexes
searchd_1     | WARNING: seamless rotate local index dailydelta1
searchd_1     | rotating index 'dailydelta1': started
searchd_1     | RW-idx for rename to .old, acquiring...
searchd_1     | RW-idx for rename to .old, acquired...
searchd_1     | rotating index 'dailydelta1': success
searchd_1     | rotating index: all indexes done

‌

Any ideas what could be wrong with my setup?

[Leszek Piatek]

Ok, second point: reindex -ap did the trick, it must be running indexer jobs on cron container marked messages as already indexed… But still why we need to run indexer jobs on searchd container? Any suggestions how to move it to cron container?

Only way I think of to achive it is to actually use docker compose exec and cron on host machine… But this sucks

[Janos SUTO]

“why do I need to run cron jobs of searchd indexer on searchd container? if not run there I see WARNING: process with PID X is not running and indexer is not run”

Because that’s how indexed data is updated. The indexer utility reads the configured mysql table, processes its content, then updates the index data files. Anyway, this cron container makes no sense. However, piler v1.4.3 or the master branch supports real time index as well, so you don’t need any cron jobs in the searchd container. See https://www.mailpiler.org/wiki/current:manticore for more.

[Leszek Piatek]

“Anyway, this cron container makes no sense.” - I would argue, having more than one process in container makes it more difficult to manage and monitor service, more dockerish way is to have one process per service. I’ll update this ticket when I’ll have something to show and maybe You will have some time to check it.

Regarding running indexer on searchd I thought indexer communicate with searchd using tcp / socket not signals, now I understand more. Thanks for pointing me into RT direction!

[Janos SUTO]

Well, you are right. The preferred method is to have a single process in a container. However, it’s not a set in stone principal. Eg. piler itself forks some child processes, they same is true fro searchd.

[Janos SUTO]

Anyway, I set the status to on-hold. Be sure to set it to open when you have something to show and we can progress.

[Leszek Piatek]

Sure @Janos SUTO I’ve found also this line:
https://bitbucket.org/jsuto/piler/src/531250ec9b18fa76bf0484943e1792ca77cbeb14/docker/start.sh#lines-181

Prevent docker image to use RT index, indexer will fail trying to create indexes because of:

searchd_1     | DEBUG: main1.spp does not exist, creating index files
searchd_1     | Manticore 5.0.2 348514c86@220530 dev (columnar 1.15.4 2fef34e@220522) (secondary 1.15.4 2fef34e@220522)
searchd_1     | Copyright (c) 2001-2016, Andrew Aksyonoff
searchd_1     | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
searchd_1     | Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
searchd_1     | 
searchd_1     | using config file '/etc/piler/manticore.conf'...
searchd_1     | FATAL: no indexes found in config file '/etc/piler/manticore.conf'

So I guess there should be a place in Wiki where one could find instruction how tu upgrade docker image to use RT index, on the other hand only way to use old way plain indexes and update them would be to create cron entry on host not on container - do we need to support plain indexes?

[Janos SUTO]

The start script is not prepared for RT index yet. Feel free to fix it.