Reindexer has problems in digest_string / openssl3 ?

[Fabian]

Hello,

we are running Version 1.4.3 on Ubuntu 22.04.02 and wanted to reindex everything.

Want to do this, to upgrade from sphinx to manticore.

Regardless of how we start reindex, it crashes here:

Starting program: /usr/local/bin/reindex -f 100000 -t 100000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f47dae47cd6 in digest_string (digestname=digestname@entry=0x7f47dae514e0 "sha256", 
    s=s@entry=0x7fff05143f60 "<[e-mail]>", digest=digest@entry=0x7fff05144160 "") at digest.c:170
#2  0x00007f47dae3d466 in post_parse (sdata=sdata@entry=0x7fff05176ee0, state=state@entry=0x7fff05143cd0, cfg=cfg@entry=0x7fff05175310) at parser.c:176
#3  0x000055f7b711c012 in retrieve_email_by_metadata_id (sdata=sdata@entry=0x7fff05176ee0, data=data@entry=0x7fff05173f60, from_id=from_id@entry=100000, 
    to_id=to_id@entry=100000, cfg=cfg@entry=0x7fff05175310) at reindex.c:131
#4  0x000055f7b711b859 in main (argc=<optimized out>, argv=<optimized out>) at reindex.c:271

Thanks for your work!

Fabian

[Janos SUTO]

I couldn’t reproduce the problem, it works as expected, see output below (note that I already use manticore realtime index in my dev environment). Please provide more context how to reproduce this issue.

root@42964a616bc0:/var/piler/imap# reindex -f 2000 -t 2000
put 1 messages to sph_index table for reindexing
root@42964a616bc0:/var/piler/imap# reindex -f 2500 -t 2600
put 101 messages to sph_index table for reindexing

‌

[Fabian]

Hi,

we had to migrate the server to an new one, we copied all files/direcotry from /var/piler to the new one. Also we dumped the database and import it to the new one.

Everythin is fine, now we want use manticore cause we had some issue with the old search engine.

We want to reindex all mails. So we tried the reindex command got this error.

‌

What informations do you need?

‌

[Janos SUTO]

Try to compile piler on the new host, and let’s see it again.

[Fabian]

Hi,

‌

i solved the problem.

‌

I add these lines to the openssl config:

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

Its activate the legacy settings for openssl.

‌

Do you know, why this is necessary?

‌

[dietrichha]

Hey,

i just saw a commit for legacy openssl 3 support (https://bitbucket.org/jsuto/piler/commits/949792ef9a1f1b755b0531bb5ed0d81ae5f0e088) and this section in the openssl 3 config:

# If no providers are activated explicitly, the default one is activated implicitly.# See man 7 OSSL_PROVIDER-default for more details.## If you add a section explicitly activating any other provider(s), you most# probably need to explicitly activate the default provider, otherwise it# becomes unavailable in openssl. As a consequence applications depending on# OpenSSL may not work correctly which could lead to significant system# problems including inability to remotely access the system.

Could it be, that somehow the legacy support is added first. Therefore no default support is active anymore, therefore no sha256 support.

But if you would load both, it would be ok. Like in my pseudocode:

#if OPENSSL_VERSION_MAJOR >= 3

OSSL_PROVIDER_load(NULL, "default");
OSSL_PROVIDER_load(NULL, "legacy");
#endif

Just an idea….

Regards

‌

[Janos SUTO]

I don’t think you lose sha256 support, because the code uses exactly sha256, and it works.

[Janos SUTO]

Fabian, please apply the patch mentioned above, revert the openssl config (save the config file just in case) and it should work.

[Fabian]

Hi,

small update:

reverted to default openssl.cnf.

Got the error as before:

‌

(gdb) run -f 1 -t 1
Starting program: /usr/local/bin/reindex -f 1 -t 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f9fb7850cd6 in digest_string (digestname=digestname@entry=0x7f9fb785a4e0 "sha256", s=s@entry=0x7ffff74df770 "[e-mail]", digest=digest@entry=0x7ffff74df970 "") at digest.c:170
#2  0x00007f9fb7846466 in post_parse (sdata=sdata@entry=0x7ffff75126f0, state=state@entry=0x7ffff74df4e0, cfg=cfg@entry=0x7ffff7510b20) at parser.c:176
#3  0x000055dc9d7bf012 in retrieve_email_by_metadata_id (sdata=sdata@entry=0x7ffff75126f0, data=data@entry=0x7ffff750f770, from_id=from_id@entry=1, to_id=to_id@entry=1, cfg=cfg@entry=0x7ffff7510b20) at reindex.c:131
#4  0x000055dc9d7be859 in main (argc=<optimized out>, argv=<optimized out>) at reindex.c:271

After that changed archive.c to:

#if OPENSSL_VERSION_MAJOR >= 3
OSSL_PROVIDER_load(NULL, "legacy");
OSSL_PROVIDER_load(NULL, "default");
#endif

‌

‌

Recomplied it.

It works now!!! The post before, i forgot an change… sorry!

‌

‌

[dietrichha]

Then openssl really unloads the “default” algos…

[Janos SUTO]

I'm glad that it works by now.

[dietrichha]

But you had to recomplie it with this change to work:

‌

#if OPENSSL_VERSION_MAJOR >= 3
OSSL_PROVIDER_load(NULL, "legacy");
OSSL_PROVIDER_load(NULL, "default");
#endif

‌

right?

‌

‌

[Fabian]

yes i had to add

#if OPENSSL_VERSION_MAJOR >= 3
OSSL_PROVIDER_load(NULL, "legacy");
OSSL_PROVIDER_load(NULL, "default");
#endif

in archive.c to get rid of the error.