Old archive can't be reindexed with newer OpenSSL library

[ijk]

Hi,

I'm sorry for the gigantic wall of text, but here it is non the less:

As the title says, there appears to be a incompatibility with old archived emails and new(er) OpenSSL libraries.
I've got a 5 year old archive running on an old Debian 10 server which I recently migrated to Debian 12.
While doing so, I've also switched to Manticore, so re-indexing was in order, but what I got was a segmentation fault.

This is what gdb had to say:

Reading symbols from /usr/local/bin/reindex...
(gdb) run -c /usr/local/etc/piler/piler.conf -f 30543 -t 30543
Starting program: /usr/local/bin/reindex -c /usr/local/etc/piler/piler.conf -f 30543 -t 30543
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f6a640204ae in raw_digest_file (digestname=digestname@entry=0x7f6a64029660 "sha256", filename=filename@entry=0x7ffe30d13418 "400000005f635651133336ec00d34925f194.a1", md_value=md_value@entry=0x7ffe30ce1230 "") at digest.c:131
#2  0x00007f6a6402054e in digest_file (filename=filename@entry=0x7ffe30d13418 "400000005f635651133336ec00d34925f194.a1", digest=digest@entry=0x7ffe30d13498 "") at digest.c:144
#3  0x00007f6a64016baf in post_parse (sdata=sdata@entry=0x7ffe30d44cb0, state=state@entry=0x7ffe30d0fe60, cfg=cfg@entry=0x7ffe30d3fce0) at parser.c:151
#4  0x000055ea6ece1bf3 in retrieve_email_by_metadata_id (sdata=sdata@entry=0x7ffe30d44cb0, data=data@entry=0x7ffe30d418b0, from_id=from_id@entry=30543, to_id=to_id@entry=30543, cfg=cfg@entry=0x7ffe30d3fce0) at reindex.c:131
#5  0x000055ea6ece157b in main (argc=<optimized out>, argv=<optimized out>) at reindex.c:271

That was version 1.4.4 and since 0x0000000000000000 did not seem helpful, I've also tried downgrading to 1.4.1:

Reading symbols from /usr/local/bin/reindex...
(gdb) run -c /usr/local/etc/piler/piler.conf -f 30543 -t 30543
Starting program: /usr/local/bin/reindex -c /usr/local/etc/piler/piler.conf -f 30543 -t 30543
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007febfa8fc040 in EVP_CIPHER_get_block_size () from /lib/x86_64-linux-gnu/libcrypto.so.3
(gdb) bt
#0  0x00007febfa8fc040 in EVP_CIPHER_get_block_size () from /lib/x86_64-linux-gnu/libcrypto.so.3
#1  0x00007febfabab015 in retrieve_file_from_archive (filename=filename@entry=0x7fff04f2c4c0 "/var/piler/store/00/5f6/f1/94/400000005f635651133336ec00d34925f194.m", mode=mode@entry=1, buffer=buffer@entry=0x7fff04f2c4b8,
    dest=dest@entry=0x55d218e5d2a0, cfg=cfg@entry=0x7fff04f8b240) at archive.c:188
#2  0x00007febfabab328 in retrieve_email_from_archive (sdata=sdata@entry=0x7fff04f90010, dest=dest@entry=0x55d218e5d2a0, cfg=cfg@entry=0x7fff04f8b240) at archive.c:291
#3  0x000055d2172faa99 in retrieve_email_by_metadata_id (sdata=sdata@entry=0x7fff04f90010, data=data@entry=0x7fff04f8cc10, from_id=from_id@entry=30543, to_id=to_id@entry=30543, cfg=cfg@entry=0x7fff04f8b240) at reindex.c:113
#4  0x000055d2172fa54d in main (argc=<optimized out>, argv=<optimized out>) at reindex.c:266

Much better output, at least to me.

After spending entirely too much time trying to find a workaround I installed Debian 11 on a hunch, as that is still on OpenSSL 1.1.1w.
Re-indexing went fine, no segfault.

While the re-index was underway, I've spun up a local Ubuntu 22.04.3 virtual machine and installed the piler_1.4.4-jammy-553ebb4f_amd64.deb from the download page.
Ubuntu 22.04.3 is on OpenSSL 3.0.2 and equally as broken as Debian 12 with OpenSSL 3.0.11.

Is there any way to fix this properly?
I'd really like to upgrade to something less ancient if possible.

Thanks.

[Janos SUTO]

Hello, I think the solution is to enable some sort compatibility mode in openssl’s config. At least I remember a similar case from the past. I’ll try to find it.

Here it is: https://bitbucket.org/jsuto/piler/issues/1284/reindexer-has-problems-in-digest_string

Let me know, how it goes.

[ijk]

I can’t believe I’ve missed that particular issue...

Anyway, I can confirm, that last comment was right on the money, adding that line fixes the segfault:

diff --git a/src/archive.c b/src/archive.c
index 13017dd..95c8db0 100644
--- a/src/archive.c
+++ b/src/archive.c
@@ -192,6 +192,7 @@ int retrieve_file_from_archive(char *filename, int mode, char **buffer, FILE *de
       } else {
       #if OPENSSL_VERSION_MAJOR >= 3
          OSSL_PROVIDER_load(NULL, "legacy");
+         OSSL_PROVIDER_load(NULL, "default");
       #endif
          rc = EVP_DecryptInit_ex(ctx, EVP_bf_cbc(), NULL, cfg->key, cfg->iv);
       }

Is there a particular reason as to why this is not at least in the master branch?
Seems like it'll eventually bite just about everyone with an existing archive.

[Janos SUTO]

Now it is on the master branch, see https://bitbucket.org/jsuto/piler/commits/4f948d3d5c584d8fe3c848a56f1fc0021340872b

Thank you for raising the issue again, and confirming the fix.