Rails: crash on DB access with custom OpenSSL config on Ubuntu 14.04

Issue #190 resolved
Andrey Novikov
created an issue

It may be not a pg bug, but for me it occurs only with pg gem.

I've patched ruby with patches from Ruby #9830 and OpenSSL configured for GOST algorithms usage on Ubuntu 14.04 Server x64.

App crashes on DB rake tasks invocation or when first request coming to it, yielding next message and no backtrace:

envek@ubuntu:~/TestOsslFail$ rake db:create
GOST engine already loaded
Auto configuration failed
140060080711488:error:260B606D:engine routines:DYNAMIC_LOAD:init failed:eng_dyn.c:521:
140060080711488:error:260BC066:engine routines:INT_ENGINE_CONFIGURE:engine configuration error:eng_cnf.c:204:section=gost_section, name=dynamic_path, value=/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so
140060080711488:error:0E07606D:configuration file routines:MODULE_RUN:module initialization error:conf_mod.c:235:module=engines, value=engine_section, retcode=-1

This appears ONLY on fresh Ubuntu (13.10 and 14.04). Everything is OK on Ubuntu 12.04 and Mac OS X with OpenSSL from Homebrew.

How to reproduce

Take a machine with Ubuntu 14.04.

Configure OpenSSL

Insert next line at very beginning of /etc/ssl/openssl.cnf file:

openssl_conf = openssl_def

Insert next lines at very end of this file:

[openssl_def]
engines = engine_section

[engine_section]
gost = gost_section

[gost_section]
engine_id = gost
dynamic_path = /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so
default_algorithms = ALL
CRYPT_PARAMS = id-Gost28147-89-CryptoPro-A-ParamSet

Check configuration: execute openssl ciphers | tr ":" "\n" | grep GOST, it should return something.

See official manual in OpenSSL sources for details: https://github.com/openssl/openssl/blob/master/engines/ccgost/README.gost

Install Ruby (example with RVM)

Crash occurs only in patched Ruby, because first patch enables Ruby to take openssl.cnf in account.

rvm install ruby-2.1.2-gost --patch https://bugs.ruby-lang.org/attachments/download/4420/respect_system_openssl_settings.patch --patch https://bugs.ruby-lang.org/attachments/download/4415/gost_keys_support_draft.patch

Patch for Rbenv (ruby-build): https://gist.github.com/Envek/82be109c58a0a565d382

Install Rails and create sample app

gem install rails --no-rdoc --no-ri
rails new pg_ossl_boom -d postgresql
cd pg_ossl_boom
echo "gem 'therubyracer',  platforms: :ruby" >> Gemfile
bundle

Setup Rails to connect to DB over HTTP

Add host: localhost to the default section.

Enjoy!

rake db:create

Questions

  1. Why pg (or libpq) ever tries to load GOST engine? Can it be disabled?
  2. Can SSL be disabled? (I don't need it)
  3. Any other workarounds? Where it is better to send this bug report?
  4. At least, how to get backtrace?

Additional information

I'm freely can load OpenSSL engine from irb or rails console, like this (it works):

require 'openssl'
OpenSSL::Engine.load
@gost_engine = OpenSSL::Engine.by_id('gost')
@gost_engine.set_default(0xFFFF)
@gost_engine.digest('md_gost94') # => <OpenSSL::Digest>

Also, all GOST-related stuff (digital signing and so on) works well (examples can be found at Ruby #9830).

I've found some issue that may be related to this in OpenSSL here: http://rt.openssl.org/Ticket/Display.html?id=3116&user=guest&pass=guest (may it help)

Tested with both 0.17.1 and 0.18.0.pre20140820094244. PostgreSQL (and all libs) version: 9.3 and 9.4 beta 2.

Thank you very much for attention, please help, I'm really lack of understanding what is going on.

Comments (10)

  1. Andrey Novikov reporter

    I've found two workarounds for this:

    1. Do not use TCP connection, use Unix socket
    2. Install custom OpenSSL and build Ruby against it, for example for RVM:

      rvm pkg install openssl         # will install OpenSSL 1.0.1i to ~/.rvm/usr
      vim ~/.rvm/usr/ssl/openssl.cnf # TODO: Apply GOST enabling configurations here
      rvm reinstall ruby-2.1.2-gost --with-openssl-dir=$HOME/.rvm/usr --patch https://bugs.ruby-lang.org/attachments/download/4420/respect_system_openssl_settings.patch --patch https://bugs.ruby-lang.org/attachments/download/4415/gost_keys_support_draft.patch
      
  2. Brian Stevens

    Andrey, I was able to reproduce this issue and I'm [slowly] working my way through it.

    The pg gem uses libpq underneath. The documentation on libpq-connect mentions sslmode in the available options. sslmode can be one of

    • disable
    • allow
    • prefer
    • require
    • verify-ca
    • verify-full

    prefer is the default. One work around is to set it to disable or allow; that will prevent the connection from attempting to use SSL (unless explicitly told to do so, in the case of allow). That can be done by adding sslmode: 'disable' in your databases.yml.

    That answers your question # 2.

    As far as question # 1: GOST pg/libpq is loading the GOST engine because, by default (as discussed above) libpq will attempt to use SSL, and your SSL configuration is set to use the GOST engine.

    "Can it be disabled?" Well, attempting to connect via SSL can be disabled or the GOST could be disabled by changing your SSL configuration file.

    The main remaining questions is: why, if the GOST engine is enabled, does this connection fail? I have not yet figured that part out.

  3. Andrey Novikov reporter

    @Brian Stevens, thank you very much!

    As for question #1 I probably meant: Can libpq check in libpq-connect for OpenSSL engines already loaded and don't try to load them one more time? Probably I should open a ticket somewhere near to libpq development :-)

    Also, why there is no error if I compile Ruby against separate OpenSSL? Is that means that Ruby code and pg/libpq configure and use two different OpenSSLs at the same time?

    And the last thing: I forced to use GOST cryptoalgorithms in my app (for digital signatures and TLS between my app and another system), so there is no choice for me, I just need it.

    Thank you again for taking your time and providing helpful info. Ask for any assistance I can provide to resolve this.

  4. Brian Stevens

    As I received your reply, I was writing another comment that it appears to me to be a problem with "double initialization" of the SSL libraries, as mentioned here: http://www.postgresql.org/docs/9.3/static/libpq-ssl.html#LIBPQ-SSL-INITIALIZE

    I tested with a two simple scripts:

    require 'pg'
    
    PG.connect( host: 'localhost', dbname: 'postgres', password: 'foo' )
    

    and

    require 'ssl'
    require 'pg'
    
    PG.connect( host: 'localhost', dbname: 'postgres', password: 'foo' )
    

    The first one works; the second crashes exactly as you mentioned.

    I assume Rails is loading the ssl library before the database connection is attempted, thus creating the problem. I'm not sure if there's a was for PG to query if SSL has already been initialized, or whether PG needs to implement the PQinitOpenSSL() call so that Ruby can call that.

  5. Brian Stevens

    I can't tell whether it's more an error in PG or in libpq.

    One option would be to implement libpq's PQinitOpenSSL in PG. I tried for several hours yesterday to do so, but was never successful.

    With that functionality, a user of PG could do something like:

    PG.initOpenSSL false
    

    if their application has already initiated OpenSSL

    or

    PG.initOpenSSL defined?(OpenSSL) ? false : true
    

    if they're unsure.

    As libpq is written, the default is effectively 'true'.

  6. Andrey Novikov reporter

    Thank you very much!

    I've installed version 0.18.0.pre20141117110243 and placed defined?(OpenSSL) ? PG.init_openssl(false, false) : PG.init_openssl(true, true) into initializer and Rakefile. And problem is gone!

    Currently tested only in testing VM, have no time for more testing now.

  7. Log in to comment