Issue #178 invalid

pg's way of linking to openssl is not correct

Antelope Salad
created an issue

I was trying to deploy my rails app on a chef-managed server today and came across an issue with the pg gem.

Env:
Postgres 9.3.2 (libpq5 and libpq-dev are both installed)
Ubuntu-server 12.04 LTS 32 bit version
OpenSSL 1.0.1

Here are the error logs:
https://gist.github.com/AntelopeSalad/8285590

Ruby is installed through omnibus using poise-ruby:
https://github.com/poise/poise-ruby

Postgres is installed through a popular chef cookbook:
https://github.com/phlipper/chef-postgresql

Here's how another postgres cookbook hacks around the broken pg issue:
https://github.com/hw-cookbooks/postgresql/blob/master/recipes/ruby.rb#L56-120

A couple of us were talking about it on IRC and the conclusion is...

  • The pg gem bypasses the RbConfig settings about which openssl it was compiled against
  • It is certainly a problem with the pg gem and not how Ruby is installed

If you're on freenode then you might want to contact coderanger in #chef or workmad3 in #chef because they know much more about this than me.

Comments (26)

  1. Lars Kanis

    ruby-pg does not link to OpenSSL at all. It uses pgconfig --libdir so that the libraries requested by libpq are respected at build time. If libssl or libcrypto are included there, they should also get linked properly. Your link to the cookbook hack doesn't make it clearer to me, what the problem is there.

    Can you post your output of pgconfig --libdir?

    Can you try the install with env CROSS_COMPILING=true gem inst pg? This explicitly links to OpenSSL, which should only be required for cross compilation.

    Can you post a link to the chat log with the relevant conversation?

    Did you find a workaround for this issue?

  2. Antelope Salad reporter
    $ pg_config --libdir  
    /usr/lib
    

    I'm not sure how to do the 2nd thing you're asking me to do.

    Chat log:
    https://gist.github.com/AntelopeSalad/8289702

    I also did not find a work around. If I install ruby through rbenv or rvm it will probably work because this is how I installed ruby on my development machine and I did not have any problems. I do not want to use rbenv/rvm on my production server though.

    That is why I went to the poise-ruby author first thinking it's likely a problem with how Ruby was installed but then after those 2 guys investigated the logs they came to the conclusion that it's really an issue with the pg gem.

  3. Lars Kanis

    I'm not sure how to do the 2nd thing you're asking me to do.

    Set the environment variable CROSS_COMPILING=true and do another install of the pg gem.

  4. Antelope Salad reporter

    Here's the result of running what you asked me to run:

    foo@cheftests:/opt/ruby-210/bin$ sudo CROSS_COMPILING=true ./gem install pg -v '0.17.0'
    Fetching: pg-0.17.0.gem (100%)
    Building native extensions.  This could take a while...
    ERROR:  Error installing pg:
        ERROR: Failed to build gem native extension.
    
        /opt/ruby-210/bin/ruby extconf.rb
    checking for CertOpenStore() in -lcrypt32... no
    checking for CreateDC() in -lgdi32... no
    checking for main() in -lsecur32... no
    checking for WSASocket() in -lws2_32... no
    checking for BIO_new() in -lcrypto... yes
    checking for SSL_new() in -lssl... yes
    checking for pg_config... yes
    Using config values from /usr/bin/pg_config
    checking for libpq-fe.h... yes
    checking for libpq/libpq-fs.h... yes
    checking for pg_config_manual.h... yes
    checking for PQconnectdb() in -lpq... no
    checking for PQconnectdb() in -llibpq... no
    checking for PQconnectdb() in -lms/libpq... no
    Can't find the PostgreSQL client library (libpq)
    *** extconf.rb failed ***
    Could not create Makefile due to some reason, probably lack of necessary
    libraries and/or headers.  Check the mkmf.log file for more details.  You may
    need configuration options.
    
    Provided configuration options:
        --with-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/opt/ruby-210/bin/ruby
        --with-pg
        --without-pg
        --with-crypt32lib
        --without-crypt32lib
        --with-gdi32lib
        --without-gdi32lib
        --with-secur32lib
        --without-secur32lib
        --with-ws2_32lib
        --without-ws2_32lib
        --with-cryptolib
        --without-cryptolib
        --with-ssllib
        --without-ssllib
        --with-pg-config
        --without-pg-config
        --with-pg_config
        --without-pg_config
        --with-pg-dir
        --without-pg-dir
        --with-pg-include
        --without-pg-include=${pg-dir}/include
        --with-pg-lib
        --without-pg-lib=${pg-dir}/lib
        --with-pqlib
        --without-pqlib
        --with-libpqlib
        --without-libpqlib
        --with-ms/libpqlib
        --without-ms/libpqlib
    
    extconf failed, exit code 1
    
    Gem files will remain installed in /opt/ruby-210/embedded/lib/ruby/gems/2.1.0/gems/pg-0.17.0 for inspection.
    Results logged to /opt/ruby-210/embedded/lib/ruby/gems/2.1.0/extensions/x86-linux/2.1.0/pg-0.17.0/gem_make.out
    

    Edit: In case the env was necessary I just re-ran it with that and I got the same error.

  5. Lars Kanis

    ruby-pg is probably using the wrong pg_config:

    Using config values from /usr/bin/pg_config

    Does it help to set gem inst pg -- --with-pg-config= to the correct chef path?

  6. Lars Kanis

    Sorry I never worked with omnibus nor chef, so I expect they install some files in a custom directory (like /opt/ruby-210/embedded/lib), but don't really know where they install their files. So I mean to search for pg_config in these paths and then run gem inst pg -- --with-pg-config=/path/to/pg_config accordingly.

  7. Antelope Salad reporter

    Normally I'm installing this with bundler using --deployment btw in case you're wondering why I'm using sudo or don't have a path exported.

    sudo /opt/ruby-210/bin/gem install pg -- --with-pg-config=/usr/bin/pg_config

    This results in an error too:

    Building native extensions with: '--with-pg-config=/usr/bin/pg_config'
    This could take a while...
    ERROR:  Error installing pg:
        ERROR: Failed to build gem native extension.
    
        /opt/ruby-210/bin/ruby extconf.rb --with-pg-config=/usr/bin/pg_config
    checking for CertOpenStore() in -lcrypt32... no
    checking for CreateDC() in -lgdi32... no
    checking for main() in -lsecur32... no
    checking for WSASocket() in -lws2_32... no
    checking for BIO_new() in -lcrypto... yes
    checking for SSL_new() in -lssl... yes
    Using config values from /usr/bin/pg_config
    checking for libpq-fe.h... yes
    checking for libpq/libpq-fs.h... yes
    checking for pg_config_manual.h... yes
    checking for PQconnectdb() in -lpq... no
    checking for PQconnectdb() in -llibpq... no
    checking for PQconnectdb() in -lms/libpq... no
    Can't find the PostgreSQL client library (libpq)
    *** extconf.rb failed ***
    Could not create Makefile due to some reason, probably lack of necessary
    libraries and/or headers.  Check the mkmf.log file for more details.  You may
    need configuration options.
    
    Provided configuration options:
        --with-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/opt/ruby-210/bin/ruby
        --with-pg
        --without-pg
        --with-crypt32lib
        --without-crypt32lib
        --with-gdi32lib
        --without-gdi32lib
        --with-secur32lib
        --without-secur32lib
        --with-ws2_32lib
        --without-ws2_32lib
        --with-cryptolib
        --without-cryptolib
        --with-ssllib
        --without-ssllib
        --with-pg-config
        --with-pg-dir
        --without-pg-dir
        --with-pg-include
        --without-pg-include=${pg-dir}/include
        --with-pg-lib
        --without-pg-lib=${pg-dir}/lib
        --with-pqlib
        --without-pqlib
        --with-libpqlib
        --without-libpqlib
        --with-ms/libpqlib
        --without-ms/libpqlib
    
    extconf failed, exit code 1
    
    Gem files will remain installed in /opt/ruby-210/embedded/lib/ruby/gems/2.1.0/gems/pg-0.17.1 for inspection.
    Results logged to /opt/ruby-210/embedded/lib/ruby/gems/2.1.0/extensions/x86-linux/2.1.0/pg-0.17.1/gem_make.out
    
  8. Lars Kanis

    That is obviously, because /usr/bin/pg_config is the file, that was used for all previous tries. Is there no other pg_config file resulting from the chef installation?

  9. Michael Granger repo owner

    I appreciate your bringing this up, but the situation is a bit more complex than you're suggesting. The problem is that libpq links to OpenSSL itself:

    $ ldd /usr/lib/libpq.so.5.4 | grep 'crypt\|ssl'
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f6d0ed09000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f6d0e92e000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f6d0d3ac000)
        libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f6d0c1e1000)
        libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f6d0b88d000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f6d0a4ea000)
    

    This means that if your Ruby's openssl library is linked against a different OpenSSL, which is often the case if you're using a version that isn't installed with the host's packaging system, using both openssl and pg often results in a segfault. If you Google for 'ruby pg net http segfault' you'll find a bunch of mentions of this problem.

    The right fix for this is to always ensure that your PostgresSQL and Ruby share the same OpenSSL library. On my Ubuntu box, for example, this Ruby is installed via the packaging system, so there's no problem:

    $ ldd /usr/lib/ruby/1.9.1/x86_64-linux/openssl.so | grep 'crypt\|ssl'
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fa6d288c000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fa6d24b0000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fa6d188e000)
    
    $ sudo gem install pg
    Fetching: pg-0.17.1.gem (100%)
    Building native extensions.  This could take a while...
    Successfully installed pg-0.17.1
    1 gem installed
    Installing RDoc documentation for pg-0.17.1...
    

    The current linking code is the setup that resulted in the fewest problems, after much trial and error. If you (or any of the people you mentioned) have a better way that works across every supported platform and environment, we'd be happy to accept pull requests/patches/suggestions.

    I'd especially be interested in hearing why "it is certainly a problem with the pg gem and not how Ruby is installed".

    I am on freenode (as 'ged' in #ruby-lang, #ruby, #postgresql, etc.), but not in #chef, as I don't use Chef.

    See also: #76, #77.

  10. Michael Granger repo owner

    Okay, I've never used 'poise-ruby', but can you confirm that there is a libcrypto.x.x.so and libssl.x.x.so in /opt/ruby-210/embedded/lib? And that /usr/lib/libpq.x.x.so isn't linked against it?

    If the system you're using installs different versions of common libraries like OpenSSL, then you'll need to ensure that everything you expect to work together uses the same libraries. E.g., if poise-ruby installs its own OpenSSL, you'll need to compile PostgreSQL using that version of OpenSSL, not the one Ubuntu installs. Or, as Lars was suggesting, use a version of 'pg' that's been statically compiled with all of its dependencies.

    Anything else will either result in compilation failures, or worse, segfaults somewhere after your code starts running.

    There's nothing whatsoever we can do in the 'pg' gem (or any other extension, for that matter) to resolve basic library conflicts on everyone's systems that are introduced before the gem is even installed. That's your responsibility as a sysadmin. The instant you move outside of your distribution's packaging system the onus is on you to be aware of how shared libraries work, and construct compilation and runtime environments accordingly.

    Does that make sense?

  11. Antelope Salad reporter

    I'm not sure how to determine if it's linked against it or not, so I grepped for the files in the directory you asked me to.

    foo@cheftests:/opt/ruby-210/embedded/lib$ ls -la | grep libcrypto
    -rw-r--r-- 1 root root  3208062 Jan  4 13:56 libcrypto.a
    lrwxrwxrwx 1 root root       18 Jan  4 13:56 libcrypto.so -> libcrypto.so.1.0.0
    -r-xr-xr-x 1 root root  2027504 Jan  4 13:56 libcrypto.so.1.0.0
    
    foo@cheftests:/opt/ruby-210/embedded/lib$ ls -la | grep libssl
    -rw-r--r-- 1 root root   594384 Jan  4 13:56 libssl.a
    lrwxrwxrwx 1 root root       15 Jan  4 13:56 libssl.so -> libssl.so.1.0.0
    -r-xr-xr-x 1 root root   446648 Jan  4 13:56 libssl.so.1.0.0
    

    The version of postgres I'm using is sitting in a PPA somewhere that's tested against the OS I'm deploying to.

    Part of the reason why I tried ruby-poise is because not having to deal with the complexity of RVM and not also not having to wait 30 minutes for Ruby to compile seemed nice.

    What should we do now? I don't have the coding skills to fix this problem from either end and it seems like coderanger is saying it's your gem's fault but you're saying it's his lib's fault for using a non-standard version of openssl.

  12. Michael Granger repo owner

    No, that's definitely not what I'm saying. I don't think coderanger wrote libpq (I could be wrong), but it's the Ubuntu libpq-dev packager's responsibility to ensure that the version it uses works with the version of OpenSSL it depends on. And it does; I use it to test pg on Linux all the time with the Ruby that comes with the same system.

    Despite what coderanger (apparently) claims, this isn't the gem's "fault" either.

    No (practical) amount of coding skill will "fix" this. It's a library conflict introduced by your installing software outside of your chosen packaging system. If you need software that isn't included with the system you choose, then you'll need to figure out a solution that meets your needs. There's no way I (or the #chef people you mentioned) can do that. If you don't want to troubleshoot problems like this, a sensible practice is to stick with one way of installing software (e.g., if you install PostgreSQL using apt-get, install Ruby and ruby-pg using apt-get). You can certainly run multiple versions of software or use multiple packaging systems on a single host, but it requires a higher degree of expertise to ensure that things work well together.

    BTW, you can use ldd to check shared libraries; see my earlier comment for examples.

  13. Antelope Salad reporter

    I think it's that all omnibus generated ruby packages use a different version of openSSL than what's on most systems. I don't know his credentials, I can only go by what I read on IRC as I'm nothing more than a middle man in this conversation.

    I moved to using RVM to setup ruby and it works fine with the pg gem as usual.

    What's the final verdict then? It's impossible to use omnibus-generated ruby packages with the pg gem unless the creator of libpg-dev fixes it?

  14. Michael Granger repo owner

    No, I'd say as long as omnibus installs its own version of OpenSSL that it should also install its own version of anything that depends on that library that's intended to work in that environment as well, which would include libpq. Since libpq-dev is a Ubuntu package, designed to work inside Ubuntu's packaging system, there's no way for the maintainer of that package to modify it for omnibus.

  15. Log in to comment