I think I've spotted a bug with 0.18.4
We are scaling out our database and using several sidekiq workers.
We are currently using postgresql 9.4 with pgbouncer to allow more light-weight connections
Rails 3.2 + Latest version of Sidekiq + pg 0.18.4
Whenever we exhaust pgbouncer real connections Sidekiq just hangs and wont even respond to TTIN commands. I have to force kill sidekiq process with -9
This happens on our production environment whenever we reach our 250 real connections and hit our database really heavy
An easy way to reproduce locally is to create 1 sidekiq process with 25 threads and 1 real(server connection) pgbouncer connection(50000 client connections allowed)
Here is my local gdb output(osx) our production environments(centos) behave the same way
Here is my pgbouncer.ini config to make it easy to reproduce
I've created a minimal rails 4.2 app with sidekiq and I'm able to reproduce the same behaviour so this is doesn't seems to be a rails specific version issue
Thread 10 (Thread 0x2103 of process 40990): 0 0x00007fff94b892a2 in poll () from /usr/lib/system/libsystem_kernel.dylib 1 0x00000001035c892d in pqSocketCheck () from /usr/local/opt/postgresql/lib/libpq.5.dylib 2 0x00000001035c87f9 in pqWaitTimed () from /usr/local/opt/postgresql/lib/libpq.5.dylib 3 0x00000001035c5fd8 in PQgetResult () from /usr/local/opt/postgresql/lib/libpq.5.dylib 4 0x00000001035c62be in PQexecFinish () from /usr/local/opt/postgresql/lib/libpq.5.dylib 5 0x00000001035c31bc in PQsetClientEncoding () from /usr/local/opt/postgresql/lib/libpq.5.dylib 6 0x000000010359d2a7 in pgconn_set_default_encoding () from /Users/user/.rbenv/versions/2.2.4/gemsets/caremessage-v1.1/extensions/x86_64-darwin-15/2.2.0-static/pg-0.18.4/pg_ext.bundle 7 0x00000001035990b7 in pgconn_init () from /Users/user/.rbenv/versions/2.2.4/gemsets/caremessage-v1.1/extensions/x86_64-darwin-15/2.2.0-static/pg-0.18.4/pg_ext.bundle 8 0x0000000102daa929 in vm_call0_body () 9 0x0000000102da9d82 in rb_call0 () 10 0x0000000102cbc969 in rb_class_new_instance ()
pg is not releasing the GVL before making a network call.
PQsetClientEncoding() is currently called without releasing the GVL while the PG::Connection initialisation. It should either be GVL aware or use the asynchronous libpq interface.