restarting mysql can cause connections to not be returned to the pool
If you restart the MySQL server while a query is being executed against it, this can result in the connection not being properly returned to the pool.
Attached is an example script which just loops, executing a query against MySQL and then asserting that no connections remain checkout out of the pool. AFAICT this is an appropriate invariant of the engine/pool - when no queries are being run, there should be nothing checked out.
If you run this script, then restart MySQL while it's running, a connection will often remain checked out after e.execute() returns, triggering the assertion.
As near as I can tell, what happens is the following:
- MySQL gets the signal to restart while a query is being executed
- It terminates the running query, producing error "1317 Query Execution was interrupted"
- The Connection._handle_dbapi_exception() method is invoked to deal with the error
- This error does not signal a disconnect, so we attempt to execute self._autorollback() before returning the connection to the pool
- MySQL has stopped, so this raises another error, now "2006 MySQL server has gone away"
- Connection._handle_dbapi_exception() is invoked again, but immediately re-raises the error since self._reentrant_error is now set to True
- The error propagates up to the calling code, and the connection is not returned to the pool
The connection is dead at this point, so it should be invalidated. Instead it stays around as a "zombie" connection, not referenced from anywhere but still marked as checked out by the pool.