break out pool into a package

Issue #4255 resolved
Michael Bayer
repo owner created an issue

There's enough attention going on with the connection pool these days that it might be nice to break it into a package, such that the main Pool/ConnectionFairy/ConnectionRecord is in one place, the ancient/never-used/crufty "manage" stuff can be elsewhere (and probably deprecated), and then the implementation classes like QueuePool, SingletonThreadPool can each get their own file and be kind of manageable.

Optionally, think about evaluating haalchemy's pool https://bitbucket.org/zzzeek/haalchemy/src/master/haalchemy/clients/sqlalchemy/pool.py for inclusion as well. For AbstractPriorityPool it still might be nice to have the "queue" part of it broken out into a separate class so that it can be more easily compared to QueuePool.

Comments (4)

  1. Michael Bayer reporter

    re: prioritypool, here's some real risk unless we change this:

      self._mutex.acquire()
            try:
                self._thread_id = me
                returned_queue.append(connection_record)
                while returned_queue:
                    rec = returned_queue.pop()
                    wrapper = rec.record_info["slot"]()
                    if self._overflow_count:
                        self._overflow_count -= 1
                        wrapper.checkin_for_overflow()
                    else:
                        wrapper.checkin_for_pooled()
                    heapq.heappush(self._queue, wrapper)
                    self._not_empty.notify()
            finally:
                # huge assumption.  That gc collection will *never* be called
                # in this thread in between these two calls.  SQLA's QueuePool
                # essentially makes a similar assumption within RLock.
                self._thread_id = None
                self._mutex.release()
    

    that's part of the checkin. that means we're doing network activity inside the mutex, with the "checkin for overflow" part. that's a thorny part b.c. we are trying to solve the re-entrant problem differently, but the queue part of this w/ the mutex needs to be separate and not have any large operations happening within. might need some creativity to do the re-entrant aspect differently.

  2. Michael Bayer reporter

    Refactor pool.py into a package

    for better separation of concerns split out the core pooling constructs from the implementations. also makes more room for new pool classes should we decide to add them. The DBProxy feature is never used so remove it from documentation and split it out.

    Change-Id: I04c9d7ae1882b17f3cf5c37ed05b781b05ef88fa Fixes: #4255

    → <<cset 9700eb126b05>>

  3. Log in to comment