association_proxy cannot be pickled

[Former user]

Classes that use association_proxy attributes cannot be pickled on account of the inline functions and weakref. This limits its usefulness.

[Former user]

test case

[Mike Bayer]

here's a patch towards addressing that:

Index: lib/sqlalchemy/ext/associationproxy.py
===================================================================
--- lib/sqlalchemy/ext/associationproxy.py  (revision 6063)
+++ lib/sqlalchemy/ext/associationproxy.py  (working copy)
@@ -240,7 +240,7 @@
             getter, setter = self._default_getset(self.collection_class)

         if self.collection_class is list:
-            return _AssociationList(lazy_collection, creator, getter, setter)
+            return _AssociationList(self, lazy_collection, creator, getter, setter)
         elif self.collection_class is dict:
             return _AssociationDict(lazy_collection, creator, getter, setter)
         elif self.collection_class is set:
@@ -251,7 +251,23 @@
                 'collection_class "%s" backing "%s"; specify a '
                 'proxy_factory and proxy_bulk_set manually' %
                 (self.collection_class.__name__, self.target_collection))
+    
+    def _inflate_collection(self, assoc_collection, collection):
+        lazy_collection = self._lazy_collection(weakref.ref(collection))
+        
+        creator = self.creator and self.creator or self.target_class
+        self.collection_class = util.duck_type_collection(collection)

+        if self.getset_factory:
+            getter, setter = self.getset_factory(self.collection_class, self)
+        else:
+            getter, setter = self._default_getset(self.collection_class)
+        
+        assoc_collection.lazy_collection = lazy_collection
+        assoc_collection.creator = creator
+        assoc_collection.getter = getter
+        assoc_collection.setter = setter
+        
     def _set(self, proxy, values):
         if self.proxy_bulk_set:
             self.proxy_bulk_set(proxy, values)
@@ -270,7 +286,7 @@
 class _AssociationList(object):
     """Generic, converting, list-to-list proxy."""

-    def __init__(self, lazy_collection, creator, getter, setter):
+    def __init__(self, parent, lazy_collection, creator, getter, setter):
         """Constructs an _AssociationList.

         lazy_collection
@@ -296,9 +312,17 @@
         self.creator = creator
         self.getter = getter
         self.setter = setter
-
+        self.parent = parent
+        
     col = property(lambda self: self.lazy_collection())

+    def __getstate__(self):
+        return {'collection':self.lazy_collection(), 'parent':self.parent}
+    
+    def __setstate__(self, state):
+        self.parent = state['parent']('parent')
+        self.parent._inflate_collection(self, state['collection']('collection'))
+        
     def _create(self, value):
         return self.creator(value)

[Mike Bayer]

q. for jason:

• it seems weird I'm pickling the actual collection above ? not thinking of a better way at the moment.

• can I place _AssociationDict/_AssociationList/_AssociationSet on a common base class that would provide __init__(), __getstate__() and __setstate__() ? or was there some reason they are broken up like that ?

• do we think anyone is actually using proxy_factory ? its signature would have to change to receive the 'parent' association proxy. the AP seems key here in order for a collection to have what it needs to re-inflate itself.

[jek]

Pickling a view doesn't make a lot of sense to me. It's pretty simple to just skip the association proxy in the mapped object's getstate & let it be recreated sanely on first access.

Will take a peek into the base class issue. I do know that proxy_factory is being used.

[jek]

Doesn't look like there'd be any issues mixing in pickling functions. They'd need separate __init__s for documentation & the differing signatures. The existing class don't share a base because the classes are not related.

[jek]

scratch the differing signatures bit!

[Mike Bayer]

Replying to jek:

Pickling a view doesn't make a lot of sense to me. It's pretty simple to just skip the association proxy in the mapped object's getstate & let it be recreated sanely on first access.

I thought the same thing but its not obvious at all:

class Thing:
    tags = association_proxy('_tags', 'label')

    def __getstate__(self):
        self.__dict__.pop(Thing.tags.key, None)
        return self.__dict__

I think having a __getstate__ __setstate__ on the proxy classes is not a big deal and will eliminate the need for users to even have a surprise like this.

[jek]

Fair enough. This can likely be fixed without any interface changes by implementing a reduce that digs up the parent reference via the collection_adapter() of the lazy_collection & uses the descriptor to recreate the proxy view.

[Mike Bayer]

I'm going to likely patch what i have above for 0.5.6 just so something works

[jek]

test case

[jek]

Attached some basic pickling tests. Had some issues with {{{reduce}}} unpickling (was firing before _sa_instance_state was reinstated on the owner object). I'll keep poking at it, but maybe an API change is needed after all.

[Mike Bayer]

a refined version that only pickles the parent object (which we would assume is usually the case) is in ed8742e6858f11d48c78fcbbad35e92834aa47f0. a user-defined proxy can in fact implement pickling if it is creative about getting the parent association proxy back off of the class, and in any case it requires knowledge of the _inflate() method.

[Mike Bayer]

Removing milestone: 0.5.6 (automated comment)