Pickle of Properties object fails

[Pieter Mulder]

Trying to pickle a Properties object as shown below fails.

import pickle

from sqlalchemy.util import Properties

data = {'hello': 'bye'}

props = Properties(data)

pickle.loads(pickle.dumps(props))

Traceback (most recent call last):
  File "test_collections.py", line 14, in <module>
    pickle.loads(pickle.dumps(props))
  File "/usr/lib64/python2.7/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib64/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib64/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/home/pieter/.virtualenvs/proigia-rest/lib64/python2.7/copy_reg.py", line 84, in _reduce_ex
    dict = getstate()
  File "/home/pieter/.virtualenvs/proigia-rest/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 202, in __getstate__
    return {'_data': self.__dict__['_data']}
  File "/home/pieter/.virtualenvs/proigia-rest/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 211, in __getattr__
    raise AttributeError(key)
AttributeError: __dict__

It's clear Properties is __slot__ based and thus has no __dict__. SQLAlchemy 1.0.13

[Mike Bayer]

Easy to fix (PR w/ test would help), however currently no part of SQLAlchemy seems to make use of Properties directly except for the ColumnCollection -> OrderedProperties chain, for which ColumnCollection provides its own getstate/setstate and that's where the test coverage is. May I ask what your use case for direct use of Properties is ? sqlalchemy.util is not "private" per-se but the constructs in here are intended only to support the library features themselves.

[Pieter Mulder]

I'll try to make a PR.

I do not use the the Properties directly: The original exception was returned from ImmutableProperties which I then traced back to the Properties object.

After a lot of searching I have found I was hitting this by pickling a column in a table which has a colanderalchemy object attached:

import sqlalchemy as sa
from sqlalchemy.ext import declarative
import colanderalchemy

Base = declarative.declarative_base()

import pickle

class Table1(Base):
    __tablename__ = 'table1'
    col1 = sa.Column(sa.String(), primary_key=True)
    col2 = sa.Column(sa.String())

def main():
    colanderalchemy.setup_schema(None, Table1) 
    pickle.dumps(Table1.col1)

if __name__ == '__main__':
    main()

Resulting in:

Traceback (most recent call last):
  File "sqlalchemy_test2.py", line 22, in <module>
    main()
  File "sqlalchemy_test2.py", line 19, in main
    pickle.dumps(Table1.col1)
  File "/usr/lib64/python2.7/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib64/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib64/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib64/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/usr/lib64/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib64/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib64/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/usr/lib64/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib64/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib64/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/usr/lib64/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib64/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/usr/lib64/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib64/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib64/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/usr/lib64/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/home/pieter/.virtualenvs/proigia-rest/lib64/python2.7/copy_reg.py", line 84, in _reduce_ex
    dict = getstate()
  File "/home/pieter/.virtualenvs/proigia-rest/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 202, in __getstate__
    return {'_data': self.__dict__['_data']}
KeyError: '_data'

Commenting out the colanderalchemy.setup_schema(None, Table1) line will result in a different pickle exception (which I have not debugged yet as I've decided to take an other route that never tries to pickle a column object).

[Mike Bayer]

Repair pickling for Properties object

Fixed bug whereby the __getstate__ / __setstate__ methods for sqlalchemy.util.Properties were non-working due to the transition in the 1.0 series to __slots__. The issue potentially impacted some third-party applications. Pull request courtesy Pieter Mulder.

Fixes: #3728 Change-Id: I01ebd425bbfe145747fea2edd0d2d412c74fd84d Pull-request: https://github.com/zzzeek/sqlalchemy/pull/286 (cherry picked from commit cab57e9bab04fbdea44690c08dff379a29eaab32)

→ <<cset fcfff7712817>>

[Mike Bayer]

Repair pickling for Properties object

Fixed bug whereby the __getstate__ / __setstate__ methods for sqlalchemy.util.Properties were non-working due to the transition in the 1.0 series to __slots__. The issue potentially impacted some third-party applications. Pull request courtesy Pieter Mulder.

Fixes: #3728 Change-Id: I01ebd425bbfe145747fea2edd0d2d412c74fd84d Pull-request: https://github.com/zzzeek/sqlalchemy/pull/286 (cherry picked from commit cab57e9bab04fbdea44690c08dff379a29eaab32)

→ <<cset 39c18b53aa7e>>

[Mike Bayer]

Fix #3728 Pickle of Properties object fails

Change-Id: I01ebd425bbfe145747fea2edd0d2d412c74fd84d Pull-request: https://github.com/zzzeek/sqlalchemy/pull/286

→ <<cset 20fe9ad17131>>

[Mike Bayer]

Repair pickling for Properties object

Fixed bug whereby the __getstate__ / __setstate__ methods for sqlalchemy.util.Properties were non-working due to the transition in the 1.0 series to __slots__. The issue potentially impacted some third-party applications. Pull request courtesy Pieter Mulder.

Fixes: #3728 Change-Id: I01ebd425bbfe145747fea2edd0d2d412c74fd84d Pull-request: https://github.com/zzzeek/sqlalchemy/pull/286

→ <<cset cab57e9bab04>>