serializer with non-ascii names

[Mike Bayer]

# coding: utf-8
from sqlalchemy import *
from sqlalchemy.util import ue

from sqlalchemy.ext import serializer

m = MetaData()
t = Table(ue('\u6e2c\u8a66'), m,
        Column(ue('\u6e2c\u8a66_id'), Integer))

expr = select([t](t)).where(t.c[ue('\u6e2c\u8a66_id')](ue('\u6e2c\u8a66_id')) == 5)

expr2 = serializer.loads(serializer.dumps(expr, -1), m)

print expr2

[Mike Bayer]

3a3aeb431e08277346078fe60437f1fcfde957e2 0.8

2aa00c49d7a1a783ff50832f2de77 0.9

[Martijn Pieters]

This change breaks pickling with protocol 0:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/mj/.buildout/eggs/SQLAlchemy-0.9.3-py2.7-macosx-10.4-x86_64.egg/sqlalchemy/ext/serializer.py", line 149, in dumps
    pickler.dump(obj)
PicklingError: persistent id must be string

because it won't accept Unicode strings as persistent ids. Protocol versions 1 and up still work. This may need documenting or fixing.

[Mike Bayer]

do you actually use the serializer? I'd love to just dump it.

[Martijn Pieters]

I just advised someone to use it: http://stackoverflow.com/q/25452179/100297

Personally I'd just store the request parameters that inform the query instead and make sure I can rebuild the query with a utility function, but apparently that's too complex for some, so I can see the attraction of the extension.

[Mike Bayer]

the thing really has a point at which it fails. pickle can't handle really complex reference cycles and other weird things.

[Martijn Pieters]

Pickle can handle most stuff just fine, certainly reference cycles (object ids are memoized and references inserted when encountered again). :-) For 'weird things' __gestate__ / __setstate__ hooks usually suffice. The ZODB has been handling 'weird things' for decades now.

That said, I can imagine that having to maintain something like this can be a bit of a pain. Any specific examples you had in mind?

[Mike Bayer]

one of the issues with pickle that was discovered by this extension is http://bugs.python.org/issue9269. it took them two years to fix and I guess the fix is in Python3.2. If you look at the test suite you can see lots of "testing.requires.non_broken_pickle" directives, which is specifically requiring cPickle on Py2K. At least one test there is just commented out anyway, it fails all over the place non-deterministically. these issues are super hard to debug and as we've observed the use case for serializer is pretty slim.

[Mike Bayer]

Removing milestone: 0.8.xx (automated comment)

[Marcin Lulek]

Seems we are affected by this bug - after migration from 0.7.3 to 0.9.7 we are getting "PicklingError: persistent id must be string" - protocol 1 fixes that in my test.

[Mike Bayer]

I really wish I never wrote this silly extension why on earth do you need it ?

[Marcin Lulek]

Some legacy code that builds a complex query from report analyzer that is later passed to celery task that I inherited. I know that the proper solution would probably be to pass the dictionary that is used to build the query that gets serialized in first place - and rebuild the query inside the task, but the code is spread across few classes - and its PITA to rewrite that pile of code. Anyways just signalizing that it bit me after upgrade ;-).