serializer with non-ascii names

Issue #2869 resolved
Michael Bayer
repo owner created an issue
# coding: utf-8
from sqlalchemy import *
from sqlalchemy.util import ue

from sqlalchemy.ext import serializer

m = MetaData()
t = Table(ue('\u6e2c\u8a66'), m,
        Column(ue('\u6e2c\u8a66_id'), Integer))

expr = select([t](t)).where(t.c[ue('\u6e2c\u8a66_id')](ue('\u6e2c\u8a66_id')) == 5)

expr2 = serializer.loads(serializer.dumps(expr, -1), m)

print expr2

Comments (11)

  1. Martijn Pieters

    This change breaks pickling with protocol 0:

    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/Users/mj/.buildout/eggs/SQLAlchemy-0.9.3-py2.7-macosx-10.4-x86_64.egg/sqlalchemy/ext/", line 149, in dumps
    PicklingError: persistent id must be string

    because it won't accept Unicode strings as persistent ids. Protocol versions 1 and up still work. This may need documenting or fixing.

  2. Martijn Pieters

    Pickle can handle most stuff just fine, certainly reference cycles (object ids are memoized and references inserted when encountered again). :-) For 'weird things' __gestate__ / __setstate__ hooks usually suffice. The ZODB has been handling 'weird things' for decades now.

    That said, I can imagine that having to maintain something like this can be a bit of a pain. Any specific examples you had in mind?

  3. Michael Bayer reporter

    one of the issues with pickle that was discovered by this extension is it took them two years to fix and I guess the fix is in Python3.2. If you look at the test suite you can see lots of "testing.requires.non_broken_pickle" directives, which is specifically requiring cPickle on Py2K. At least one test there is just commented out anyway, it fails all over the place non-deterministically. these issues are super hard to debug and as we've observed the use case for serializer is pretty slim.

  4. Marcin Lulek

    Seems we are affected by this bug - after migration from 0.7.3 to 0.9.7 we are getting "PicklingError: persistent id must be string" - protocol 1 fixes that in my test.

  5. Marcin Lulek

    Some legacy code that builds a complex query from report analyzer that is later passed to celery task that I inherited. I know that the proper solution would probably be to pass the dictionary that is used to build the query that gets serialized in first place - and rebuild the query inside the task, but the code is spread across few classes - and its PITA to rewrite that pile of code. Anyways just signalizing that it bit me after upgrade ;-).

  6. Log in to comment