Cant JSON serialize sqlalchemy.util.langhelpers.symbol

[Ken Sheppardson]

Starting with version 0.9.0, simplejson sees util.symbol objects as integers, causing it to generate invalid JSON.

We're trying to serialize stack traces and function arguments as JSON, and although we've defined a custom JSONEncoder default() method to handle non-standard data types, we're getting invalid JSON like:

...
"args": [
    "<sqlalchemy.orm.attributes.ScalarAttributeImpl object at 0x4966090>",
    "<sqlalchemy.orm.state.InstanceState object at 0x52ca650>",
    "{'_data': {'follower_count': None, 'followers': None, 'following': None, 'locale': u'zh_hans', 'region': None, 'timezone': u'Africa/Cairo', 'utc_offset': Decimal('2.0')}, '_modified': {'locale': u'zh_hans'}, '_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x52ca650>, 'locale': u'zh_hans'}",
    symbol('PASSIVE_OFF')
],
...

...because while our custom default() method handles data types simplejson doesn't recognize, it thinks "symbol('PASSIVE_OFF')" is an int, and dumps the raw value into the JSON.

To reproduce (using python 2.7.6, sqlalchemy 1.0.4, simplejson 3.6.5):

Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlalchemy as sa
>>> import json
>>> sym = sa.util.symbol("PASSIVE_OFF")
>>> data = ['str1', 123, sym]
>>> print json.JSONEncoder().encode(data)
["str1", 123, symbol('PASSIVE_OFF')]
>>> print sym
symbol('PASSIVE_OFF')
>>> type(sym)
<class 'sqlalchemy.util.langhelpers.symbol'>
>>> isinstance(sym,int)
True

Python 3 handles this slightly differently: rather than "symbol('PASSIVE_OFF')" in the JSON, we see integers like -4343837943729882877 in the output of encode().

[Mike Bayer]

IMO this is a bug in Python's json library. This is more than just serializing it correctly, these are constants that are compared using "is", so you need to pull them from the module.

However, I'm not able to get it to recognize the int subclass, in the script below "default" fails:

from sqlalchemy import util
import sys
import json


def _str_to_symbol(dct):
    if '__sa_symbol__' in dct:
        module = sys.modules[dct['module']]
        return getattr(module, dct['name'])
    else:
        return dct


def _symbol_to_str(obj):
    if isinstance(obj, util.symbol):
        return {
            "__sa_symbol__": 'true',
            'module': obj.__module__, 'name': obj.name}
    else:
        return obj


def json_loads(string):
    return json.loads(string, object_hook=_str_to_symbol)


def json_dumps(obj):
    return json.dumps(obj, default=_symbol_to_str)


from sqlalchemy.orm import attributes
sym = attributes.ATTR_WAS_SET

dumped = json_dumps([1, 2, sym])
loaded = json_loads(dumped)
assert loaded[2] is attributes.ATTR_WAS_SET

Though I can't see why "need to be able to JSON serialize the internals of the library" has to be a supported use case. If these constants were just simple ints, your program would still be failing, as these need to be the same int value; it's not safe to rely on "1 is 1" in Python, I've seen this fail on Pypy under cloudy circumstances.

[Ken Sheppardson]

Mike,

Though I can't see why "need to be able to JSON serialize the internals of the library" has to be a supported use case.

For context, this came up here at Rollbar, where we're trying capture as much app context information as is possible/practical when an exception's thrown.

We have a hack/workaround for our issue (https://github.com/rollbar/pyrollbar/issues/60)

...and I've created a simplejson issue (https://github.com/simplejson/simplejson/issues/118)

I'm not sure if I can really add anything else.

[Mike Bayer]

looks like simplejson accommodated it upstream: https://github.com/simplejson/simplejson/issues/118