MutableDict associatess only with first level of JSON sql data

[Valentin Kaplov]

If we associate MutableDict with JSON data, then MutableDict accociates with first level elements only. As result if we have on second and/or next levels mutable objects (like list, dict) and we modify them, then MutableDict will not marked as dirty/changed and modification will not take effect to database. Thus we'll lose consistency. Example. We have next JSON data from database:

{
    "key1": {"key1.2": "value1.2"},
    "key2": ["value2.1", "value2.2"],
    "key3": "value3",
}

If we pass the data to MutableDict, then it will create next object tree:

Actual result:

m_dict = MutableDict({
    "key1": dict({"key1.2": "value1.2"}),    # value builtin dict
    "key2": list(["value2.1", "value2.2"]),  # value builtin list
    "key3": "value3"
})

m_dict['key1']['key1.2'] = 'new_value'   # m_dict doesn't marked as changed
m_dict['key1'][0] = 'new_value'          # m_dict doesn't marked as changed

Expected result:

MutableDict({
    "key1": MutableDict({"key1.2": "value1.2"}),     # value MutableDict asocciated with JSON
    "key2": MutableList(["value2.1", "value2.2"]),   # value MutableDict asocciated with JSON
    "key3": "value3"
})

m_dict['key1']['key1.2'] = 'new_value'   # m_dict marked as changed
m_dict['key1'][0] = 'new_value'          # m_dict marked as changed

Additional information: To fix the problem needs to have implemented MutableList https://bitbucket.org/zzzeek/sqlalchemy/issues/3297

Need to change all builtin mutables in JSON on corresponding associated Mutables from sqlalchemy. BTW Mirantis has already implemented wrappers that fix the bug. It can be usefull https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/db/sqlalchemy/models/mutable.py

[Mike Bayer]

• add more documentation to MutableDict explaining that this structure is only intended to track additions and removals from the dictionary, not recursive tracking of embedded changes. fixes #3646.

(cherry picked from commit 5497b1086f9fe44e0594079b036ea939fbc3c479)

→ <<cset 639b76778035>>

[Mike Bayer]

• add more documentation to MutableDict explaining that this structure is only intended to track additions and removals from the dictionary, not recursive tracking of embedded changes. fixes #3646.

→ <<cset 287aaa9d416b>>

[Mike Bayer]

• add more documentation to MutableDict explaining that this structure is only intended to track additions and removals from the dictionary, not recursive tracking of embedded changes. fixes #3646.

(cherry picked from commit 287aaa9d416b4f72179da320af0624b9ebc43846)

→ <<cset 3215a2dac0e5>>

[Mike Bayer]

Sorry, IMO it should not be expected that MutableDict is an automatically recursive structure, as this is a highly complex and error prone use case which cannot be directly supported by the project at this time. Documentation has been clarified.

[Mike Bayer]

Also, if Mirantis wants to publish their implementation as an extension project on Pypi, I'll gladly link to it in these documentation.

[Valentin Kaplov]

Unfortunately it is not intuitive behaviour. When client wraps JSON structure in Mutable, he expects, that any change of JSON structure will take effect in database, but not only first level. Also most of clients works with mutable objects like with builtin lists and builtin dicts. Client doesn't care about exact object type and ofcourse he doesn't know, that only changes on first level take effect in database. So current design of Mutable objects make them impossible to use safe and comfortable. One example.

# We'll change dict on second level of JSON tree
m_obj = MutableDict({
    'k1': {
        'k2': 'value'
    }
})

tmp_dict = m_obj['k1']
tmp_dict['k2'] = 'new_value'
m_obj['k1'] = tmp_dict

A new developer in sqlalchemy, but with good knowledge of python never guess do like above. Because it's unusual behaviour for JSON represented by builtin structures.

[Mike Bayer]

The Mutable system as a whole is not enabled by default, is not mentioned in the ORM tutorial, and is only referred to in terms of specific datatypes where the use of a Mutable structure might be useful. So the user has already found Mutable by reading the documentation. When they see that MutableDict is there, they will read my new documentation that explicitly says, "this will not work for JSON without changes". So there's no need for the developer to guess - it's written in the docs.

Will Mirantis contribute to the SQLAlchemy ecosystem by packaging their mutable structure standalone and publishing it on Pypi? I will gladly link to it as the canonical solution for this problem.

[Valentin Kaplov]

We discussing now is it possible to contribute to SQLAlchemy. We'll let you now results as soon as possible. Thank you.