append event can fire after the parent object is already gc'ed

[Former user]

I've noticed a small problem going from 0.5.5 to 0.6.6 which the attached unit test should illustrate. It runs without errors on 0.5.5 but the first test errors out on 0.6.6. The second test shows a workaround. No biggie but curious, I think. Here's the output when running this test:

sas% python test_sa66.py
sqlalchemy.version: 0.5.5
..
----------------------------------------------------------------------
Ran 2 tests in 0.031s

OK



sas% python test_sa66.py
sqlalchemy.version: 0.6.6
F.
======================================================================
FAIL: test_01_failing_on_sa66 (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_sa66.py", line 30, in test_01_failing_on_sa66
    self.assertEqual(1, self.db.m2m.count())
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 2 tests in 0.033s

FAILED (failures=1)

[Former user]

Bugger, I wanted to CC myself but turns out I can't modify after submission :) Here's my address; [e-mail] if you'd be so kind and add it. Thanks!

[Mike Bayer]

OK well, yeah, nothing has actually "changed" in SQLA since 0.5 regarding what happens here, its a garbage collection artifact allowing it to work in 0.5. If I do this to 0.5:

--- a/lib/sqlalchemy/orm/state.py   Wed Sep 15 12:02:46 2010 -0400
+++ b/lib/sqlalchemy/orm/state.py   Thu Feb 10 10:24:56 2011 -0500
@@ -277,6 +277,8 @@

         self.modified = True
         if self._strong_obj is None:
+            import gc
+            gc.collect()
             self._strong_obj = self.obj()

     def commit(self, dict_, keys):

it fails there too.

Will poke around and see if we can find some way to keep obj() (in this case it's your MappedFoo) from falling out of scope while its receiving an append event, though that suggests artificially creating a reference, always dangerous.

[Mike Bayer]

OK thanks for reporting this I'm happy with the solution which is to raise an exception for this condition. It's a warning in 0.6 to ensure existing code that "works" regardless continues to run.

e155b7b8640c18e5ca33d5529fbc11f4776ae4c5 036d2c1e70f10f0986c14d0d9be198b4c6fa566c

[Mike Bayer]

that should be d8aa3383d5c88e2b503087d6dd0480146fe8bdf5 for the 0.6 branch.

[Former user]

Thanks for the quick response and fix! Just to understand the implications, you're saying (in CHANGES) that

foo.first().bars.append(b)

should fail, because in this rare case append is sent to an already dereferenced parent object. I suppose the 'proper' way to write this is then

f = foo.first()
f.bars.append(b)

as I did in my work-around. I usually don't chain calls like that anyway but I'm still surprised that 'chaining' vs 'not-chaining' behaves in such a fundamentally different way. Is this due to the way mapped objects work? I wouldn't have expected anything to be garbage collectable on that line (but then again I don't know much about the details of SQLAlchemy).

Just curious to understand what's happening :)

[Mike Bayer]

Replying to guest:

Thanks for the quick response and fix! Just to understand the implications, you're saying (in CHANGES) that as I did in my work-around. I usually don't chain calls like that anyway but I'm still surprised that 'chaining' vs 'not-chaining' behaves in such a fundamentally different way. Is this due to the way mapped objects work? I wouldn't have expected anything to be garbage collectable on that line (but then again I don't know much about the details of SQLAlchemy).

Just curious to understand what's happening :)

nothing to do with SQLAlchemy, except that SQLA needs a reference to the parent object when an append occurs.

You can try this at home!

class fancylist(list):
    def append(self, obj):
        print "appending an obj!"
        list.append(self, obj)

class Foo(object):
    def __del__(self):
        print "foo is gc'ed !"
    def __init__(self):
        self.bars = fancylist()

def give_me_a_foo():
    return Foo()

give_me_a_foo().bars.append(5)

output:

classics-MacBook-Pro:sqlalchemy classic$ python test.py
foo is gc'ed !
appending an obj!

The "foo is gc'ed" message prints before the "append" message does - Foo is gone by the time you're appending. If anything, the fact that this also happens with SQLA is because we've worked so hard to not get in the way of normal Python gc behavior ;)

[Former user]

Oh, I see, it never occurred to me that first() returns a new object that's not being referenced. In my mind it was more akin to foo0, i.e. a reference to an object retained elsewhere. Thanks for clearing that up!

[Mike Bayer]

Removing milestone: 0.6.7 (automated comment)