memory leak by using relation with backref

[Former user]

When I try to use objects linked by relation with backref memory is not released neither after objectstore.flush() nor objectstore.clear() nor objectstore.expunge() nor gc.collect() When I break backref relation memory cleared by gc.collect() good. Bug reproduced by usage User/Address objects from your tests.

for i in xrange(1000):
  objectstore.push_session(objectstore.Session())
  for j in xrange(1000):
    u = User()
    u.name = str(randint(0, 100000000))
    for k in xrange(10):
      a = Address()
      a.email_address = str(randint(0, 100000000))
      a.user = u
  objectstore.pop_session() 
  gc.collect()

with backrefs memory grows intensivelly and does not released

[Mike Bayer]

thanks for the full test ;)

anyway, this issue goes around and around....everyone has an opinion on how objects should work, and the solution for each person's issue breaks the other one. mainly #129. but this time, i bit the bullet and read the pickle docs (pickle? huh?) and so now i think its solved.

here are the conflicting requirements:

• object instances need to have associated information about their attributes, since SA uses this to track changes. the state of a single instance's attribute is stored in an object called a ManagedAttribute.
• the ManagedAttribute must be able to locate the object to which it is attached.
• this association cannot be a circular reference on an object instance by default, since people really like cPythons behavior that non-circular objects get zapped immediately without having the garbage collector run (they really like it...).
• you cant put a weakref directly on the instance anywhere since objects need to be pickled and you cant pickle weakrefs.

So, Ive been playing on and off with all these various module-level Weak(Key|Value)Dictionaries to associate the instances with ManagedAttributes. But from this issue, it is now clear that when you create a circular reference between two instances, i.e. a.b = b, b.a = a, the garbage collector's "locate the circular reference" logic gets screwed up if those instances have strong references elsewhere, namely in the WeakKeyDictionary I was using, since one side of a WeakXXXXDictionary always has a strong reference.

So, big duh, I took all that crap out, spent 5 minutes re-reading the pickle docs and put __getstate__ and __setstate__ methods on the base ManagedAttribute class, problem solved.

tests for this:

• test/attributes.py - a unittest that tests that managed objects can be pickled
• test/massload.py - a unittest loads 25000 objects from the database, 100 at a time, and insures only 100 objects stay in memory at one time
• test/masscreate2.py - a non-unit test (just a python script) that runs a simplified behavior of the test case attached to this ticket; watch "top" while it runs.

changeset changeset:1345