error flag not being cleared

[Former user]

I come across the following error: ArgumentError: Error creating backref .. on relation .. property of that name exists on mapper...

I fix the bug in my code, and use my system's reload mechanism (reloads the module containing my engine, base, metadata, classes, tables, etc), and rerun the code that caused the error, I now get the following error: InvalidRequestError: One or more mappers failed to compile. Exception was probably suppressed within a hasattr() call. Message was: Error creating backref .. on relation ..: property of that name exists on mapper ..

Despite reloading the modules with all my classes etc (and fixed code), this error still persists. It strikes me that an error flag has has been raised that is being repeatedly checked and has not been cleared.

If I call orm.clear_mappers() before reloading the module with my mappings, the problem is resolved. However, if there's an error in reloading this module, my system's reload mechanism will try to restore the module to it's previous state, but restoring all the objects is not enough if their mappers have been cleared.

I'm not sure if this is a bug or not. Is there some flag that should/could be reset that's storing this hasattr error? Should it not be cleared when the mapper's class that the error refers to is deleted (no more refs)? Or maybe there could be some mechanism added in to clear it manually?

[Mike Bayer]

clear_mappers() is what we give you to reset ORM state - _compile_failed is a flag local to a specific mapper so its gone when the mapper is removed. So the answer to your question is "yes" - clear_mappers() clears the hasattr error and everything else.

I'm not sure what the issue is because its unclear what "all the objects" refer to here - if its objects created/loaded from mapped classes, they're all invalid too and you need to blow those away as well. There's no feature that can "fix" an individual mapper from a broken setup - your newly reloaded modules need to call mapper() all over again !

Also Python's "reload" is widely known/accepted to be completely unreliable and I don't know anyone who relies upon it.

[Former user]

(original author: ged) For one, SQLAlchemy-migrate relies on reload (which sadly breaks Elixir in the default configuration).

[Former user]

OK, so assuming I've called clear_mappers(), and I now want to undo that and recreate all my mappings, what do I do? All my classes are using declarative, so I'm cycling through each object and doing some checks (something like isinstance(obj, DeclarativeMeta) and hasattr(obj,"table") dunno if there's a recommended way?), before calling mapper(object, object.table). However this doesn't recreate my relations/dynamic_loaders. I guess this is because normally you'd specify the relations inside the call to mapper(), but I'm relying on the magic of declarative to do that for me. So is there a way to reinitialize the mapper fully or am I doing something wrong?

From the other perspective, is there a way I can make a copy of the mappers stored and create a backup cache myself? Something like sqlalchemy.orm.mapper._mapper_registry? Though I'm not sure how well this would work given the weak references.

[Mike Bayer]

Replying to ged:

For one, SQLAlchemy-migrate relies on reload (which sadly breaks Elixir in the default configuration).

huh ? I use Migrate all the time. If Migrate has managed to write an application that carefully avoids all the serious issues with "reload", good for them.

I'm not saying "reload" should never be used, I'm saying, it's a bad idea to rely upon it or expect it to work perfectly without the need for ugly hacks and workarounds, since it doesn't.

[Mike Bayer]

Unfortunately unless you moved all of your class construction into functions which you can re-call, you'd have to do this all the "normal" way and re-import your modules, or otherwise find some way to re-execute the body of each module (I sort of thought reload() does this....). While it is theoretically possible for the mapper/relation datastructure to be recreated from an existing one, it would be a lot of work and ongoing maintenance to keep such a feature running.

[Mike Bayer]

The python documentation also suggests a re-import() (http://docs.python.org/library/functions.html#reload):

If a module is syntactically correct but its initialization fails, the first import statement for it does not bind its name locally, but does store a (partially initialized) module object in sys.modules. To reload the module you must first import it again (this will bind the name to the partially initialized module object) before you can reload() it.

[Former user]

Replying to zzzeek:

Unfortunately unless you moved all of your class construction into functions which you can re-call, you'd have to do this all the "normal" way and re-import your modules, or otherwise find some way to re-execute the body of each module (I sort of thought reload() does this....). While it is theoretically possible for the mapper/relation datastructure to be recreated from an existing one, it would be a lot of work and ongoing maintenance to keep such a feature running.

You're right that reload does this, however I'm caching the contents of each module, before calling reload() and if any fail I restore the contents of each module. This works great for library modules but not so well for modules that execute code at the module level, (which is the problem we've discovered here), which I'm avoiding except for my declarative classes (simple class definitions don't usually count like this!).

Class construction inside functions would work, thanks for that idea, it's a bit of an ugly solution but better than the status quo!

[Mike Bayer]

I just did a grep of migrate's source for reload(), and other than within unit tests, the only one I found appears to be unnecessary for normal usage, in importpath.py.

the other thing about Migrate is that I don't understand why their documentation calls for a careful re-declaration of all tables (and I would expect this is why Elixir even cares). I've never done that - I just reflect individual Table objects that I wish to maintain. That's the best way to go since you get exactly the state of the database that way. Though I suppose its still an issue for Elixir since users still want to say things like "add_has_many()" instead of "ForeignKeyConstraint.create()".

[Mike Bayer]

Replying to guest:

You're right that reload does this, however I'm caching the contents of each module, before calling reload() and if any fail I restore the contents of each module. This works great for library modules but not so well for modules that execute code at the module level, (which is the problem we've discovered here), which I'm avoiding except for my declarative classes (simple class definitions don't usually count like this!).

the bigger issue is, what problem are you trying to solve ? I still maintain that reload() is relied upon rarely. Those of us who use Pylons with Paster server enjoy the server's "reload everything" feature, which just re-invokes the whole application, for example.

Just for completeness I checked Mako's source to see how I implemented reloading there - I just discard the old module and reload a new one using imp.load_source(). You might want to poke around imp and similar to see if you can construct something more closely tailored towards your needs.

[Former user]

Sorry for the confusion, reload() really has nothing to do with this. Through this discussion I know understand the problem better myself! The issue is reconstructing mappers after a call to clear_mappers() without being able to re-execute the module level code. This is possible by putting code in functions and calling the functions. With standard SQLA this is very simple, the function just needs to include a call to mapper() and pass in the relevant arguments.

However when using declarative, the whole class must be inside a function, and there needs to be code outside the function to assign the returned class to a local variable. In terms of creating reusable code, this would be quite complicated or at least in elegant. This is bringing me to consider moving away from declarative!

Reopening the ticket in the hope that the issue is much better explained now and you understand the problem.

Replying to zzzeek:

While it is theoretically possible for the mapper/relation datastructure to be recreated from an existing one, it would be a lot of work and ongoing maintenance to keep such a feature running.

That said, it might still be the case that this would require significant change and is rare enough to warrant a "wontfix", though maybe you could consider this as something for the future if so, at least!

(As a possible (and potentially horrible - just a suggestion!) fix, the declarative classes could be given a method at the time of the original mapper configuration / class definition, that reinitializes/recreates the class' mapper from some data stored (stored also at the time of mapper config / class def) in the class.)

[Mike Bayer]

Replying to guest:

Reopening the ticket in the hope that the issue is much better explained now and you understand the problem.

I did understand the issue, yes.

That said, it might still be the case that this would require significant change and is rare enough to warrant a "wontfix", though maybe you could consider this as something for the future if so, at least!

(As a possible (and potentially horrible - just a suggestion!) fix, the declarative classes could be given a method at the time of the original mapper configuration / class definition, that reinitializes/recreates the class' mapper from some data stored (stored also at the time of mapper config / class def) in the class.)

right, that would be the "lot of work and ongoing maintenance feature". In particular, things like relation() and column_proprerty() and similar would have to support some kind of memoization, such that declarative would pull of that memoized information out of a mapping and be able to rebuild everything. It would require that declarative can do everything it does in two entirely different ways (which means 2x the test coverage) and its absolutely miles out of scope for it.

so yeah its a "wontfix", if that makes it clearer ;) .