sqlalchemy sharding

hibernate is already doing it:

http://www.hibernate.org/hib_docs/shards/reference/en/html/shards-shardstrategy.html

My previous take on this, which is somewhere in the ML (google wont find it), talks about instrumenting the SQL layer in order to make this happen. but in fact, after thinking about this some more and still looking at hibernate, a simpler version of this that works only with ORM/sessions is extremely easy to get started - the attached patch requires some extremely minor hooks within Session, a little bit of "multi-connection" awareness in Mapper (which historically is always the part i haven't wanted to do...but the approach i came up with is fairly optimized), and none within Query.

the approach mimics hibernate's three strategies of "id resolution", "instance resolution", and "query order". from what people tell me theyre looking for, this should meet their needes. obviously the hard part here is to implement reasonable "starter" versions of those functions, but the attached framework, while untested, allows the basic idea to work.

my motivations for simplifying are related to some continued requests for this feature as well as the desire to not overcomplicate it. so im going to try implementing some sample "strategies" and getting some unit test/examples going. It would be nice to release a rudimental version of this in 0.4; then, if people start using it, it will start getting refined as feature/fix requests come in.

Comments (5)