Splitting ORM and Core into seperate packages

[nyov]

Notice: This is a knee-jerk proposal, without much forthought about possibility of implementation.

I feel it would be nice, if at all possible, if SQLAlchemy could have a more obvious split into the abstraction layer / Core and ORM parts, at least providing two seperate packages and maybe a slightly different name for the core part even; having the ORM package depend on the core, obviously.

The reason is that I have found myself using SQLAlchemy Core with alternative strategies, most notably the Alchimia project for using the core abstraction layer with twisted and shunning the ORM, which is so problematic with these programming models.

I think projects like this could benefit from a dedicated Core package, and vice versa. I'm not saying the code parts should deviate apart, not at all, both packages could build from the same source repository.

But more importantly, while the official documentation makes a distinct split between ORM and Core, people casually equate SQLAlchemy with ORM only throughout third party documentation: When looking for solutions to problems, ideas, or code examples, with the restriction of not using the ORM, it's easy to get frustrated when opening up yet another blog post or code snippet going with an ORM solution.

That is of course inevitable when both parts share the same name and one is somewhat "hidden" in the guts of the other. There is just no good way to search the web for solutions with e.g. "SQLAlchemy prepared queries no ORM". All you'll get is "SQLAlchemy ORM" pages ;)

Because of this, some people will not even know that SQLAlchemy is useable as anything else than just an ORM, I know I didn't when first reading about it. That might also explain why the great abstraction layer alone isn't so well known and represented on the web.

All of which is why I feel having a distinct name and package for the Core part (under the same SQLAlchemy umbrella) could be a good strategy for the project, if such a refactoring can be achieved.

Thanks for reading. Cheers

[Mike Bayer]

Obviously I'm familiar with the "people don't know about core" thing. This has been a long running issue since the beginning, and the greatest achievement towards solving it was to create the name "Core". It used to be "SQLAlchemy", and "SQLAlchemy ORM". So I'm not sure what the proposal is here, as AFAICT this actual change, short of changing import namespaces (I guess that's what you're proposing) and/or producing two separate Pypi downloads, has been made.

But the Core / ORM misunderstanding has many variants. The most common is, people think that the moment you use a function to write a SQL query, it's an "ORM". In fact I think that's the majority of folks who don't understand that SQLAlchemy is Core and ORM - those who are checked out enough that they don't really know what ORM means.

For the users who are my targets, e.g. those who can read words and understand them, I've made a huge effort for years to publicize Core, but not only that, to even create the term "Core". Because there was no name "Core" until like 0.5 or so. Hence you see our main documentation page broken into two extremely obvious halves with ORM / Core, people know that Core is there.

I do often think of moving all of core into sqlalchemy.core, and then "sqlalchemy" by itself would just be nothing. If Python actually worked correctly in this regard, they'd be namespace packages. I did dogpile that way; there's dogpile.core and dogpile.cache. Unfortunately namespace packages are problematic in many ways that are difficult to describe and it would be a pretty unpopular change if I made SQLAlchemy this way.

So the fact that namespace packages are still awful makes it less appealing to have sqlalchemy.core / sqlalchemy.orm.

Overall, making that name change at all would be an enormous effort, not in terms of the code itself, but figuring out how to get people migrated towards it. It takes years for me to remove just simple names, like renaming "sqlalchemy.exceptions" to "sqlalchemy.exc" was still getting me bug reports from unmigrated projects for years.

For that reason, at the moment it seems likely that this change would only be something I'd do if I were producing "SQLAlchemy 2.0", e.g. a total compatibility break from 1.0.

At best, I could see introduction of the "core" name and possibly moving the documentation onto it, so that you'd see "from sqlalchemy.core import ..." in docs, but if the goal is googling for blog posts and stack overflow answers, those have an extremely long shelf life. Old stackoverflow answers from 2009, 2010 are actually an issue right now.

[nyov]

Thank you for elaborating, I didn't know much about this background for establishing Core.

As for what the proposal here is, I would think of it like a marketing thing.
Such a split between the two components would likely get better perception if one part would be renamed entirely (establishing a new brand).

I do often think of moving all of core into sqlalchemy.core, and then "sqlalchemy" by itself would just be nothing.

This in itself wouldn't be much different, or help peoples perception, from the current layout (I think), since the split is somewhat in the codebase, but people just copy&paste their imports and don't take much digging into package components or whether they are useable standalone (exaggerating here).

But if "Core" would be a different package, say, SQLElements (as the believed basis of alchemical work), and the ORM would be SQLAlchemy (as people like to perceive), with SQLA depending on SQLE, it would be like moving core into sqlalchemy.core with the benefit of being two distinct packages and also pushing this distinction into peoples minds.

This should also solve the problem of the long shelf-life blogs and SO-answers, where people would have a new name to search for and write about when referring to the abstraction layer.
(Now I saw SQLElements is already a product name of some company, but maybe something else, or maybe they wouldn't mind. Name-space collisions happen all the time in english ;)

If such a split would include splitting into two repositories, it might benefit other projects such as the mentioned Alchimia, in that they could track "Core" as a "shallow" fork with their additions and possibly have some upstream contributions which wouldn't make sense in the current project scope - I'm only speculating here of course. It might also free "Core" to evolve by itself and attract other developers who have no care for ORM or can now stomach the decreased codebase of one of both projects.

In any event, the upgrade path to having two packages might be, starting to depend on the "Core" package, say, sqlelements, from sqlalchemy while retaining imports of core into the current layout in sqlalchemy.
If possible (to split orm/core so completely), this might even be less problem-prone than a rename in-project, as sqlalchemy would still provide the same facade to the users while people migrate to importing from "sqlelements" (or some-such) directly.

Another sales pitch for this: the project just feels massive and complex, I'm not ashamed to say I feel like drowning in all the possibilities and ways of doing things.
And I believe this learning curve can be demotivational and responsible for people just scratching at the surface.
(I have looked at other projects regularly just to get at something I can at least pretend to have an overview of, or take a dig at the code.)

Sure, this is SQL's fault in a big way. But splitting the project right down the middle and thereby setting harder boundaries between components and documentation, changelogs etcetera, could also help a lot IMHO with perceived complexity.

Whether you agree or disagree with my viewpoints here, thank you for a really great software.

[Mike Bayer]

OK, if the proposal is, "create a new project with a new name that is not SQLAlchemy", then that's not something I'm willing to do. the names are "SQLAlchemy Core" and "SQLAlchemy ORM". if you have other concrete proposals, let me know, otherwise I don't see any action here, thanks!

[nyov]

Alright, let me close this then.