allow a migration name to be a sub-package rather than a single file

Issue #91 new
Derek Harland
created an issue

I wonder if it might be a nice enhancement to allow a revision to be a python package instead of being constrained to be a python module.

This may have a few advantages:

  • if a revision consists of a lot of code, or support functions specific to that revision, it might be nicer to be able to split that amongst several files.

  • this is particularly the case for the base revision of a database, which could be large.

  • it might allow for easier storage of symbolic names//tags for a revision. eg if we had a file structure something like:

    • versions/<revision_number>/

      • upgrade/downgrade code goes here
    • versions/<revision_number>/

      • the values of the up/down revisions could be stored in here

      • this means the revision tree could be analysed by only loading for such revisions

      • a symbolic name for the revision could be in here etc.

Comments (7)

  1. Michael Bayer repo owner

    shrugs. seems like it would be a rarely used case. if the upgrade/downgrade code is still in one file, what exactly is in that you wouldn't just have in an file elsewhere? Seems like scaffolding to support migrations belongs outside of the actual migrations directory.

  2. Derek Harland reporter

    Yeah the suggestion would be that would define upgrade and downgrade and would define any properties. Actually, a cfg.ini would be better.

    That might then make it easier to do things like splicing branches, as it would be easier to change the cfg.ini's than to try and regex the modules to find where the up/down revisions are defined.

    And tagged revisions would be quite nice so you can tag "important" release revisions e.g..

    It would also make it possible to eg fold revisions together and combine them into one ... move all of them into a new revision package who's upgrade command just calls all of theirs in succession.

    My use case is that I am establishing a first base revision of an existing database. And this changeset looks huge, especially as it contains a lot of procedures etc that are just inlined sql. To make it more organised/readable I have split it up into submodules (and sql files) in a subdirectory of the versions directory that the base revision can just load. Nicer than a single x-thousand line file.

    In general, for procedures and views it might be easier to implement them by writing the sql into a file and then referencing it in your upgrade ... so it makes sense for those sql files to be isolated somewhere (like a directory for that particular revision).

    Just random thoughts!

  3. Michael Bayer repo owner

    im not understanding most of this, though I understand one of the use cases. If you just go to #4, we don't need to have a big dialogue getting me to understand #1, #2, #3, #5.

    1. why do you need to regex modules ? functions like alembic history show you exactly what is where. also the revision id is in the filename. and how does having a directory with a name change anything vs. a file with a name?

    2. what does "tagging revisions" mean? how does a directory vs. a file have anything to do with that ?

    3. fold revisions - you mean, take two's and stick them into one package ? is there a naming scheme? is this really something anyone needs ?

    4. (huge first revision) OK, that makes more sense, for that I'd just have you make a directory that uses the naming convention of the filename, and the usual alembic boilerplate goes into - and that's all. Your can import anything else within that package using "from . import xyz". A flag can be set up for "alembic revision" so you'd say "alembic revision --package" and it will create <revision_name>/ instead of <revision_name>.py.

    5. I guess this is some argument for breaking a large migration into files again.

  4. Derek Harland reporter

    Point 4

    This is pretty much what I'm suggesting. The current revision file gets replaced by a revision package.

    I guess this would entail changes in the code to how you:

    • search for candidate revision files
    • possibly how you extract revision numbers from them if you did isolate that into a cfg.ini
    • how you load and execute the revision package (can't simply use imp.load_source any more I suspect)

    Point 2

    I just mean allowing a more meaningful symbolic name to a particular revision, similarly to hg tag. So that you could for example create a blank database and then do alembic upgrade -r production_v_2_1

    Don't need directories to do that. It could be put inside the file like a revision number, or held in an external file (like hgtags is)

    Point 1

    I haven't really thought at all about this. Lets say you have two heads and you want to put one after the other. Then you'll need to replace the "down_revision" value in one of the files won't you?

    You could search/replace on the file but thats a bit hairier than doing a direct write to a cfg.ini.

    And there's no guarantee the revision number is in the filename itself, or at least thats my understanding of the current code.

    Point 3

    No, no one needs this!

  5. Log in to comment