Issue #163 resolved

load_module in compat.py doesn't handle .pyo

eulerreich avatareulerreich created an issue

I'm in a situation where I need to run alembic in a compiled package with all source files stripped away. Right now it's giving me a file not found error with the line

fp = open(path, 'rb')

where path is the full path of where the .py file is supposed to be; so the .pyo file is right in there but load_module doesn't see it.

Comments (8)

  1. Mike Bayer

    this is not a simple request. the .py files here are located using a directory listing approach. Logic would need to be added that first looks in place for .pyc or .pyo files, then the load module system needs to be modified.

    Also, pep3147 might need to be taken into account; this is where the files would instead be in a __pycache__ directory and there's no simple way to locate these files. The good news is that per pep3147 it seems they will continue to support a .pyc or .pyo file sitting next to the .py (http://www.python.org/dev/peps/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports). However the system you're using to "strip the .py files away" may be too simplistic to work like this - in Python 3 the .pyc file won't just be sitting there normally.

    Here's a partial diff to handle being able to load the .pyc/.pyo. It's quite complicated/hacky for the Py3k case:

    --- a/alembic/compat.py
    +++ b/alembic/compat.py
    @@ -45,14 +45,31 @@ if py2k:
    
     if py33:
         from importlib import machinery
    +    import os
         def load_module(module_id, path):
    -        return machinery.SourceFileLoader(module_id, path).load_module()
    +        dir_, fname = os.path.split(path)
    +        modname, ext = os.path.splitext(fname)
    +        finder = machinery.FileFinder(dir_,
    +                        (machinery.SourceFileLoader, [".py"]),
    +                        (machinery.SourcelessFileLoader, [".pyc", ".pyo"])
    +                    )
    +        loader, portion = finder.find_loader(modname)
    +        # note we assume loader here, and not portion
    +        if loader is None:
    +            raise ImportError("Can't load file: %s" % path)
    +        else:
    +            # is this supported?  shrugs
    +            loader.name = module_id
    +        return loader.load_module(module_id)
     else:
         import imp
    +    import os
         def load_module(module_id, path):
    -        fp = open(path, 'rb')
    +        dir_, fname = os.path.split(path)
    +        modname, ext = os.path.splitext(fname)
    +        fp, pathname, description = imp.find_module(modname, [dir_])
             try:
    -            mod = imp.load_source(module_id, path, fp)
    +            mod = imp.load_module(module_id, fp, path, description)
                 if py2k:
                     source_encoding = parse_encoding(fp)
                     if source_encoding:
    
  2. Mike Bayer

    of course the above system assumes we're using the finder system, which is sort of the complicated part here. If we fed in completed .py .pyc .pyo paths we could perhaps just call directly to the appropriate loader function.

  3. Mike Bayer
    • changed status to open

    sorry, I have to alter this so that a flag is required in alembic.ini to enable. reading the .pyc files is very annoying and other users are reporting it as a bug.

  4. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.