load_module in compat.py doesn't handle .pyo

Issue #163 resolved
Greg Yang
created an issue

I'm in a situation where I need to run alembic in a compiled package with all source files stripped away. Right now it's giving me a file not found error with the line

fp = open(path, 'rb')

where path is the full path of where the .py file is supposed to be; so the .pyo file is right in there but load_module doesn't see it.

Comments (8)

  1. Michael Bayer repo owner

    this is not a simple request. the .py files here are located using a directory listing approach. Logic would need to be added that first looks in place for .pyc or .pyo files, then the load module system needs to be modified.

    Also, pep3147 might need to be taken into account; this is where the files would instead be in a __pycache__ directory and there's no simple way to locate these files. The good news is that per pep3147 it seems they will continue to support a .pyc or .pyo file sitting next to the .py (http://www.python.org/dev/peps/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports). However the system you're using to "strip the .py files away" may be too simplistic to work like this - in Python 3 the .pyc file won't just be sitting there normally.

    Here's a partial diff to handle being able to load the .pyc/.pyo. It's quite complicated/hacky for the Py3k case:

    --- a/alembic/compat.py
    +++ b/alembic/compat.py
    @@ -45,14 +45,31 @@ if py2k:
     if py33:
         from importlib import machinery
    +    import os
         def load_module(module_id, path):
    -        return machinery.SourceFileLoader(module_id, path).load_module()
    +        dir_, fname = os.path.split(path)
    +        modname, ext = os.path.splitext(fname)
    +        finder = machinery.FileFinder(dir_,
    +                        (machinery.SourceFileLoader, [".py"]),
    +                        (machinery.SourcelessFileLoader, [".pyc", ".pyo"])
    +                    )
    +        loader, portion = finder.find_loader(modname)
    +        # note we assume loader here, and not portion
    +        if loader is None:
    +            raise ImportError("Can't load file: %s" % path)
    +        else:
    +            # is this supported?  shrugs
    +            loader.name = module_id
    +        return loader.load_module(module_id)
         import imp
    +    import os
         def load_module(module_id, path):
    -        fp = open(path, 'rb')
    +        dir_, fname = os.path.split(path)
    +        modname, ext = os.path.splitext(fname)
    +        fp, pathname, description = imp.find_module(modname, [dir_])
    -            mod = imp.load_source(module_id, path, fp)
    +            mod = imp.load_module(module_id, fp, path, description)
                 if py2k:
                     source_encoding = parse_encoding(fp)
                     if source_encoding:
  2. Michael Bayer repo owner

    of course the above system assumes we're using the finder system, which is sort of the complicated part here. If we fed in completed .py .pyc .pyo paths we could perhaps just call directly to the appropriate loader function.

  3. Michael Bayer repo owner
    • changed status to open

    sorry, I have to alter this so that a flag is required in alembic.ini to enable. reading the .pyc files is very annoying and other users are reporting it as a bug.

  4. Log in to comment