WIP: Don't freeze importlib into base executables

#47 Merged at d5f7109
Repository
takluyver
Branch
thaw-importlib
Repository
anthony_tuininga
Branch
default
Author
  1. Thomas Kluyver
Reviewers
Description

This addresses issue #61, by avoiding freezing importlib into the base executables. Where necessary, it is copied into the zip file instead, so __file__ is set when it is loaded.

Our extension loader needs imp for extension modules, so when one is added to the finder, we include imp as well (which brings in importlib on Python 3.3+). We should check whether older versions of Python need imp to be in the frozen modules; if so, we can keep freezing it for Python < 3.3.

Comments (6)

  1. Jurko Gospodnetić

    I'm just now trying out your changes from this pull request.

    Environment:

    • Windows 7 SP1 x64
    • Python 3.4.0rc3 with modified Lib/distutils/msvc9compiler.py so cx_Freeze build passes
    • Microsoft Visual Studio 2010

    -- 1. -- warning when freezing application

    When freezing a simple application with the following content:

    print("Hello world")
    import asyncio
    import sys
    print(sys.version_info)
    

    I get the following UserWarning:

    C:\Program Files\Python\Python34rc3\lib\site-packages\cx_freeze-4.3.2-py3.4-win-amd64.egg\cx_Freeze\freezer.py:574: UserWarning: Duplicate name: 'importlib/__init__.pyc'
      outFile.writestr(zinfo, data)
    

    -- 2. -- possible error in frozen application's library.zip archive

    Frozen application's library.zip file now contains the importlib package and the imp.pyc module, however its importlib/__init__.py module seems to have been added to the ZIP file twice. This has been observed by opening the ZIP archive using 7-Zip (version 9.20).

    -- 3. -- collections.abc not importable from a frozen application - separate issue

    In order to test this on a more complex application now, where we originally encountered the problem (uses extension modules implemented in C++), I also needed to patch up cx_Freeze's finder.py module some more and add:

            if sys.version_info[:2] >= (3, 4):
                self.IncludeModule("collections.abc")
    

    to its ModuleFinder._AddBaseModules() method, but I'll open a separate issue for that.

    -- 4. -- original issue resolved successfully

    Running the frozen application no longer reports an error when importing the importlib package. This has been tested both with a simple hello_world.py application as above, and with a more complex application using external Python extension modules.

    Hope this helps.

    Best regards, Jurko Gospodnetić

  2. Juraj Ivancic

    This patch does not work in case import importlib is found in user code. If you try to freeze a one-line script importing importlib you will get the following traceback:

    Traceback (most recent call last):
      File "setup.py", line 7, in <module>
        Executable('test.py', base='Console')
      File "D:\Python\Python34\lib\site-packages\cx_freeze-4.3.2-py3.4-win32.egg\cx_Freeze\dist.py", line 362, in setup
        distutils.core.setup(**attrs)
      File "D:\Python\Python34\lib\distutils\core.py", line 149, in setup
        dist.run_commands()
      File "D:\Python\Python34\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "D:\Python\Python34\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "D:\Python\Python34\lib\site-packages\cx_freeze-4.3.2-py3.4-win32.egg\cx_Freeze\dist.py", line 232, in run
        freezer.Freeze()
      File "D:\Python\Python34\lib\site-packages\cx_freeze-4.3.2-py3.4-win32.egg\cx_Freeze\freezer.py", line 582, in Freeze
        self.compress, self.copyDependentFiles)
      File "D:\Python\Python34\lib\site-packages\cx_freeze-4.3.2-py3.4-win32.egg\cx_Freeze\freezer.py", line 500, in _WriteModules
        itemsToSort.sort()
    TypeError: unorderable types: Module() < Module()
    

    What happens is that removing importlib and importlib._bootstrap from self._modules, and later finding these imports in user code will result in having these modules twice in self.modules (self being an instance of ModuleFinder). This will cause the above error later on - itemsToSort will contains tuples with conflicting first element, and the second one (Module) is not orderable.

    I worked around this by adding the following:

                # importlib itself must not be frozen
                # -------
                self.modules.remove(self._modules["importlib"])
                self.modules.remove(self._modules["importlib._bootstrap"])
                # -------
                del self._modules["importlib"]
                del self._modules["importlib._bootstrap"]
    

    With this, freezing works regardless of importlib in user code.

    Tested with Python 3.4 on Windows.