Moving constants to a configuration file breaks py2exe and py2app

Issue #498 wontfix
Jonathan Harris
created an issue

Commit 3cead85 breaks tools that freeze an app which uses openpyxl.

There are two problems:

  • The freezing tools in general can only automatically know about Python source files and dynamic libraries, and can't automatically know about data files.
  • It's difficult for the user to add the data file "manually" to the packaged app; by default the freezing tools bundle Python files into a .zip archive so a construct like os.path.dirname(__file__) doesn't even point to a real directory on disk.

Both py2exe and py2app contain special-case code for handling such data file dependencies (e.g. py2app has a "recipe" for lxml), but I suggest that it would be easier for users if you reverted to the pre-2.3 way of handling these constants rather than wait for py2exe, py2app etc to work around this.

app packaged with py2exe:

...
  File "openpyxl\__init__.pyo", line 12, in <module>
IOError: [Errno 2] No such file or directory: '<exe_path>\\library.zip\\openpyxl\\.constants.json'

app packaged with py2app:

...
  File "openpyxl/__init__.pyo", line 12, in <module>
IOError: [Errno 20] Not a directory: '<app_path>.app/Contents/Resources/lib/python2.7/site-packages.zip/openpyxl/.constants.json'

Comments (22)

  1. Charlie Clark

    Interesting but we need the constants. Being able to build the project and documentation reliably is more important than any downstream packaging concerns. This is really a deficiency of Python's packaging.

  2. Mathieu Gueydan

    My 2 cents on this topic : One may read here that a zip archive is a valid place to find a module (at least python parse zips from import path looking for modules).

    That's why my application (that embed the python interpreter) also includes all the modules it needs (currently about 200 of them) in the same zip file.

    ".constants.json" file doesn't affect me in this context, but the current way to load it does, as it is failing with following error :

    No such file or directory: 'c:\\workplace\\Python35lib.zip\\openpyxl\\.constants.json'
    Traceback (most recent call last):
      File "./DOFormat\entryPoint.py", line 6, in <module>
        from funcTree2Xls import funcTree2Xls
      File "./DOFormat\funcTree2Xls.py", line 2, in <module>
        from openpyxl import load_workbook
      File "<frozen importlib._bootstrap>", line 969, in _find_and_load
      File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
      File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
      File "Lib\openpyxl\__init__.py", line 12, in <module>
    
  3. Steven Vereecken

    Heungsub Lee's suggestion seems ok to me (why does that pose a "problem of non-importability"?), but here is an alternative workaround:

    couldn't you use if not hasattr(sys, 'frozen') when setting the constants? This will only impact frozen applications: the constants will not be available in that case, but they are rather useless in frozen apps anyway (and at the very least it's better than not being able to use openpyxl in frozen apps at all)

  4. Charlie Clark

    @Steven Vereecken This really isn't openpyxl's problem and I don't want to include any code that makes it our problem and that we have to maintain. This includes your suggestion: I have no idea about sys.frozen. Might as well just handle IOError gracefully.

  5. Steven Vereecken

    I understand your point, at the same time, it's unfortunate that people who want to use openpyxl in certain ways (frozen applications) are stuck with this problem, for something that is not "really" a data file needed by the library, but metadata done in an uncommon way. So yes, there is an unsolved problem with packaging data files, but maybe we shouldn't even have to encounter that problem. You might also argue that it's a problem of the documentation generation, and that the openpyxl init.py shouldn't have to contain code to solve that? But I respect your decision.

    You're right about handling IOError vs sys.frozen by the way. That would be a cleaner way of saying "it 's not really needed, no problem if it's missing" rather than making a special case for frozen apps. Kind of an optional dependency for a data file, which does not seem unreasonable perhaps?

  6. Chung-Yi Shen

    You can use hook pyinstaller --onefile --additional-hooks-dir=. xxx.py

    hook-openpyxl.py

    from PyInstaller.utils.hooks import collect_data_files
    
    datas = collect_data_files('openpyxl')
    
  7. Margarethe Meier

    A bit late to the party, but this doesn't work for me:

    Traceback (most recent call last):
      File "<string>", line 10, in <module>
      File "c:\users\name\appdata\local\temp\pip-build-9iggvu\pyinstaller\PyInstaller\loader\pyimod03_importers.py", lin
    e 389, in load_module
      File "site-packages\openpyxl\__init__.py", line 35, in <module>
      File "c:\users\name\appdata\local\temp\pip-build-9iggvu\pyinstaller\PyInstaller\loader\pyimod03_importers.py", lin
    e 389, in load_module
      File "site-packages\openpyxl\workbook\__init__.py", line 5, in <module>
      File "c:\users\name\appdata\local\temp\pip-build-9iggvu\pyinstaller\PyInstaller\loader\pyimod03_importers.py", lin
    e 389, in load_module
      File "site-packages\openpyxl\workbook\workbook.py", line 16, in <module>
      File "c:\users\name\appdata\local\temp\pip-build-9iggvu\pyinstaller\PyInstaller\loader\pyimod03_importers.py", lin
    e 389, in load_module
      File "site-packages\openpyxl\writer\write_only.py", line 23, in <module>
      File "c:\users\name\appdata\local\temp\pip-build-9iggvu\pyinstaller\PyInstaller\loader\pyimod03_importers.py", lin
    e 389, in load_module
      File "site-packages\openpyxl\writer\excel.py", line 36, in <module>
      File "c:\users\name\appdata\local\temp\pip-build-9iggvu\pyinstaller\PyInstaller\loader\pyimod03_importers.py", lin
    e 389, in load_module
      File "site-packages\openpyxl\packaging\extended.py", line 4, in <module>
    ImportError: cannot import name __version__
    

    (at least in the following configuration)

    # This file may be used to create an environment using:
    # $ conda create --name <env> --file <this file>
    # platform: win-32
    et_xmlfile=1.0.1=py27_0
    ipython=2.1.0=py27_2
    jdcal=1.2=py27_0
    lxml=3.5.0=py27_0
    msvc_runtime=1.0.1=vc9_0
    openpyxl=2.4.1=py27_0
    pip=7.1.2=py27_0
    pyreadline=2.1=py27_0
    python=2.7.10=5
    pywin32=220=py27_1
    setuptools=19.2=py27_0
    wheel=0.26.0=py27_1
    

    The correct fix from my point of view is the comment of @Chung-Yi Shen:

    from PyInstaller.utils.hooks import collect_data_files
    
    datas = collect_data_files('openpyxl')
    

    This would fix the issue on pyinstaller side. This has been added for the 3.3 release of pyinstaller (Commit, PR). Its working also for me.

  8. Aleksandr Smyshliaev

    The issue is still there with version 2.5.0a2

    Perhaps openpyxl could repeat the trick used in pytz?

    try:
        from pkg_resources import resource_stream
    except ImportError:
        resource_stream = None
    
    ...
    
        if not os.path.exists(src_file) and resource_stream is not None:
            src = resource_stream(__name__, ".constants.json")
    
  9. Aleksandr Smyshliaev

    Uh. Sorry, but try .. except does not help in this case:

      File "D:\check.py", line 13, in <module>
        import openpyxl
      File "C:\Python27\lib\site-packages\openpyxl\__init__.py", line 29, in <module>
        from openpyxl.workbook import Workbook
      File "C:\Python27\lib\site-packages\openpyxl\workbook\__init__.py", line 5, in <module>
        from .workbook import Workbook
      File "C:\Python27\lib\site-packages\openpyxl\workbook\workbook.py", line 16, in <module>
        from openpyxl.writer.write_only import WriteOnlyWorksheet, save_dump
      File "C:\Python27\lib\site-packages\openpyxl\writer\write_only.py", line 23, in <module>
        from .excel import ExcelWriter
      File "C:\Python27\lib\site-packages\openpyxl\writer\excel.py", line 37, in <module>
        from openpyxl.packaging.extended import ExtendedProperties
      File "C:\Python27\lib\site-packages\openpyxl\packaging\extended.py", line 4, in <module>
        from openpyxl import __version__
    ImportError: cannot import name __version__
    

    The constants are required by openpyxl own modules, but cannot be installed because __init__.py path is not on the file system.

    I cannot even install these constants by monkey patching the loaded module since the module won't load.

  10. Log in to comment