Issue #43 resolved

Import errors when using cx_Freeze with specific scipy modules

azylstra
created an issue

Following up on StackOverflow issue

My project uses scipy in a few places and after freezing I would get a variety of import errors. Those seemed to be naming issues (see in the StackOverflow link, the executable was looking for _csr when scipy.sparse.sparsetools._csr was what was packaged).

Unfortunately the project cannot be public but I tried to replicate the issue in an example. The _csr problem appears to be coming from a piece of code that uses import scipy.optimize. Other modules when imported specifically cause other errors. For example, here is some logged console output from an executable:

10/27/13 9:58:08.856 AM Python[8116]: Start cx_Freeze test
10/27/13 9:58:08.959 AM Python[8116]: Imported scipy OK
10/27/13 9:58:08.961 AM Python[8116]: Uh oh, failed to import scipy.interpolate: No module named 'scipy.special._ufuncs_cxx'
10/27/13 9:58:08.962 AM Python[8116]: Uh oh, failed to import scipy.integrate: No module named 'scipy.special._ufuncs_cxx'
10/27/13 9:58:09.020 AM Python[8116]: Uh oh, failed to import scipy.optimize: No module named '_csr'
10/27/13 9:58:09.023 AM Python[8116]: Uh oh, failed to import scipy.stats: No module named 'scipy.special._ufuncs_cxx'
10/27/13 9:58:09.023 AM Python[8116]: cx_Freeze test completed

I've attached a zip of the setup.py and python code used to generate that output, along with a text file containing the console output of the build, and the entire build folder.

Here are the details of my system:

  • OS X 10.9
  • python 3.3.2
  • cx_Freeze 4.3.2
  • scipy 0.13.0

Comments (39)

  1. azylstra reporter

    FYI I wrote the following rules as a work-around to get scipy.interpolate to work:

    "include_files": [('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csr.so','_csr.so'),
    ('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csc.so','_csc.so'),
    ('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_coo.so','_coo.so'),
    ('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_dia.so','_dia.so'),
    ('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_bsr.so','_bsr.so'),
    ('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csgraph.so','_csgraph.so')]
    
  2. Martin Fitzpatrick

    Just to add I'm also having a similar problem on Windows. Using anything that imports scipy.optimize results in it breaking at from scipy.optimize import minpack2 with ImportError: DLL load failed: The specified module could not be found.

    Strangely though if I look in the library.zip there is a minpack2.pyc file.

    (Later...)

    Based on what azylstra wrote above I realised it's not the .pyc files it's looking for but the .pyds (the Windows equivalent of .so). So a simple hack to get around this problem for the particular modules experience them was:

        import os, glob2, numpy, scipy, nmrglue
        explore_dirs = [
            os.path.dirname(numpy.__file__),
            os.path.dirname(scipy.__file__),
            os.path.dirname(nmrglue.__file__),
        ]
    
        files = []
        for d in explore_dirs:
            files.extend( glob2.glob( os.path.join(d, '**', '*.pyd') ) )
    
        # Now we have a list of .pyd files; iterate to build a list of tuples into 
        # include files containing the source path and the basename
        for f in files:
            build_all['include_files'].append( (f, os.path.basename(f) ) )
    

    This is an ugly hack - copying all .pyd files from each module regardless of whether they are used - and results in bigger packages. But it works - thought it may be of use to others.

  3. Thomas Kluyver

    Martin Fitzpatrick : the .pyc file is a wrapper to load the extension module - it should be copied into the build directory (outside the zip file) as something like scipy.optimize.minpack2.pyd - is that there? Can you show the log from freezing?

    I think the "DLL load failed" message means it's trying to dynamically load another DLL, which may not have been copied. You might be able to use process explorer on a Python process with that module loaded to see what else it loads.

  4. Ian Bell

    +1 on this issue.

    I have heaps of problems with scipy and cx_Freeze. I get the same error with

    scipy.special._ufuncs_cxx

    and

    scipy.sparse.csgraph._validation

    not being found. It seems there is something wrong with the search for all things that are part of a package.

  5. Thomas Kluyver

    I think the trouble is actually with SWIG. It does some clever things to try to do relative imports in a backwards compatible way - see this function in Scipy: https://github.com/scipy/scipy/blob/master/scipy/sparse/sparsetools/csr.py#L11

    That function won't work in a frozen application. The critcal part is the fallback on line 18 - if I change that to from . import _csr it works (until the next such import - that autogenerated function exists in several places).

    The code is generated here in SWIG.

    I'm filing an issue on SWIG to bring this to their attention.

  6. Marcin Bu

    Facing the same problem, so +1.

    Configuration: - Windows 7 x64 - Python 2.7.6 x64 - SciPy 0.13 x64 - cx_Freeze 4.3.2

    Script (example from scipy's documentation):

    import numpy as np
    from scipy.interpolate import griddata
    
    
    def func(x, y):
        return x * (1 - x) * np.cos(4 * np.pi * x) * np.sin(4 * np.pi * y ** 2) ** 2
    
    
    grid_x, grid_y = np.mgrid[0:1:100j, 0:1:200j]
    points = np.random.rand(1000, 2)
    values = func(points[:, 0], points[:, 1])
    grid_z0 = griddata(points, values, (grid_x, grid_y), method='nearest')
    

    Build command: C:\python27\scripts\cxfreeze cxTest.py --target-dir C:\temp\dist

    Error: Traceback (most recent call last): File "C:\Python27\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module> exec code in m.dict File "cxTest.py", line 2, in <module> File "C:\Python27\lib\site-packages\scipy\interpolate__init.py", line 156, in <module> from .interpolate import * File "C:\Python27\lib\site-packages\scipy\interpolate\interpolate.py", line 12, in <module> import scipy.special as spec File "C:\Python27\lib\site-packages\scipy\special__init.py", line 531, in <module> from ._ufuncs import * File "ExtensionLoader_scipy_specialufuncs.py", line 22, in <module> File "ExtensionLoader_scipy_specialufuncs.py", line 14, in bootstrap ImportError: DLL load failed: The specified module could not be found.

  7. BubuIIC

    Another report here. I'm trying to do the following:

    from scipy import linalg
    

    And get the following error:

      from scipy import linalg
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1616, in _handle _fromlist
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 313, in _call_with_frames_removed
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1567, in _find_and_load
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1534, in _find_and_load_unlocked
      File "C:\Python33\lib\site-packages\scipy\linalg\__init__.py", line 158, in <module>
        from .basic import *
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1567, in _find_and_load
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1534, in _find_and_load_unlocked
      File "C:\Python33\lib\site-packages\scipy\linalg\basic.py", line 17, in <module>
        from scipy.linalg import calc_lwork
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1616, in _handle_fromlist
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 313, in _call_with_frames_removed
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1567, in _find_and_load
      File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1534, in _find_and_load_unlocked
      File "ExtensionLoader_scipy_linalg_calc_lwork.py", line 22, in <module>
      File "ExtensionLoader_scipy_linalg_calc_lwork.py", line 14, in __bootstrap__
    ImportError: DLL load failed: The specified module could not be found.
    

    The scipy.linalg.calc_lwork.pyd exists in the build directory.

    Is there any workaround or fix for this?

    Edit: Ok, I figured out which dll's were missing. turns out these actually belong to numpy and not Scipy. The .pyd file listed above needs libifcoremd.dll which then needs libmmd.dll. If I copy these two from the numpy install to the build dir, the frozen app works. (Found this out using http://www.dependencywalker.com/).

    So responsible for this is still scipy which is doing strange imports? Or can this be fixed in cx_freeze? How can I add the .dll's to the build process from cx_freeze without specifying the absolute path?

  8. Thomas Kluyver

    That last one sounds like a different problem - it appears cx_Freeze isn't finding some DLLs properly. They might be loaded dynamically, in which case a hook will be needed.

  9. BubuIIC

    I have looked into cx_freeze hooks, but I have no idea on what I should hook (I guess on scipy.linalg, but that doesn't seem to work). And then, how I can point cx_freeze to the right .dlls.

    Any more hints on this?

  10. Thomas Kluyver

    A load_scipy_linalg hook should be right. If the DLLs are importable Python modules, you should be able to add them with finder.IncludeModule() in the hook. Otherwise, you'll need to find where they're located and use finder.IncludeFiles().

  11. Thomas Kluyver

    Can someone try with the beta version of scipy 0.14? They seem to be moving away from SWIG, which I think will fix the issue, though I don't know if they've moved entirely away from it yet.

  12. Marcin Bu

    Copying all PYD, PYC, PYO and DLL files didn't help, although the error message is now different - "missing _validate".

    Will try scipy 0.14 later and report on the result.

  13. Marcin Bu

    Ok, got it!

    Two things I had to do: 1. copy all the numpy DLLs (from c:\Python27\Lib\site-packages\numpy\core), regardless of using scipy 13 or 14 2. add the following to my code:

    def dependencies_for_myprogram():
        from scipy.sparse.csgraph import _validation
        from scipy.special import _ufuncs_cxx
        from matplotlib.backends import backend_tkagg #this one is for matplotlib
    

    Now I have a new problem with multiprocessing, which opens multiple windows of my app...

    UPDATE: multiprocessing issue sorted out by:

    if __name__ == '__main__':
        # freeze_support must be the first line
        multiprocessing.freeze_support()
    

    :)

  14. Thomas Kluyver

    Probably to the build directory, next to your exe.

    My approach now to solving this, and many other issues, is not to store packages inside a zip file by default (see pull request #73). If packages are loaded from regular files and folders, this kind of issue shouldn't come up. You'll still be able to tell it to zip up packages you know work that way.

  15. Franco C

    same problem here, updating to scipy 0.16.0 from gohlke pythonlibs prevents cx_freeze to build exe. If "scipy" is added to packages options this error is raised: ImportError: No module named 'scipy'

    If no scipy is added to packages options, build runs until end but upon executing .exe an error window is raised with: ImportError: No module named 'scipy.config'

  16. Keegan Owsley

    To get latest scipy 0.16 package from gohlke to freeze, I had to change line 548 of hooks.py to read:

    finder.IncludePackage("scipy._lib")
    

    Note the extra underscore.

  17. Keegan Owsley

    It appears so. I still need to include the libmmd and libifcoremd binaries as mentioned elsewhere in the thread, and list "scipy" manually in the packages as a build option.

    Edit: I am only using a few submodules, so I don't know if (eg) scipy.sparse.csgraph._validation works or not. I also have not tried just omitting the line entirely.

  18. Florian Zierer

    Ok, so I was updating to scipy 0.16.1 and encountered some combination of the above posted errors.

    I had to change the hooks.py as Keegan Owsley wrote and to include scipy as a package. Additionally I also had the problem that the two numpy dlls (libifcoremd.dll and libmmd.dll) where not included. It helped to import numpy at the beginning of the setup.py

  19. Jürg Rast

    I expirienced the same problems and fixed it by patching the hooks function and including numpy on top of the setup.py. I think patching the hook function in setup.py is nicer than editing the code directly:

    from cx_Freeze import hooks
    
    def load_scipy_patched(finder, module):
        """the scipy module loads items within itself in a way that causes
            problems without the entire package and a number of other subpackages
            being present."""
        finder.IncludePackage("scipy._lib")  # Changed include from scipy.lib to scipy._lib
        finder.IncludePackage("scipy.misc")
    
    hooks.load_scipy = load_scipy_patched
    
  20. henfri

    Hello,

    I spent a day on this topic recently. Here my summary: I am using cx_freeze 5 and anaconda with python 2.7.

    I found that the dlls for the following packages were missing in my package: -PyQt -vtk -numpy -scipy -shapely

    Thus we can conclude that this is not an isolated numpy issue. It could be related to anaconda though. Building the same application with pythonxy (not sure which cx_freeze version I had used though; thus I cannot exclude it to be related to a different cx_freeze version)

    The first problem was, that the qtcore4.dll and qtgui4.dll were not stored in /bin/PyQt/ but in /bin/, whereas the package was -according to Process-Monitor - searching in /bin/PyQt/. Moving the dlls fixed this issue.

    Next was numpy. For that, several mkl_* and a libiompd5.dll were completely missing (i.e. not just in the wrong folder).

    Next was vtk where I needed all the vtk_*.dll

    In the end we concluded that it is not efficient to create a list of all missing dlls (plus debugging any future missing dlls). Instead, we decided to accept a bit of overhead:

    In the setup.py we used the glob package to create a list of all dlls in Lib/bin (or was it /Library/bin), and added these to the 'include' list to be put in /bin/ (i.e. we did NOT sort them into the appropriate directories like pyqt, vtk, ...

    The we added to the python code of the program a check:

    if getattr(sys, 'frozen', False):
       sys.path.insert(0, join(dirname(sys.executable),'bin')
    

    This adds the path into which we copied all dlls to the path and the application should search there for it. This last point did not yet work out though. Thus we had to add the path to a batch-file before running the application

    PATH=%WINDIR%\System32;c:\path\to\frozen\application\bin
    

    That's not nice, but it was late and we were happy to have a solution. I hope, this post will help someone with the same problem.

    I'd like to stress that this issue is not fixed. cx_freeze 5.0 with anaconda will not pick the right dlls and/or put them into the wrong folders for at least the list of the packages I listed above.

    Regards, Hendrik

  21. Jumabek Alikhanov

    I had a similar problem and found a simple solution.

    In short,
    I removed anything that has to do with numpy from C:\Anaconda2\Lib\site-packages

    Here is my whole scenario:

    • I converted .py to .exe using cx_freeze

    • Check the exe by running on my own PC. Everything is fine.

    • Run it on collegues' PC, got the error:

    Traceback (most recent call last):
      File "C:\Anaconda2\lib\site-packages\cx_Freeze\initscripts\__startup__.py", li
    ne 12, in <module>
      File "C:\Anaconda2\lib\site-packages\cx_Freeze\initscripts\Console.py", line 2
    4, in <module>
      File "PatchBasedObjectAnnotator.py", line 1, in <module>
      File "C:\Anaconda2\lib\site-packages\numpy\__init__.py", line 142, in <module>
    
      File "C:\Anaconda2\lib\site-packages\numpy\add_newdocs.py", line 13, in <modul
    e>
      File "C:\Anaconda2\lib\site-packages\numpy\lib\__init__.py", line 8, in <modul
    e>
      File "C:\Anaconda2\lib\site-packages\numpy\lib\type_check.py", line 11, in <mo
    dule>
      File "C:\Anaconda2\lib\site-packages\numpy\core\__init__.py", line 24, in <mod
    ule>
    ImportError:
    Importing the multiarray numpy extension module failed.  Most
    likely you are trying to import a failed build of numpy.
    If you're working with a numpy git repo, try `git clean -xdf` (removes all
    files not under version control).  Otherwise reinstall numpy.
    
    • Did a lot of google search, and been here as well.

    • Removed anything that has to do with numpy from C:\Anaconda2\Lib\site-packages

    • Confirmed, exe file is running on other PC's without any problem.

  22. Romain H

    Hello, I am really interested in the last comment by Jumabek Alikhanov since I exactly have the same problem, but cannot find a solution to solve it.

    You said that you "Removed anything that has to do with numpy from C:\Anaconda2\Lib\site-packages", but I do need Numpy in my .exe, and if I remove it, it fails already at the freezing part. Can you develop your solution? Thanks,

  23. Romain H

    After a Numpy update (1.13.1), the error was mentionning a DLL error. Dependency Walker saved my day: it was looking for the "libiomp5md.dll" which was missing on other computers. Once I added this dll to the include_files, everything went fine.

  24. Log in to comment