Import errors when using cx_Freeze with specific scipy modules

Issue #43 resolved
azylstra
created an issue

Following up on StackOverflow issue

My project uses scipy in a few places and after freezing I would get a variety of import errors. Those seemed to be naming issues (see in the StackOverflow link, the executable was looking for _csr when scipy.sparse.sparsetools._csr was what was packaged).

Unfortunately the project cannot be public but I tried to replicate the issue in an example. The _csr problem appears to be coming from a piece of code that uses import scipy.optimize. Other modules when imported specifically cause other errors. For example, here is some logged console output from an executable:

10/27/13 9:58:08.856 AM Python[8116]: Start cx_Freeze test
10/27/13 9:58:08.959 AM Python[8116]: Imported scipy OK
10/27/13 9:58:08.961 AM Python[8116]: Uh oh, failed to import scipy.interpolate: No module named 'scipy.special._ufuncs_cxx'
10/27/13 9:58:08.962 AM Python[8116]: Uh oh, failed to import scipy.integrate: No module named 'scipy.special._ufuncs_cxx'
10/27/13 9:58:09.020 AM Python[8116]: Uh oh, failed to import scipy.optimize: No module named '_csr'
10/27/13 9:58:09.023 AM Python[8116]: Uh oh, failed to import scipy.stats: No module named 'scipy.special._ufuncs_cxx'
10/27/13 9:58:09.023 AM Python[8116]: cx_Freeze test completed


I've attached a zip of the setup.py and python code used to generate that output, along with a text file containing the console output of the build, and the entire build folder.

Here are the details of my system:

• OS X 10.9
• python 3.3.2
• cx_Freeze 4.3.2
• scipy 0.13.0

1. reporter

FYI I wrote the following rules as a work-around to get scipy.interpolate to work:

"include_files": [('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csr.so','_csr.so'),
('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csc.so','_csc.so'),
('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_coo.so','_coo.so'),
('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_dia.so','_dia.so'),
('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_bsr.so','_bsr.so'),
('/usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csgraph.so','_csgraph.so')]


Just to add I'm also having a similar problem on Windows. Using anything that imports scipy.optimize results in it breaking at from scipy.optimize import minpack2 with ImportError: DLL load failed: The specified module could not be found.

Strangely though if I look in the library.zip there is a minpack2.pyc file.

(Later...)

Based on what azylstra wrote above I realised it's not the .pyc files it's looking for but the .pyds (the Windows equivalent of .so). So a simple hack to get around this problem for the particular modules experience them was:

    import os, glob2, numpy, scipy, nmrglue
explore_dirs = [
os.path.dirname(numpy.__file__),
os.path.dirname(scipy.__file__),
os.path.dirname(nmrglue.__file__),
]

files = []
for d in explore_dirs:
files.extend( glob2.glob( os.path.join(d, '**', '*.pyd') ) )

# Now we have a list of .pyd files; iterate to build a list of tuples into
# include files containing the source path and the basename
for f in files:
build_all['include_files'].append( (f, os.path.basename(f) ) )


This is an ugly hack - copying all .pyd files from each module regardless of whether they are used - and results in bigger packages. But it works - thought it may be of use to others.

@Martin Fitzpatrick : the .pyc file is a wrapper to load the extension module - it should be copied into the build directory (outside the zip file) as something like scipy.optimize.minpack2.pyd - is that there? Can you show the log from freezing?

I think the "DLL load failed" message means it's trying to dynamically load another DLL, which may not have been copied. You might be able to use process explorer on a Python process with that module loaded to see what else it loads.

+1 on this issue.

I have heaps of problems with scipy and cx_Freeze. I get the same error with

scipy.special._ufuncs_cxx

and

scipy.sparse.csgraph._validation

not being found. It seems there is something wrong with the search for all things that are part of a package.

I think the trouble is actually with SWIG. It does some clever things to try to do relative imports in a backwards compatible way - see this function in Scipy: https://github.com/scipy/scipy/blob/master/scipy/sparse/sparsetools/csr.py#L11

That function won't work in a frozen application. The critcal part is the fallback on line 18 - if I change that to from . import _csr it works (until the next such import - that autogenerated function exists in several places).

The code is generated here in SWIG.

I'm filing an issue on SWIG to bring this to their attention.

Great - thanks for the feedback. Would be great to get this working.

Facing the same problem, so +1.

Configuration: - Windows 7 x64 - Python 2.7.6 x64 - SciPy 0.13 x64 - cx_Freeze 4.3.2

Script (example from scipy's documentation):

import numpy as np
from scipy.interpolate import griddata

def func(x, y):
return x * (1 - x) * np.cos(4 * np.pi * x) * np.sin(4 * np.pi * y ** 2) ** 2

grid_x, grid_y = np.mgrid[0:1:100j, 0:1:200j]
points = np.random.rand(1000, 2)
values = func(points[:, 0], points[:, 1])
grid_z0 = griddata(points, values, (grid_x, grid_y), method='nearest')


Build command: C:\python27\scripts\cxfreeze cxTest.py --target-dir C:\temp\dist

Error: Traceback (most recent call last): File "C:\Python27\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module> exec code in m.dict File "cxTest.py", line 2, in <module> File "C:\Python27\lib\site-packages\scipy\interpolate__init.py", line 156, in <module> from .interpolate import * File "C:\Python27\lib\site-packages\scipy\interpolate\interpolate.py", line 12, in <module> import scipy.special as spec File "C:\Python27\lib\site-packages\scipy\special__init.py", line 531, in <module> from ._ufuncs import * File "ExtensionLoader_scipy_specialufuncs.py", line 22, in <module> File "ExtensionLoader_scipy_specialufuncs.py", line 14, in bootstrap ImportError: DLL load failed: The specified module could not be found.

Another report here. I'm trying to do the following:

from scipy import linalg


And get the following error:

  from scipy import linalg
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1616, in _handle _fromlist
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 313, in _call_with_frames_removed
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1567, in _find_and_load
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1534, in _find_and_load_unlocked
File "C:\Python33\lib\site-packages\scipy\linalg\__init__.py", line 158, in <module>
from .basic import *
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1567, in _find_and_load
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1534, in _find_and_load_unlocked
File "C:\Python33\lib\site-packages\scipy\linalg\basic.py", line 17, in <module>
from scipy.linalg import calc_lwork
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1616, in _handle_fromlist
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 313, in _call_with_frames_removed
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1567, in _find_and_load
File "C:\Python\64-bit\3.3\lib\importlib\_bootstrap.py", line 1534, in _find_and_load_unlocked
File "ExtensionLoader_scipy_linalg_calc_lwork.py", line 22, in <module>
File "ExtensionLoader_scipy_linalg_calc_lwork.py", line 14, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.


The scipy.linalg.calc_lwork.pyd exists in the build directory.

Is there any workaround or fix for this?

Edit: Ok, I figured out which dll's were missing. turns out these actually belong to numpy and not Scipy. The .pyd file listed above needs libifcoremd.dll which then needs libmmd.dll. If I copy these two from the numpy install to the build dir, the frozen app works. (Found this out using http://www.dependencywalker.com/).

So responsible for this is still scipy which is doing strange imports? Or can this be fixed in cx_freeze? How can I add the .dll's to the build process from cx_freeze without specifying the absolute path?

That last one sounds like a different problem - it appears cx_Freeze isn't finding some DLLs properly. They might be loaded dynamically, in which case a hook will be needed.

I have looked into cx_freeze hooks, but I have no idea on what I should hook (I guess on scipy.linalg, but that doesn't seem to work). And then, how I can point cx_freeze to the right .dlls.

Any more hints on this?

A load_scipy_linalg hook should be right. If the DLLs are importable Python modules, you should be able to add them with finder.IncludeModule() in the hook. Otherwise, you'll need to find where they're located and use finder.IncludeFiles().

I'm confused now... so I can get this to work manually somehow, right? How? I'm on Windows with Python 2.7.6 x64 and I'm getting this:

Help!

Can someone try with the beta version of scipy 0.14? They seem to be moving away from SWIG, which I think will fix the issue, though I don't know if they've moved entirely away from it yet.

Copying all PYD, PYC, PYO and DLL files didn't help, although the error message is now different - "missing _validate".

Will try scipy 0.14 later and report on the result.

I take it you mean _validation, if you notice my script, I added that to includes :

includes = ['numpy','scipy.sparse.csgraph._validation']

Yep, validate sorted out. with scipy 0.14 I don't need to copy any DLLs, yet I still get "importerror no module named _ufuncs_cxx"...

Gah, it's one extension module trying to load another extension module. I don't know quite what to do about that.

Ok, got it!

Two things I had to do: 1. copy all the numpy DLLs (from c:\Python27\Lib\site-packages\numpy\core), regardless of using scipy 13 or 14 2. add the following to my code:

def dependencies_for_myprogram():
from scipy.sparse.csgraph import _validation
from scipy.special import _ufuncs_cxx
from matplotlib.backends import backend_tkagg #this one is for matplotlib


Now I have a new problem with multiprocessing, which opens multiple windows of my app...

UPDATE: multiprocessing issue sorted out by:

if __name__ == '__main__':
# freeze_support must be the first line
multiprocessing.freeze_support()


:)

I've fixed it (problem with _ufuncs) by coping libifcoremd.dll and libmmd.dll from the numpy\core directory.

Where do you copy them to exactly? I'm getting this problem as well.

Probably to the build directory, next to your exe.

My approach now to solving this, and many other issues, is not to store packages inside a zip file by default (see pull request #73). If packages are loaded from regular files and folders, this kind of issue shouldn't come up. You'll still be able to tell it to zip up packages you know work that way.

In current version of cx_freeze, at least for my uses, just putting "scipy" into the packages got rid of this issue...

They'll only be DLLs on Windows. On Linux, the equivalent files are .so, and on Mac they're .dylib, I think.

same problem here, updating to scipy 0.16.0 from gohlke pythonlibs prevents cx_freeze to build exe. If "scipy" is added to packages options this error is raised: ImportError: No module named 'scipy'

If no scipy is added to packages options, build runs until end but upon executing .exe an error window is raised with: ImportError: No module named 'scipy.config'

To get latest scipy 0.16 package from gohlke to freeze, I had to change line 548 of hooks.py to read:

finder.IncludePackage("scipy._lib")


Note the extra underscore.

Did this freeze the entire scipy library?

It appears so. I still need to include the libmmd and libifcoremd binaries as mentioned elsewhere in the thread, and list "scipy" manually in the packages as a build option.

Edit: I am only using a few submodules, so I don't know if (eg) scipy.sparse.csgraph._validation works or not. I also have not tried just omitting the line entirely.

Ok, so I was updating to scipy 0.16.1 and encountered some combination of the above posted errors.

I had to change the hooks.py as @Keegan Owsley wrote and to include scipy as a package. Additionally I also had the problem that the two numpy dlls (libifcoremd.dll and libmmd.dll) where not included. It helped to import numpy at the beginning of the setup.py

I expirienced the same problems and fixed it by patching the hooks function and including numpy on top of the setup.py. I think patching the hook function in setup.py is nicer than editing the code directly:

from cx_Freeze import hooks

"""the scipy module loads items within itself in a way that causes
problems without the entire package and a number of other subpackages
being present."""
finder.IncludePackage("scipy._lib")  # Changed include from scipy.lib to scipy._lib
finder.IncludePackage("scipy.misc")


2. repo owner

This issue should be resolved with cx_Freeze 5.0. Please re-open if this is not the case. Thanks.

It doesn't appear fixed yet in 5.0.1, I had to make the same patch.

Hello,

I spent a day on this topic recently. Here my summary: I am using cx_freeze 5 and anaconda with python 2.7.

I found that the dlls for the following packages were missing in my package: -PyQt -vtk -numpy -scipy -shapely

Thus we can conclude that this is not an isolated numpy issue. It could be related to anaconda though. Building the same application with pythonxy (not sure which cx_freeze version I had used though; thus I cannot exclude it to be related to a different cx_freeze version)

The first problem was, that the qtcore4.dll and qtgui4.dll were not stored in /bin/PyQt/ but in /bin/, whereas the package was -according to Process-Monitor - searching in /bin/PyQt/. Moving the dlls fixed this issue.

Next was numpy. For that, several mkl_* and a libiompd5.dll were completely missing (i.e. not just in the wrong folder).

Next was vtk where I needed all the vtk_*.dll

In the end we concluded that it is not efficient to create a list of all missing dlls (plus debugging any future missing dlls). Instead, we decided to accept a bit of overhead:

In the setup.py we used the glob package to create a list of all dlls in Lib/bin (or was it /Library/bin), and added these to the 'include' list to be put in /bin/ (i.e. we did NOT sort them into the appropriate directories like pyqt, vtk, ...

The we added to the python code of the program a check:

if getattr(sys, 'frozen', False):
sys.path.insert(0, join(dirname(sys.executable),'bin')


This adds the path into which we copied all dlls to the path and the application should search there for it. This last point did not yet work out though. Thus we had to add the path to a batch-file before running the application

PATH=%WINDIR%\System32;c:\path\to\frozen\application\bin


That's not nice, but it was late and we were happy to have a solution. I hope, this post will help someone with the same problem.

I'd like to stress that this issue is not fixed. cx_freeze 5.0 with anaconda will not pick the right dlls and/or put them into the wrong folders for at least the list of the packages I listed above.

Regards, Hendrik

I had a similar problem and found a simple solution.

In short,
I removed anything that has to do with numpy from C:\Anaconda2\Lib\site-packages

Here is my whole scenario:

• I converted .py to .exe using cx_freeze

• Check the exe by running on my own PC. Everything is fine.

• Run it on collegues' PC, got the error:

Traceback (most recent call last):
File "C:\Anaconda2\lib\site-packages\cx_Freeze\initscripts\__startup__.py", li
ne 12, in <module>
File "C:\Anaconda2\lib\site-packages\cx_Freeze\initscripts\Console.py", line 2
4, in <module>
File "PatchBasedObjectAnnotator.py", line 1, in <module>
File "C:\Anaconda2\lib\site-packages\numpy\__init__.py", line 142, in <module>

File "C:\Anaconda2\lib\site-packages\numpy\add_newdocs.py", line 13, in <modul
e>
File "C:\Anaconda2\lib\site-packages\numpy\lib\__init__.py", line 8, in <modul
e>
File "C:\Anaconda2\lib\site-packages\numpy\lib\type_check.py", line 11, in <mo
dule>
File "C:\Anaconda2\lib\site-packages\numpy\core\__init__.py", line 24, in <mod
ule>
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control).  Otherwise reinstall numpy.

• Did a lot of google search, and been here as well.

• Removed anything that has to do with numpy from C:\Anaconda2\Lib\site-packages

• Confirmed, exe file is running on other PC's without any problem.

Hello, I am really interested in the last comment by @Jumabek Alikhanov since I exactly have the same problem, but cannot find a solution to solve it.

You said that you "Removed anything that has to do with numpy from C:\Anaconda2\Lib\site-packages", but I do need Numpy in my .exe, and if I remove it, it fails already at the freezing part. Can you develop your solution? Thanks,

After a Numpy update (1.13.1), the error was mentionning a DLL error. Dependency Walker saved my day: it was looking for the "libiomp5md.dll" which was missing on other computers. Once I added this dll to the include_files, everything went fine.