Can’t import numpy on pypy 2.5 on Mac OS X 10.8.5

Issue #21 resolved
damiancugley created an issue

I cannot install numpy in to a pypy virtual env. It seems to be failing to find lib_mtrand.so

Details in this gist: https://gist.github.com/pdc/8cb3ce1af55749dfd92d

Comments (35)

  1. mattip

    Do you have a file with a name somewhat like lib_mtrand.so in that directory? What is the name of it (sorry, I commented in the gist instead of here...)?

  2. Former user Account Deleted

    I've tried with 6.0.8 and it still fails. as pointed by @martinp23 avoiding pip works:

    git clone https://bitbucket.org/pypy/numpy.git
    cd numpy
    pypy setup.py install
    
  3. Alex Rothberg

    This is on OS X 10.10:

    $ pypy -c 'import numpy'
    Traceback (most recent call last):
      File "app_main.py", line 75, in run_toplevel
      File "app_main.py", line 581, in run_it
      File "<string>", line 1, in <module>
      File "/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/__init__.py", line 170, in <module>
        from . import add_newdocs
      File "/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/add_newdocs.py", line 13, in <module>
        from numpy.lib import add_newdoc
      File "/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/lib/__init__.py", line 18, in <module>
        from .polynomial import *
      File "/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/lib/polynomial.py", line 19, in <module>
        from numpy.linalg import eigvals, lstsq, inv
      File "/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/linalg/__init__.py", line 51, in <module>
        from .linalg import *
      File "/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/linalg/linalg.py", line 29, in <module>
        from numpy.linalg import _umath_linalg
      File "/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/linalg/_umath_linalg.py", line 59, in <module>
        umath_linalg_capi = umath_ffi.dlopen(os.path.dirname(__file__) + so_name)
      File "/usr/local/Cellar/pypy/2.5.0/libexec/lib_pypy/cffi/api.py", line 120, in dlopen
        lib, function_cache = _make_ffi_library(self, name, flags)
      File "/usr/local/Cellar/pypy/2.5.0/libexec/lib_pypy/cffi/api.py", line 492, in _make_ffi_library
        backendlib = _load_backend_lib(backend, libname, flags)
      File "/usr/local/Cellar/pypy/2.5.0/libexec/lib_pypy/cffi/api.py", line 481, in _load_backend_lib
        return backend.load_library(name, flags)
    OSError: Cannot load library /usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/linalg/libumath_linalg_cffi.so: dlopen(/usr/local/Cellar/pypy/2.5.0/libexec/site-packages/numpy/linalg/libumath_linalg_cffi.so, 2): image not found
    
    $ pypy -m pip freeze
    cffi==0.8.6
    greenlet==0.4.5
    mock==1.0.1
    numpy==1.9.0
    python-dateutil==2.4.2
    readline==6.2.4.1
    six==1.9.0
    
    $ pypy -m pip -V
    pip 6.1.1 from /usr/local/Cellar/pypy/2.5.0/libexec/site-packages (python 2.7)
    
  4. mattip

    It seems the library is not getting built or copied during installation. Could someone who has this problem start over again but this time do::

     DISTUTILS_DEBUG=1 pip --global-option="-vv" install
    

    and post the log here or find out what happened to libumath_linalg_cffi.so ?

  5. Jason Madden

    There may be two (related?) issues in play here.

    First, at least in develop (python setup.py develop) mode, it seems the library is getting created, but it's being nested inside a directory with the same name:

    $ ls -FR sources/numpy/numpy/linalg/
    sources/numpy/numpy/linalg/:
    __init__.py  _umath_linalg.py  bento.info  bscript  info.py  lapack_lite/  lapack_lite.py  lapack_litemodule.c  liblapack_lite.so/  libumath_linalg_cffi.so/  linalg.py  setup.py  setup.pyc  tests/  umath_linalg.c  umath_linalg.c.src
    
    ...
    
    sources/numpy/numpy/linalg/liblapack_lite.so:
    liblapack_lite.so*
    
    sources/numpy/numpy/linalg/libumath_linalg_cffi.so:
    libumath_linalg_cffi.so*
    

    It's built flat:

    $ ls sources/numpy/build/temp.macosx-10.10-x86_64-2.7/
    lib_mtrand.so  liblapack_lite.so  libnpymath.a  libumath_linalg_cffi.so  numpy
    

    Relevant portion of the log:

    building 'umath_linalg_cffi' library
    compiling C sources
    C compiler: cc -arch x86_64 -O2 -fPIC -Wimplicit -O2 -fPIC -Wimplicit
    
    compile options: '-D_UMATH_LINALG_CAPI_DLL -DNO_ATLAS_INFO=3 -Inumpy/core/include -Inumpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/Users/jmadden/bin/pypy/include -Inumpy/core/src/private -I/Users/jmadden/bin/pypy/include -c'
    extra options: '-msse3 -Wl,-framework -Wl,Accelerate'
    cc: numpy/linalg/umath_linalg.c
    clang: warning: -Wl,-framework: 'linker' input unused
    clang: warning: -Wl,Accelerate: 'linker' input unused
    In file included from numpy/linalg/umath_linalg.c.src:14:
    In file included from numpy/core/src/private/npy_pycompat.h:4:
    numpy/core/include/numpy/npy_3kcompat.h:125:14: warning: implicit declaration of function 'PyUnicode_Concat' is invalid in C99 [-Wimplicit-function-declaration]
        newobj = PyUnicode_Concat(*left, right);
                 ^
    numpy/core/include/numpy/npy_3kcompat.h:125:12: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
        newobj = PyUnicode_Concat(*left, right);
               ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    numpy/core/include/numpy/npy_3kcompat.h:135:12: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
        newobj = PyUnicode_Concat(*left, right);
               ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    numpy/linalg/umath_linalg.c.src:813:32: warning: unknown warning group '-Wmaybe-uninitialized', ignored [-Wunknown-pragmas]
    #pragma GCC diagnostic ignored "-Wmaybe-uninitialized"
                                   ^
    numpy/linalg/umath_linalg.c.src:3151:49: warning: incompatible pointer types passing 'PyUFuncGenericFunction *' (aka 'void (**)(char **, npy_intp *, npy_intp *, void *)') to parameter of type 'void (***)(char **, long *, long *, void *)'; take the address with & [-Wincompatible-pointer-types]
            f = PyUFunc_FromFuncAndDataAndSignature(d->funcs,
                                                    ^~~~~~~~
                                                    &
    /Users/jmadden/bin/pypy/include/pypy_decl.h:420:69: note: passing argument to parameter 'arg0' here
    PyAPI_FUNC(PyObject *) PyUFunc_FromFuncAndDataAndSignature(void (***arg0)(char **, Signed *, Signed *, void *), void *arg1, char *arg2, Py_ssize_t arg3, Py_ssize_t arg4, Py_ssize_t arg5, Py_ssize_t arg6, char *arg7, char *arg8, Py_ssize_t arg9, char *arg10);
                                                                        ^
    5 warnings generated.
    cc -shared -undefined dynamic_lookup -arch x86_64 -L/opt/local/lib -L/opt/local/lib build/temp.macosx-10.10-x86_64-2.7/numpy/linalg/umath_linalg.o -Lbuild/temp.macosx-10.10-x86_64-2.7 -lnpymath -llapack_lite -o build/temp.macosx-10.10-x86_64-2.7/libumath_linalg_cffi.so -msse3 -Wl,-framework -Wl,Accelerate
    building '_mtrand' library
    compiling C sources
    C compiler: cc -arch x86_64 -O2 -fPIC -Wimplicit -O2 -fPIC -Wimplicit
    
    creating build/temp.macosx-10.10-x86_64-2.7/numpy/random
    creating build/temp.macosx-10.10-x86_64-2.7/numpy/random/mtrand
    compile options: '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Inumpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/Users/jmadden/bin/pypy/include -Inumpy/core/src/private -I/Users/jmadden/bin/pypy/include -c'
    cc: numpy/random/mtrand/randomkit.c
    cc: numpy/random/mtrand/distributions.c
    cc: numpy/random/mtrand/initarray.c
    cc -shared -undefined dynamic_lookup -arch x86_64 -L/opt/local/lib -L/opt/local/lib build/temp.macosx-10.10-x86_64-2.7/numpy/random/mtrand/randomkit.o build/temp.macosx-10.10-x86_64-2.7/numpy/random/mtrand/distributions.o build/temp.macosx-10.10-x86_64-2.7/numpy/random/mtrand/initarray.o -Lbuild/temp.macosx-10.10-x86_64-2.7 -o build/temp.macosx-10.10-x86_64-2.7/lib_mtrand.so
    creating numpy/linalg/liblapack_lite.so
    creating numpy/linalg/libumath_linalg_cffi.so
    creating numpy/random/lib_mtrand.so
    

    Like others noted, I can't reproduce the problem using python ./setup.py install, or setupegg.py with setuptools 15.2. Both of those commands copy the library as expected and produce a working installation.

    Pip, however, fails to copy the libraries entirely. But I can't get much debugging info out of it:

    $  pip --version
    pip 6.1.1 from /...bin/pypy/site-packages (python 2.7)
    $ DISTUTILS_DEBUG=1 pip -v install --global-option="-vvv" .
    Processing /...sources/numpy
      Running setup.py (path:/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-aT9O06-build/setup.py) egg_info for package from file:///.../numpy
        Running command python setup.py egg_info
      Source in /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-aT9O06-build has version 1.9.0, which satisfies requirement numpy==1.9.0 from file:///.../numpy
    Installing collected packages: numpy
      Running setup.py install for numpy
        Running command /Users/jmadden/bin/pypy/bin/pypy -c "import setuptools, tokenize;__file__='/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-aT9O06-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" -vvv install --record /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-quBkje-record/install-record.txt --single-version-externally-managed --compile
    Successfully installed numpy-1.9.0
    Cleaning up...
    $ ls ~/bin/pypy/site-packages/numpy/linalg/
    __init__.py  __init__.pyc  _umath_linalg.py  _umath_linalg.pyc  info.py  info.pyc  lapack_lite.py  lapack_lite.pyc  linalg.py  linalg.pyc  setup.py  setup.pyc  tests
    
  6. mattip

    Jason, apparently there may be a bug in pip preventing debug output, rumor has it that this is fixed on a developer branch. Can you try installing, in your virtualenv:

    pip install git+https://github.com/pypa/pip.git#egg=pip
    

    you may need to add @develop after the pip.git

    also, you might want to do --no-cache-dir and --no-use-wheel, the develop branch has a on by default wheel cache now. If we could get a log maybe I could see where I need to special-case some os.path.join() cruft, since the numpy installation is quite verbose

  7. Jason Madden

    @mattip I'm sorry, I installed the dev version of pip in a fresh virtualenv and got no more output (using the latest numpy sources too), and still the same failing results :(

    $ pip --version
    pip 7.0.0.dev0 from //VirtualEnvs/pypy-numpy-pip/site-packages (python 2.7)
    $ DISTUTILS_DEBUG=1 pip -vvv --no-cache-dir install  --global-option="-vvv"  --no-use-wheel .
    //VirtualEnvs/pypy-numpy-pip/site-packages/pip/commands/install.py:197: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
      cmdoptions.check_install_build_global(options)
    Processing //pypy_numpy
      Running setup.py (path:/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-N_GT2U-build/setup.py) egg_info for package from file:///pypy_numpy
        Running command python setup.py egg_info
      Source in /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-N_GT2U-build has version 1.9.0, which satisfies requirement numpy==1.9.0 from file:////pypy_numpy
    Installing collected packages: numpy
      Running setup.py install for numpy
        Running command /VirtualEnvs/pypy-numpy-pip/bin/pypy -c "import setuptools, tokenize;__file__='/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-N_GT2U-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" -vvv install --record /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Yiq_j9-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/jmadden/Projects/VirtualEnvs/pypy-numpy-pip/include/site/python2.7/numpy
      Removing source in /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-N_GT2U-build
    Successfully installed numpy-1.9.0
    Cleaning up...
    $ ls  //VirtualEnvs/pypy-numpy-pip/site-packages/numpy/linalg
    __init__.py  __init__.pyc  _umath_linalg.py  _umath_linalg.pyc  info.py  info.pyc  lapack_lite.py  lapack_lite.pyc  linalg.py  linalg.pyc  setup.py  setup.pyc  tests
    
  8. Jason Madden

    @mattip It seems like Pip goes out of its way to completely hide any output when it runs setup.py. I made some code changes to pip.req.req_install and got the output captured. It's attached.

  9. Rui Lopes

    I am having the same issue on OSX Yosemite. I am using the pip from the dev branch as suggested before but it has the same problem. The only work around so far seems to be to install without pip.

  10. Jason Madden

    I'm sorry @rlamy. I just tried on OS X (fresh virtualenv, updated pip, pip install from git, recent pypy nightly) but it still fails with the same error reported in this comment above.

  11. Jason Madden

    Hey, at least pip 7.1.2 produces better debugging output! The .so files are being built, they're just not being copied to the right place---they're left in the temp directory. Compare the first and last parts of this output; the .h files are making it into the virtualenv, the .so files are left in the temp directory:

        copying build/src.macosx-10.10-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h -> /VENV/site-packages/numpy/core/include/numpy
        copying build/src.macosx-10.10-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h -> /VENV/site-packages/numpy/core/include/numpy
        copying build/src.macosx-10.10-x86_64-2.7/numpy/core/include/numpy/multiarray_api.txt -> /VENV/site-packages/numpy/core/include/numpy
        copying build/src.macosx-10.10-x86_64-2.7/numpy/core/include/numpy/__ufunc_api.h -> /VENV/site-packages/numpy/core/include/numpy
        copying build/src.macosx-10.10-x86_64-2.7/numpy/core/include/numpy/ufunc_api.txt -> /VENV/site-packages/numpy/core/include/numpy
        not copying /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Xb3JlC-build/numpy/core/lib/npy-pkg-config/npymath.ini (output up-to-date)
        not copying /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Xb3JlC-build/numpy/core/lib/npy-pkg-config/mlib.ini (output up-to-date)
        running install_clib
        copying build/temp.macosx-10.10-x86_64-2.7/libnpymath.a -> /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Xb3JlC-build/numpy/core/lib
        copying build/temp.macosx-10.10-x86_64-2.7/liblapack_lite.so -> /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Xb3JlC-build/numpy/linalg/.
        copying build/temp.macosx-10.10-x86_64-2.7/libumath_linalg_cffi.so -> /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Xb3JlC-build/numpy/linalg/.
        copying build/temp.macosx-10.10-x86_64-2.7/lib_mtrand.so -> /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Xb3JlC-build/numpy/random/.
        running install_egg_info
        running egg_info
        creating numpy.egg-info
        writing numpy.egg-info/PKG-INFO
    
  12. mattip

    Progress, thanks! Could you somehow find out how that directory /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Xb3JlC-build is getting set for self.install_dir in numpy/distutils/command/install_clib.py

    It seems install_dir is the pip build_dir not the numpy one? FWIW, I added the self.distribution.shared_libraries copying code, but the rest of it is upstream's code, so how does upstream cpython successfully do the install_clib step?

  13. Jason Madden

    Here's what I know so far:

    • The instance of install_clib itself has a relative path for install_dir, inside the wheel that pip is trying to build, e.g., build/bdist.macosx-10.10-x86_64/wheel/. This is set in finalize_options and is copied from the install_lib attribute of an install object (IIRC). That looks roughly correct to me.
    • However, the libraries in self.distribution all have absolute paths for their target_dir attribute (for distribution.installed_libraries) or l[2] (for distribution.shared_libraries). When an absolute path is passed to os.path.join, of course, it trumps the relative path. Here's one shared_libraries entry:
    ('umath_linalg_cffi',
        {'depends': ['/private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-sxZ38r-build/numpy/linalg/umath_linalg.c.src'],
         'library_dirs': [],
         ...
         'sources': ['build/src.macosx-10.10-x86_64-2.7/numpy/linalg/umath_linalg.c']},
        '/private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-sxZ38r-build/numpy/linalg/.')
    

    So line 42, copy_file(source, target_dir) winds up with target_dir set to exactly that last element, /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-sxZ38r-build/numpy/linalg/., and that's where the file is copied. * Indeed, if I do change that absolute path to strip off the temporary directory (so it gets placed underneath install_clib.install_dir), the library ends up installed.

    I haven't tracked down how the distribution object gets its libraries populated yet.

  14. mattip

    Thanks. The easiest way to track this down might be to diff from upstream, i.e. look at the changesets on the build_shared_object branch starting at changeset b2637d9

  15. Jason Madden

    So that absolute path is coming in from numpy.distutils.mist_util:Configuration. It sets the target_dir to self.package_path. That value in turn comes from, of all places, the absolute path to setup.py (which is derived by looking at the frame of the calling function) (self.local_path). It's possible to pass the value in as an argument from setup.py, I'm going to try that.

  16. Jason Madden

    Ahh, nice. So I'll see if I can figure out a correct way to get a relative path everywhere.

    (Passing in package_path directly did what I thought it would do, copied all the libraries, but to the root of the egg.)

  17. Jason Madden

    configuration itself is called with no arguments, so the default value of None gets used.

       /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-bRMMkw-build/setup.py(251)<module>()
      -> setup_package()
        /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-bRMMkw-build/setup.py(243)setup_package()
      -> setup(**metadata)
        /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-bRMMkw-build/numpy/distutils/core.py(135)setup()
      -> config = configuration()
        /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-bRMMkw-build/setup.py(142)configuration()
      -> config = Configuration(None, parent_package, top_path, package_path="numpy")
      > /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-bRMMkw-build/numpy/distutils/misc_util.py(669)__init__()
      -> def __init__(self,
      (Pdb) p top_path
      None
    
  18. Jason Madden

    We've found the problem and we do have a workaround. Still working on a solution.

    It turns out that pip install . or pip install git+... copies files into the temporary directory. On OS X, this is set to a per-user directory under /var/folders/.... But on OS X, /var is a symlink to /private/var. When Python code executes from there, it gets a __file__ variable that points to /private/var.

    Now, numpy.distutils relies on looking at the relative paths from __file__ variables to figure out where the root of the installation is, by looking at the difference between setup.py's __file__ and say, numpy.linalg.setup. So far so good.

    But Pip doesn't actually execute setup.py. Instead it reads it in and evals it. When it does this, it explicitly sets __file__. Unfortunately, instead of setting it to the fully resolved path (/private/var/...) it sets it to what it thinks it is, containing the symlink /var:

    Running command ///bin/pypy -c "import setuptools;__file__='/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-Qbvnph-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/tmpxiI_kcpip-wheel-..."
    

    This confuses the relative path machinery, because /private/var/... is not an extension of /var/.... So files are dumped into their absolute build location instead of being copied into the built wheel.

    The workaround is to do something like $ TMPDIR=/private$TMPDIR pip install ..

    FWIW, numpy under CPython has exactly this same issue. Only it doesn't matter. It only wants to copy over a static .a archive which isn't used at runtime; all of the actual .so files it needs are built as distututils Extension objects which uses a different mechanism:

        running install_clib
        copying build/temp.macosx-10.10-x86_64-2.7/libnpymath.a -> /private/var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-YRMc1H-build/numpy/core/lib
        running install_egg_info
    
  19. Jason Madden

    Adding this snippet near the top of the main setup.py also "fixes" the problem, albeit in a very hacky way:

    if sys.platform == 'darwin' and __file__.startswith('/var'):
        __file__ = '/private' + __file__
    
  20. Log in to comment