ImportError with VecCUSPRestoreCUDAArrayWrite on Linux Server with CUDA PETSc installation

Dear Developers,

First of all, thanks a lot for maintaining the PETSc bindings to Python. While petsc4py works fine with CPUs, I am facing an issue installing and working with petsc4py when PETSc has been installed with CUDA support. It gives an ImportError with VecCUSPRestoreCUDAArrayWrite.

Since the CUDA installation on this server (Linux compute-b18-12.edu 2.6.32-696.18.7.el6.x86_64 SMP Thu Dec 28 20:15:47 EST 2017 x86_64 x86_64 x86_64 GNU/Linux) was not installed with CUSP libraries, I successfully built PETSc using the following command after downloading CUSP github repo to a custom location:

* ./configure PETSC_ARCH=linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8 --with-64-bit-indices --download-fblaslapack --with-mpi-dir=/cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys --with-cuda=1 --with-cusp-dir=/home/moulik/Software/cusplibrary-0.5.0 --with-thrust=1 --with-precision=double --with-clanguage=c *

The libraries I am using on the server during installation are : * 1) hpcc 5) java/current(default) 9) netcdf/gnu/4.6.1/openmpi/1.6.5/4.3.2 2) slurm/16.05.1(default) 6) python/3.5.1 10) gdrive/2.1.0 3) openmpi/1.6.5/gnu/4.6.1 7) cuda/7.5.18(default) 4) texlive/2015 8) hdf5/gnu/4.6.1/openmpi/1.6.5/shared/1.8.13 *

Thereafter, I built petsc4py with the above PETSC_ARCH and PETSC_DIR which seems to go fine:

* [moulik@login-2 ~/Software/petsc4py]> python setup.py build running build running build_src running build_py creating build creating build/lib.linux-x86_64-3.5 creating build/lib.linux-x86_64-3.5/petsc4py copying src/main.py -> build/lib.linux-x86_64-3.5/petsc4py copying src/PETSc.py -> build/lib.linux-x86_64-3.5/petsc4py copying src/init.py -> build/lib.linux-x86_64-3.5/petsc4py creating build/lib.linux-x86_64-3.5/petsc4py/lib copying src/lib/init.py -> build/lib.linux-x86_64-3.5/petsc4py/lib creating build/lib.linux-x86_64-3.5/petsc4py/include creating build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/petsc4py.h -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/petsc4py.PETSc.h -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/numpy.h -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/petsc4py.PETSc_api.h -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/petsc4py.i -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/PETSc.pxd -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/init.pxd -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/include/petsc4py/init.pyx -> build/lib.linux-x86_64-3.5/petsc4py/include/petsc4py copying src/PETSc.pxd -> build/lib.linux-x86_64-3.5/petsc4py copying src/lib/petsc.cfg -> build/lib.linux-x86_64-3.5/petsc4py/lib running build_ext PETSC_DIR: /home/moulik/Software/petsc-3.9.1 PETSC_ARCH: linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8 version: 3.9.1 release integer-size: 64-bit scalar-type: real precision: double language: CONLY compiler: /cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys/bin/mpicc linker: /cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys/bin/mpicc building 'PETSc' extension creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8 creating build/temp.linux-x86_64-3.5/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/src /cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys/bin/mpicc -pthread -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -g3 -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/gdbm/include -I/usr/local/tcl/8.6.1/include -DPETSC_DIR=/home/moulik/Software/petsc-3.9.1 -I/cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys/include -I/cell_root/software/cuda/7.5.18/sys/include -I/home/moulik/Software/cusplibrary-0.5.0/ -I/home/moulik/Software/petsc-3.9.1/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/include -I/home/moulik/Software/petsc-3.9.1/include -Isrc/include -I/usr/local/python3/3.5.1/lib/python3.5/site-packages/numpy/core/include -I/usr/local/python3/3.5.1/include/python3.5m -c src/PETSc.c -o build/temp.linux-x86_64-3.5/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/src/PETSc.o In file included from src/PETSc.c:4:0: src/petsc4py.PETSc.c: In function ‘PyInit_PETSc’: src/petsc4py.PETSc.c:270181:22: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without /cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys/bin/mpicc -pthread -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -g3 -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/gdbm/include -I/usr/local/tcl/8.6.1/include -DPETSC_DIR=/home/moulik/Software/petsc-3.9.1 -I/cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys/include -I/cell_root/software/cuda/7.5.18/sys/include -I/home/moulik/Software/cusplibrary-0.5.0/ -I/home/moulik/Software/petsc-3.9.1/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/include -I/home/moulik/Software/petsc-3.9.1/include -Isrc/include -I/usr/local/python3/3.5.1/lib/python3.5/site-packages/numpy/core/include -I/usr/local/python3/3.5.1/include/python3.5m -c src/libpetsc4py.c -o build/temp.linux-x86_64-3.5/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/src/libpetsc4py.o creating build/lib.linux-x86_64-3.5/petsc4py/lib/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8 /cell_root/software/openmpi/1.6.5/gnu/4.6.1/threaded/sys/bin/mpicc -pthread -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -g3 -shared -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/gdbm/include -I/usr/local/tcl/8.6.1/include -L/usr/local/gdbm/lib -Wl,-rpath,/usr/local/gdbm/lib -L/usr/local/tcl/8.6.1/lib -Wl,-rpath,/usr/local/tcl/8.6.1/lib -L/usr/local/python3/3.5.1/lib -Wl,-rpath,/usr/local/python3/3.5.1/lib build/temp.linux-x86_64-3.5/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/src/PETSc.o build/temp.linux-x86_64-3.5/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/src/libpetsc4py.o -L/home/moulik/Software/petsc-3.9.1/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/lib -L/usr/local/python3/3.5.1/lib -Wl,--enable-new-dtags,-R/home/moulik/Software/petsc-3.9.1/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/lib -Wl,--enable-new-dtags,-R/usr/local/python3/3.5.1/lib -lpetsc -lpython3.5m -o build/lib.linux-x86_64-3.5/petsc4py/lib/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/PETSc.cpython-35m-x86_64-linux-gnu.so writing build/lib.linux-x86_64-3.5/petsc4py/lib/petsc.cfg*

However, when I try to run petsc4py, it gives the following ImportError:

* [moulik@compute-b17-10 ~/Software/petsc-3.9.1]> ipython Python 3.5.1 (default, Mar 14 2018, 16:26:29) Type 'copyright', 'credits' or 'license' for more information IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from petsc4py import PETSc /home/moulik/.python-venvs/globalmcmc/lib/python3.5/site-packages/petsc4py/lib/init.py:124: UserWarning: ignored arch: 'linux-gnu4.6.1-openmpi1.6.5', using: 'linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8' (os.path.pathsep.join(arch_env), arch))

ImportError Traceback (most recent call last) <ipython-input-1-1e4f0d94f070> in <module>() ----> 1 from petsc4py import PETSc

/home/moulik/.python-venvs/globalmcmc/lib/python3.5/site-packages/petsc4py/PETSc.py in <module>() 1 ARCH = None 2 from petsc4py.lib import ImportPETSc ----> 3 PETSc = ImportPETSc(ARCH) 4 PETSc._initialize() 5 del PETSc

/home/moulik/.python-venvs/globalmcmc/lib/python3.5/site-packages/petsc4py/lib/init.py in ImportPETSc(arch) 27 """ 28 path, arch = getPathArchPETSc(arch) ---> 29 return Import('petsc4py', 'PETSc', path, arch) 30 31 def getPathArchPETSc(arch=None):

/home/moulik/.python-venvs/globalmcmc/lib/python3.5/site-packages/petsc4py/lib/init.py in Import(pkg, name, path, arch) 65 pathlist = [os.path.join(path, arch)] 66 f, fn, info = imp.find_module(name, pathlist) ---> 67 with f: module = imp.load_module(fullname, f, fn, info) 68 module.arch = arch # save arch value 69 setattr(sys.modules[pkg], name, module)

/home/moulik/.python-venvs/globalmcmc/lib/python3.5/imp.py in load_module(name, file, filename, details) 240 return load_dynamic(name, filename, opened_file) 241 else: --> 242 return load_dynamic(name, filename, file) 243 elif type_ == PKG_DIRECTORY: 244 return load_package(name, filename)

/home/moulik/.python-venvs/globalmcmc/lib/python3.5/imp.py in load_dynamic(name, path, file) 340 spec = importlib.machinery.ModuleSpec( 341 name=name, loader=loader, origin=path) --> 342 return _load(spec) 343 344 else:

ImportError: /home/moulik/.python-venvs/globalmcmc/lib/python3.5/site-packages/petsc4py/lib/linux-gnu4.6.1-openmpi1.6.5-cuda7.5.8/PETSc.cpython-35m-x86_64-linux-gnu.so: undefined symbol: VecCUSPRestoreCUDAArrayWrite *

Is there something trivial that I am missing here? I have tried both PETSc versions 3.9.1 and 3.9.2 with the current petsc4py github repo version (3.9.1) to the same effect.

Thanks in advance.

Comments (8)