AttributeError: module 'petsc' has no attribute 'get_petsc_dir'

[Simon]

Hello,

I just tried to install petsc4py with a python3.8, pip19.3.1 on a ubuntu 18.04. I created a clean conda environment with mpi4py3.0.3 and numpy1.17.4 but if I try:

‌

pip install petsc petsc4py

I got the following error:

‌

Collecting petsc
  Downloading https://files.pythonhosted.org/packages/13/b6/e7efd498872b6a8de72879cb7731e4a48f8129e14f14eb5ee30ce3653fb0/petsc-3.12.0.tar.gz (14.2MB)
     |████████████████████████████████| 14.2MB 387kB/s 
Collecting petsc4py
  Downloading https://files.pythonhosted.org/packages/e4/51/7eaa24a5797f0728007acfbfafed026ff2b0602f9ddd07564829423663df/petsc4py-3.12.0.tar.gz (1.8MB)
     |████████████████████████████████| 1.8MB 141kB/s 
    ERROR: Command errored out with exit status 1:
     command: /home/slegrand/miniconda3/envs/pysit/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5zj8tpzo/petsc4py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5zj8tpzo/petsc4py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-5zj8tpzo/petsc4py/pip-egg-info
         cwd: /tmp/pip-install-5zj8tpzo/petsc4py/
    Complete output (52 lines):
    running egg_info
    creating /tmp/pip-install-5zj8tpzo/petsc4py/pip-egg-info/petsc4py.egg-info
    writing /tmp/pip-install-5zj8tpzo/petsc4py/pip-egg-info/petsc4py.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-install-5zj8tpzo/petsc4py/pip-egg-info/petsc4py.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-install-5zj8tpzo/petsc4py/pip-egg-info/petsc4py.egg-info/requires.txt
    writing top-level names to /tmp/pip-install-5zj8tpzo/petsc4py/pip-egg-info/petsc4py.egg-info/top_level.txt
    writing manifest file '/tmp/pip-install-5zj8tpzo/petsc4py/pip-egg-info/petsc4py.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-5zj8tpzo/petsc4py/setup.py", line 270, in <module>
        main()
      File "/tmp/pip-install-5zj8tpzo/petsc4py/setup.py", line 267, in main
        run_setup()
      File "/tmp/pip-install-5zj8tpzo/petsc4py/setup.py", line 119, in run_setup
        setup(packages     = ['petsc4py',
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 296, in run
        self.find_sources()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
        mm.run()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 534, in run
        self.add_defaults()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
        sdist.add_defaults(self)
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/command/sdist.py", line 226, in add_defaults
        self._add_defaults_python()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages/setuptools/command/sdist.py", line 127, in _add_defaults_python
        build_py = self.get_finalized_command('build_py')
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages/setuptools/command/build_py.py", line 34, in finalize_options
        orig.build_py.finalize_options(self)
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/command/build_py.py", line 43, in finalize_options
        self.set_undefined_options('build',
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/cmd.py", line 287, in set_undefined_options
        src_cmd_obj.ensure_finalized()
      File "/home/slegrand/miniconda3/envs/pysit/lib/python3.8/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-install-5zj8tpzo/petsc4py/conf/baseconf.py", line 398, in finalize_options
        self.petsc_dir  = config.get_petsc_dir(self.petsc_dir)
      File "/tmp/pip-install-5zj8tpzo/petsc4py/conf/baseconf.py", line 336, in get_petsc_dir
        petsc_dir = petsc.get_petsc_dir()
    AttributeError: module 'petsc' has no attribute 'get_petsc_dir'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

‌

Then I tried to install petsc first with

‌

pip install petsc

which led me to this error:

‌

Collecting petsc
  Using cached https://files.pythonhosted.org/packages/13/b6/e7efd498872b6a8de72879cb7731e4a48f8129e14f14eb5ee30ce3653fb0/petsc-3.12.0.tar.gz
Building wheels for collected packages: petsc
  Building wheel for petsc (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/slegrand/miniconda3/envs/pysit/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-u5vmd78g/petsc/setup.py'"'"'; __file__='"'"'/tmp/pip-install-u5vmd78g/petsc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-kixrwrnt --python-tag cp38
       cwd: /tmp/pip-install-u5vmd78g/petsc/
  Complete output (1 lines):
  petsc: this package cannot be built as a wheel
  ----------------------------------------
  ERROR: Failed building wheel for petsc
  Running setup.py clean for petsc
Failed to build petsc
Installing collected packages: petsc
    Running setup.py install for petsc ... \

done
Successfully installed petsc-3.12.0

At this point I can install petsc4py by running

‌

pip install petsc4py

but it doesn’t really reliable….

Hope this will help.

Cheers,

Simon

[Lisandro Dalcin]

Sorry about that, but the way pip handles dependencies is always an issue. You are definitely right, this is not reliable, but I hope you understand I’m not the one to blame.

Does it work if you start from scratch and do pip install petsc petsc4py? The issue is simply that the petsc package has to get installed FIRST in the final Python’s site-packages destination BEFORE building the petsc4py package, and pip does not always do that. IIRC, doing pip install petsc petsc4py somehow works because now petsc gets installed first.

PETSc is not really a Python package, all that support to install it with pip is a hack, very hard to maintain, and it may break at any time as the pip codebase evolves over the years. Any suggestions for improvements most welcome.

If you really care about reliability, please build&install PETSc yourself the traditional way, ie.m ./configure and make and make install). Next, export PETSC_DIR=/path/to/petsc and finally pip install petsc4py.

PS: Ignore the pip install petsc error. This is part of the hack. Pip tries first to build petsc as a wheel, and this cannot be done easily, so I make it fail on purpose, then pip retries with a standard setup.py install, and finally it succeeds.

[Simon]

Sorry about that, but the way pip handles dependencies is always an issue. You are definitely right, this is not reliable, but I hope you understand I’m not the one to blame

Of course! And thanks for the effort of packaging!

Does it work if you start from scratch and do pip install petsc petsc4py? The issue is simply that the petsc package has to get installed FIRST in the final Python’s site-packages destination BEFORE building the petsc4py package, and pip does not always do that. IIRC, doing pip install petsc petsc4py somehow works because now petsc gets installed first.

The installation now succeed despite these errors in the middle:

‌

$ pip install petsc petsc4py
Collecting petsc
  Downloading https://files.pythonhosted.org/packages/13/b6/e7efd498872b6a8de72879cb7731e4a48f8129e14f14eb5ee30ce3653fb0/petsc-3.12.0.tar.gz (14.2MB)
     |████████████████████████████████| 14.2MB 143kB/s 
Collecting petsc4py
  Downloading https://files.pythonhosted.org/packages/e4/51/7eaa24a5797f0728007acfbfafed026ff2b0602f9ddd07564829423663df/petsc4py-3.12.0.tar.gz (1.8MB)
     |████████████████████████████████| 1.8MB 357kB/s 
Requirement already satisfied: numpy in /home/slegrand/miniconda3/envs/pysit/lib/python3.8/site-packages (from petsc4py) (1.17.4)
Building wheels for collected packages: petsc, petsc4py
  Building wheel for petsc (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/slegrand/miniconda3/envs/pysit/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-86jt5ng2/petsc/setup.py'"'"'; __file__='"'"'/tmp/pip-install-86jt5ng2/petsc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-negv9d_u --python-tag cp38
       cwd: /tmp/pip-install-86jt5ng2/petsc/
  Complete output (1 lines):
  petsc: this package cannot be built as a wheel
  ----------------------------------------
  ERROR: Failed building wheel for petsc
  Running setup.py clean for petsc
  Building wheel for petsc4py (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/slegrand/miniconda3/envs/pysit/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-86jt5ng2/petsc4py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-86jt5ng2/petsc4py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-tgyxhg_6 --python-tag cp38
       cwd: /tmp/pip-install-86jt5ng2/petsc4py/
  Complete output (27 lines):
  running bdist_wheel
  PETSC_DIR not specified
  running build
  running build_src
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/petsc4py
  copying src/PETSc.py -> build/lib.linux-x86_64-3.8/petsc4py
  copying src/__init__.py -> build/lib.linux-x86_64-3.8/petsc4py
  copying src/__main__.py -> build/lib.linux-x86_64-3.8/petsc4py
  creating build/lib.linux-x86_64-3.8/petsc4py/lib
  copying src/lib/__init__.py -> build/lib.linux-x86_64-3.8/petsc4py/lib
  creating build/lib.linux-x86_64-3.8/petsc4py/include
  creating build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/numpy.h -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/petsc4py.PETSc_api.h -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/petsc4py.h -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/petsc4py.PETSc.h -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/petsc4py.i -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/__init__.pxd -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/PETSc.pxd -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/include/petsc4py/__init__.pyx -> build/lib.linux-x86_64-3.8/petsc4py/include/petsc4py
  copying src/PETSc.pxd -> build/lib.linux-x86_64-3.8/petsc4py
  copying src/lib/petsc.cfg -> build/lib.linux-x86_64-3.8/petsc4py/lib
  running build_ext
  error: PETSc not found
  ----------------------------------------
  ERROR: Failed building wheel for petsc4py
  Running setup.py clean for petsc4py
Failed to build petsc petsc4py
Installing collected packages: petsc, petsc4py
    Running setup.py install for petsc ... done
    Running setup.py install for petsc4py ... done
Successfully installed petsc-3.12.0 petsc4py-3.12.0

But as you said, they can be ignored.

‌

If you really care about reliability, please build&install PETSc yourself the traditional way, ie.m ./configure and make and make install). Next, export PETSC_DIR=/path/to/petsc and finally pip install petsc4py

Indeed that is the way I used to install petsc, but I’m trying to automatize the installation procedure of a code that depends on petsc4py.

Thanks for your quick reply!

Simon

[Lisandro Dalcin]

OK, I see. If you are just going for automation, then I really recommend to do it this way, this should never break, even if pip changes in the future:

pip install mpi4py # not strictly required, but good to have around
export PETSC_CONFIGURE_OPTIONS="" # not required, but this way you can extra PETSc configure options, e.g `--download-parmetis`, or whatever you need.
pip install petsc
pip install petsc4py

‌

[Simon]

Thanks a lot, it’s gonna be useful!