installing error

Lisandro Dalcin

I guess you installed PETSc 3.5. A version of petsc4py supporting that new PETSc release is about to appear, but not ready yet. However, you can install Cython first, then ask pip to install from the main repository.

$ pip install https://bitbucket.org/petsc/petsc4py/get/master.tar.gz

2014-07-09T09:29:43+00:00

Francis Poulin

Thanks for the comment. I suppose now that there is a transition it is an awkward time to get started but oh well.

I installed cython on my ubuntu machine using apt-get. That worked fine.

I followed your instructions and tried to install petsc4py but found an error. I am copying part of the pip.log in case you can make any sense of it. I have tried but am not clear what's failing. Sorry.

/usr/bin/pip run on Wed Jul 9 09:54:57 2014 Downloading/unpacking https://bitbucket.org/petsc/petsc4py/get/master.tar.gz Downloading from URL https://bitbucket.org/petsc/petsc4py/get/master.tar.gz Running setup.py egg_info for package from https://bitbucket.org/petsc/petsc4py/get/master.tar.gz running egg_info creating pip-egg-info/petsc4py.egg-info writing requirements to pip-egg-info/petsc4py.egg-info/requires.txt writing pip-egg-info/petsc4py.egg-info/PKG-INFO writing top-level names to pip-egg-info/petsc4py.egg-info/top_level.txt writing dependency_links to pip-egg-info/petsc4py.egg-info/dependency_links.txt writing manifest file 'pip-egg-info/petsc4py.egg-info/SOURCES.txt' warning: manifest_maker: standard file '-c' not found

PETSC_DIR not specified
reading manifest file 'pip-egg-info/petsc4py.egg-info/SOURCES.txt'
writing manifest file 'pip-egg-info/petsc4py.egg-info/SOURCES.txt'

Downloading/unpacking numpy (from petsc4py==3.5) Getting page http://pypi.python.org/simple/numpy URLs to search for versions for numpy (from petsc4py==3.5): * http://pypi.python.org/simple/numpy/ Getting page http://pypi.python.org/simple/numpy/ Analyzing links from page https://pypi.python.org/simple/numpy/ Skipping link https://pypi.python.org/packages/2.4/n/numpy/numpy-1.0.1.dev3460.win32-py2.4.exe#md5=a55b13b1f141de2aa965d5c5554c4ad8 (from https://pypi.python.org/simple/numpy/); unknown archive format: .exe ... https://pypi.python.org/simple/petsc/), version 3.4.4 doesn't match >=3.5,<3.6 Ignoring link http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-3.4.5.tar.gz#egg=petsc-3.4.5 (from https://pypi.python.org/simple/petsc/), version 3.4.5 doesn't match >=3.5,<3.6 Ignoring link https://bitbucket.org/petsc/petsc/get/master.tar.gz#egg=petsc-dev (from https://pypi.python.org/simple/petsc/), version dev doesn't match >=3.5,<3.6 Could not find a version that satisfies the requirement petsc>=3.5,<3.6 (from petsc4py==3.5) (from versions: ) No distributions matching the version for petsc>=3.5,<3.6 (from petsc4py==3.5) Exception information: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 223, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 948, in prepare_files url = finder.find_requirement(req_to_install, upgrade=self.upgrade) File "/usr/lib/python2.7/dist-packages/pip/index.py", line 180, in find_requirement raise DistributionNotFound('No distributions matching the version for %s' % req) DistributionNotFound: No distributions matching the version for petsc>=3.5,<3.6 (from petsc4py==3.5)

2014-07-09T14:03:13+00:00

Lisandro Dalcin

Before pip install, please export PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-name, these environment variables should match the ones you used to build PETSc. This way, pip will know where to look for PETSc libraries and headers.

2014-07-09T14:58:27+00:00

Francis Poulin

Thanks for the quick reply.

I think a problem that I had before is that I was using sudo and root doesn't have the paths defined. I do have them in my .bashrc script.

Now that I try it (without sudo), it seems to know where everything is but it still has problems with the permissions. Do I need to export these paths for sudo to get this to work? It seems odd if that's the case, so maybe I'm missing something?

...

writing build/lib.linux-x86_64-2.7/petsc4py/lib/petsc.cfg

running install_lib

creating /usr/local/lib/python2.7/dist-packages/petsc4py

error: could not create '/usr/local/lib/python2.7/dist-packages/petsc4py': Permission denied

Command /usr/bin/python -c "import setuptools;file='/tmp/pip-l0BnZ8-build/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --single-version-externally-managed --record /tmp/pip-VyP44b-record/install-record.txt failed with error code 1 Exception information: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 228, in run requirement_set.install(install_options, global_options) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1093, in install requirement.install(install_options, global_options) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 566, in install cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False) File "/usr/lib/python2.7/dist-packages/pip/init.py", line 255, in call_subprocess % (command_desc, proc.returncode)) InstallationError: Command /usr/bin/python -c "import setuptools;file='/tmp/pip-l0BnZ8-build/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --single-version-externally-managed --record /tmp/pip-VyP44b-record/install-record.txt failed with error code 1

2014-07-09T15:12:43+00:00

Lisandro Dalcin

Pass -E flag to sudo, i.e, like this: sudo -E python setup.py install, or do not use sudo at all, and install in your home directory: python setup.py install --user.

2014-07-09T15:49:07+00:00

Lisandro Dalcin

Actually, use sudo -E pip install ... or pip install --user ...

2014-07-09T15:50:06+00:00

Francis Poulin

Thank you! That worked great.

The last think I would like to install is slepc4py.

I tried the generic pip install --user slepc4py and that failed with error code 1. It seems that a lot of things are not declared.

I also tried the command you suggested before for the master and replaced petsc with slepc, maybe naive. That failed because of permissions.

Sorry to bug you but I hope that after I get this installed I can spend more time programming than bugging you with the installation.

Thanks again, Francis

... src/slepc4py.SLEPc.c:41610:30: error: ‘QEP_DIVERGED_ITS’ undeclared (first use in this function)

src/slepc4py.SLEPc.c:41622:30: error: ‘QEP_DIVERGED_BREAKDOWN’ undeclared (first use in this function)

src/slepc4py.SLEPc.c:41634:30: error: ‘QEP_CONVERGED_ITERATING’ undeclared (first use in this function)

error: command '/usr/bin/mpicc' failed with exit status 1

Command /usr/bin/python -c "import setuptools;file='/tmp/pip-BpJ8nK-build/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --single-version-externally-managed --record /tmp/pip-Ycczio-record/install-record.txt --user failed with error code 1 Exception information: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 228, in run requirement_set.install(install_options, global_options) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1093, in install requirement.install(install_options, global_options) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 566, in install cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False) File "/usr/lib/python2.7/dist-packages/pip/init.py", line 255, in call_subprocess % (command_desc, proc.returncode)) InstallationError: Command /usr/bin/python -c "import setuptools;file='/tmp/pip-BpJ8nK-build/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --single-version-externally-managed --record /tmp/pip-Ycczio-record/install-record.txt --user failed with error code 1

2014-07-09T16:53:34+00:00

Francis Poulin

Actually, when I tried importing petsc4py in python I got an error, so something seems to have failed.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fpoulin/.local/lib/python2.7/site-packages/petsc4py/PETSc.py", line 3, in <module>
    PETSc = ImportPETSc(ARCH)
  File "/home/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc
    return Import('petsc4py', 'PETSc', path, arch)
  File "/home/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 64, in Import
    module = imp.load_module(fullname, fo, fn, stuff)
ImportError: /home/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/arch-linux2-c-debug/PETSc.so: undefined symbol: TAO_CLASSID

2014-07-09T18:46:07+00:00

Lisandro Dalcin

Have you configured PETSc to use complex scalars? In such a case, sorry, but I've just found a couple of days ago that petsc4py cannot work. It is actually a problem in PETSc that me and other PETSc devs are trying to fix. If you are adventurous, you can use this branch from the PETSc repository https://bitbucket.org/petsc/petsc/branch/dalcinl/tao-complex_scalars

2014-07-10T10:55:05+00:00

Francis Poulin

I am happy to give this a try this out (and maybe it might help you with debugging?). Just to make sure that I understand. I should use

pip install --user https://bitbucket.org/petsc/petsc/branch/dalcinl/tao-complex_scalars

to install the testing version of petsc. Do I need to remove the petsc and slepc that I already have?

2014-07-10T14:51:38+00:00

Lisandro Dalcin

Perhaps it is better if you:

git clone -b dalcinl/tao-complex_scalars  https://bitbucket.org/petsc/petsc
cd petsc
./configure --with-scalar-type=complex ...
make

Then you need to rebuild SLEPc (you should use the code in the master branch of the git repository) with that new copy of PETSc, then pip install petsc4py and slepc4py (using the URLs to the respective git repositories I pointed before).

2014-07-10T15:40:01+00:00

Francis Poulin

Thanks for the step-by-step guide.

1) PETSc was built successfully and I even got the nice plot. When I try doing the test case I get that the example performs very poorly in parallel. If I wanted to raise the number of grid points to prove to myself that I can get much better performance, is there a simple way of doing that?

2) I installed SLEPc successfully.

3) When I first tried installing petsc4py I got some Tracebacks of errors. Not sure why. I think I tried upgrade once or twice and then reinstalling and somehow that worked. Seems mysterious but I tried an example, heat.py, and it ran.

4) When I try pip install I get a similar error to before but this time updating does nothing.

fpoulin@vortex:~/software$ pip install --user slepc4py
Downloading/unpacking slepc4py
  Downloading slepc4py-3.4.tar.gz (567Kb): 567Kb downloaded
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main
    self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 223, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 955, in prepare_files
    self.unpack_url(url, location, self.is_download)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1072, in unpack_url
    return unpack_http_url(link, location, self.download_cache, only_download)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 447, in unpack_http_url
    unpack_file(temp_location, location, content_type, link)
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 468, in unpack_file
    untar_file(filename, location)
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 393, in untar_file
    os.makedirs(location)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/home/fpoulin/software/build/slepc4py'

Storing complete log in /home/fpoulin/.pip/pip.log

I am sorry to bother you yet again but any idea what is wrong? I could try and send the pip.log if you like but it is mostly what I have above.

2014-07-10T16:09:52+00:00

Lisandro Dalcin

If you previously tried to install using sudo, I bet that /home/fpoulin/software/build/slepc4py has wrong user/group/permisions, please check the directory /home/fpoulin/software/build, and eventually remove it with sudo rm -rf /home/fpoulin/software/build.

2014-07-10T16:13:37+00:00

Lisandro Dalcin

About the performance issues, remember that by default PETSc configures in debug mode, that could be up to 3X or 5X times slower. Try configuring PETSc with the option --with-debuging=0.

2014-07-10T16:14:50+00:00

Francis Poulin

I decided to go back and reinstall PETSc without debugging.

1) The efficiency is still not much better. I was bold and decided to test it on 16 cores and it flattened out after about 3 or 4. Does it change the number of grid points? I presume that would make a difference

2) The good news is that PETSc and SLEPc are installed.

3) The bad news is that petsc4py doesn't install. I had removed build and maybe that had some ill effects. I did go back and reinstall numpy and cython that were in build and I had to use sudo to do it. Below I copied the tail in. It is strange that /usr/bin/mpicc fails.

src/petsc4py.PETSc.c:179751:30: error: ‘DMDA_BOUNDARY_GHOSTED’ undeclared (first use in this function)

src/petsc4py.PETSc.c:179763:30: error: ‘DMDA_BOUNDARY_MIRROR’ undeclared (first use in this function)

src/petsc4py.PETSc.c:179775:30: error: ‘DMDA_BOUNDARY_PERIODIC’ undeclared (first use in this function)

error: command '/usr/bin/mpicc' failed with exit status 1

----------------------------------------
Command /usr/bin/python -c "import setuptools;__file__='/home/fpoulin/software/build/petsc4py/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-bM1J37-record/install-record.txt --user failed with error code 1
Storing complete log in /home/fpoulin/.pip/pip.log

2014-07-10T16:42:06+00:00

Lisandro Dalcin

About the install issue, I guess you are not really installing from git master. Please pip uninstall petsc4py and reinstall like this:

$ pip install https://bitbucket.org/petsc/petsc4py/get/master.tar.gz

About the performance issues on multicore machines: welcome to the real world. Take a look to these comments http://www.mcs.anl.gov/petsc/documentation/faq.html#computers , and also run the streams benchmark (cd $PETSC_DIR && make streams NPMAX=16) to check the (likely lame) parallel performance of your machine for sparse linear algrebra (or any other memory-bound computation).

2014-07-10T18:43:27+00:00

Lisandro Dalcin

changed status to invalid

I'm closing this issue as invalid. Please continue commenting here or mail me to my personal address for further discussion.

2014-07-10T18:44:58+00:00

Francis Poulin

Sorry that I forgot how to install the master. Now it works great.

To install slepc4py I tried

pip install --user slepc4py

and I get similar errors to before.
I am looking around how to install the master for slepc4py, which I assume I should do. Do you know how I can do that?

Thank you again.

2014-07-10T19:15:08+00:00

Lisandro Dalcin

For any serious Python package that lives in Bitbucket, you can build the URL https://bitbucket.org/<user>/<repository>/get/<branch>.tar.gz and ask pip to install it.

$ pip install https://bitbucket.org/slepc/slepc4py/get/master.tar.gz

If you have git installed, it is also possible to do (note the git+ tag at the beginning of the URL):

$ pip install git+https://bitbucket.org/slepc/slepc4py

2014-07-10T19:40:54+00:00

Francis Poulin

I am sorry for asking so many basic equations but thank you very much for the help. I am happy to say that slepc4py is now working.

I will let you know if I can give you any helpful feedback on this software.

2014-07-10T19:46:51+00:00

Francis Poulin

After all that I learned today I thought I would try and install petsc4py on my macbook pro right away, before I forget some details.

I have been able to install PETSc (dalcinl/tao-complex_scalars), SLEPc (master) and was told that petsc4py was installed successfully. However, when I try and run an example I get a lot of trouble, see below.

One issue that concerns me is that I have python installed from anaconda. If necessary I can always install another version.

fpoulin$ python heat.py 
Traceback (most recent call last):
  File "heat.py", line 4, in <module>
    petsc4py.init(sys.argv)
  File "/Users/fpoulin/.local/lib/python2.7/site-packages/petsc4py/__init__.py", line 42, in init
    PETSc = petsc4py.lib.ImportPETSc(arch)
  File "/Users/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc
    return Import('petsc4py', 'PETSc', path, arch)
  File "/Users/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/__init__.py", line 64, in Import
    module = imp.load_module(fullname, fo, fn, stuff)
ImportError: dlopen(/Users/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/darwin10.6.0-c-debug/PETSc.so, 2): Symbol not found: _ompi_mpi_c_double_complex
  Referenced from: /Users/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/darwin10.6.0-c-debug/PETSc.so
  Expected in: flat namespace

2014-07-11T02:41:34+00:00

Lisandro Dalcin

There is an issue with the MPI libraries. Please paste the output of the following command:

otool -L /Users/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/darwin10.6.0-c-debug/PETSc.so

2014-07-11T09:28:07+00:00

Francis Poulin

I had a choice of what libraries to use when I installed PETSc. In the end I used clang and gfortran and asked it to download mpich2. I had mpi libraries installed in two places: 1) /usr/local and 2) /Applications/anaconda. Each had their problems but might be fixable.

The output is

/Users/fpoulin/.local/lib/python2.7/site-packages/petsc4py/lib/darwin10.6.0-c-debug/PETSc.so: /Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib/libpetsc.3.5.dylib (compatibility version 3.5.0, current version 3.5.0) /Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib/libpmpich.12.dylib (compatibility version 13.0.0, current version 13.0.0) /Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib/libmpich.12.dylib (compatibility version 13.0.0, current version 13.0.0) /Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib/libopa.1.dylib (compatibility version 2.0.0, current version 2.0.0) /Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib/libmpl.1.dylib (compatibility version 2.0.0, current version 2.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.4) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

2014-07-11T10:52:07+00:00

Lisandro Dalcin

OK, it seems that you have a mess with your MPI. You have to use the same MPI to build petsc, slepc, petsc4py, slepc4py. I guess that having an MPI installed in /usr/local will be really problematic, because in OS X /usr/local is always added to the compiler and linker search paths. You either use the MPI you have in /usr/local to build all the other packages, or remove any traces of your MPI in /usr/local and ask PETSc to download MPICH.

2014-07-11T14:54:05+00:00

Francis Poulin

I am happy to try and do that. The problem that I get is that it cannot find mpif.h. Do you know of a way of specifying that in the configure file? I presume it must be there I just need to point it in the right direction?

===============================================================================
             Configuring PETSc to compile on your system                       
===============================================================================
TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:661)                                                                           *******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Fortran error! mpif.h could not be located at: []
*******************************************************************************

2014-07-11T15:04:34+00:00

Lisandro Dalcin

I bet you installed MPI in /usr/local before installing gfortran, so you MPI was build without Fortran support. Either reinstall MPI, or pass the option --with-fc=0 to PETSc's configure.

2014-07-11T20:42:52+00:00

Francis Poulin

I got some help, a lot of help, from some PETSc people and managed to get petsc4py installed. This uses the clang and gfortran libraries and downloaded mpich. This required removing my anaconda from the path but well worth it.

At the moment I am trying to install slepc4py but having problems. It seems to not to find the distribution. Is this something you could help with?

Thanks, Francis

pip install --user git+https://bitbucket.org/slepc/slepc4py
Downloading/unpacking git+https://bitbucket.org/slepc/slepc4py
  Cloning https://bitbucket.org/slepc/slepc4py to /var/folders/nl/wfwj3nps2wx8m6jx9bh_gslc0000gn/T/pip-zT2CzS-build
  Running setup.py egg_info for package from git+https://bitbucket.org/slepc/slepc4py

Downloading/unpacking petsc4py>=3.5 (from slepc4py==3.5)
  Could not find a version that satisfies the requirement petsc4py>=3.5 (from slepc4py==3.5) (from versions: 0.6.3, 0.7.0, 0.7.1, 0.7.2, 0.6.3, 0.7.0, 0.7.1, 0.7.2, 0.7.4, 0.7.5, 1.0.1, 1.0.2, 1.0.3, 1.1.1, 1.1.2, 1.1, 1.2, 3.3.1, 3.3, 0.6.0, 3.4, dev)
Cleaning up...
No distributions matching the version for petsc4py>=3.5 (from slepc4py==3.5)
Storing complete log in /Users/fpoulin/.pip/pip.log

2014-07-12T12:08:32+00:00

Lisandro Dalcin

I'm not sure what's going on. Did you install petsc4py using pip? Perhaps the issue is related to petsc4py 3.5 not being yet registered in the Python package index.

So, in order to get slepc4py installed, I recommend to do the following:

git clone https://bitbucket.org/slepc/slepc4py.git
cd slepc4py
python setup.py install --user

2014-07-15T16:01:27+00:00

Francis Poulin

I installed petsc4py when I installed petsc, I was told I could and thought that might be easier.

I tried what you posted and it worked beautifully.

Thank you very much for the help.

2014-07-15T16:49:30+00:00

Comments (29)