is concurrent installation of real and complex petsc4py modules possible?

[Drew Parsons]

The structure of the PETSc installation makes it possible to install different versions and different configurations side by side. You can choose which one you want to work with at run time (or build time) using PETSC_DIR, choosing between real and complex number support, for instance.

Is it feasible to maintain a similar kind of concurrent installation for petsc4py? Installation of the modules themselves could be easy, e.g. rename as /usr/lib/python3/dist-packages/petsc4py_real and petsc4py_complex, perhaps provide petsc4py as a symlink to the preferred configuration. Then each of "import petsc4py_real", "import petsc4py_complex", "import petsc4py" would run in python.

But then the internals of the module still refer to petsc4py. So while "import petsc4py_complex" will present get_config(), get_include() and init(), and petsc4py_complex.init() will run without error, at the same time it does not load up petsc4py_complex.PETSc.

So only "import petsc4py" runs correctly, though its petsc configuration can be switched by swapping a petsc4py symlink in the python module dir between petsc4py_real and petsc4py_complex.

Would there be a straightforward way of activating petsc4py_real and petsc4py_complex modules so that different configurations can be used by different python applications (one might use import petsc4py_complex as petsc4py for instance) at the same time?

[Lisandro Dalcin]

Maybe the following simple approach currently implemented will fit your needs?

Install petsc4py this way:

export PETSC_DIR=/path/to/petsc
export PETSC_ARCH=arch-real:arch-complex:arch-whatever # colon separated PETSC_ARCH values
python setup.py install # or `pip install petsc4py` 

Then, at runtime, you just export PETSC_ARCH=arch-you-want, and then from petsc4py import PETSc will give you what you asked for.

If you do not like environment variables, you can also try:

import petsc4py
petsc4py.init(arch="arch-complex")
from petsc4py import PETSc
# ready to conquer the (complex) world ?

If this is what you had in mind and enough for you, please close this issue. Otherwise, let's continue the discussion... And remember: never EVER complain about all this not being documented, or you risk getting back a "Pull requests very welcome!". Oh, hold on, it is documented, though admittedly obscure and hard to find: https://www.mcs.anl.gov/petsc/petsc4py-current/docs/apiref/petsc4py-module.html#init

[Drew Parsons]

Thanks Lisandro. The config step doesn't seem to be set up for PETSC_ARCH. e.g. conf/baseconf.py: petscversion_h = os.path.join(petsc_dir, 'include', 'petscversion.h'), petsc_arch is not invoked between petsc_dir and include. _get_petsc_version doesn't take petsc_arch as an option.

I'm trying to work with the Debian pybuild system, which calls setup.py config/build/install, each in turn.

[Lisandro Dalcin]

Can you please provide a reproducible example of how things break? My suggestions apply for the case of you using a PETSc source tree where you have multiple archs built under a single $PETSC_DIR, and not for prefix-installations of PETSc (which maybe is the Debian case?). In any case, I'm totally fine with changing things to make this possible.

Can you reproduce the following command line? (of course, adjust PETSC_ARCH to fit your case):

$ PETSC_ARCH=arch-linux2-c-debug:arch-test python setup.py config
running config
----------------------------------------------------------------------
PETSC_DIR:   /home/devel/petsc/dev
----------------------------------------------------------------------
PETSC_ARCH:  arch-linux2-c-debug
 * scalar-type: real
 * precision:   double
 * language:    CONLY
 * compiler:    mpicc
 * linker:      mpicc
----------------------------------------------------------------------
PETSC_ARCH:  arch-test
 * scalar-type: complex
 * precision:   double
 * language:    CXXONLY
 * compiler:    mpicxx
 * linker:      mpicxx
----------------------------------------------------------------------

[Drew Parsons]

You're right, Debian uses the prefix installation, so PETSC_ARCH is not used after installation (PETSC_ARCH="" in the installed petscconf.h). PETSC_DIR is /usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-real or /usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-complex.

If you interpret that as base PETSC_DIR + PETSC_ARCH then config says:

$ PETSC_DIR=/usr/lib/petscdir/petsc3.10 PETSC_ARCH=x86_64-linux-gnu-real:x86_64-linux-gnu-complex python3 setup.py config
running config
----------------------------------------------------------------------
PETSC_DIR:   /usr/lib/petscdir/petsc3.10
error: [Errno 2] No such file or directory: '/usr/lib/petscdir/petsc3.10/include/petscversion.h'

The full PETSC_DIR does work, but of course only handles one configuration at a time

$ PETSC_DIR=/usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-real python3 setup.py config
running config
----------------------------------------------------------------------
PETSC_DIR:   /usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-real
----------------------------------------------------------------------
PETSC_ARCH:  
 * scalar-type: real
 * precision:   double
 * language:    CONLY
 * compiler:    mpicc
 * linker:      mpicc
----------------------------------------------------------------------

$ PETSC_DIR=/usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-complex python3 setup.py config
running config
----------------------------------------------------------------------
PETSC_DIR:   /usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-complex
----------------------------------------------------------------------
PETSC_ARCH:  
 * scalar-type: complex
 * precision:   double
 * language:    CONLY
 * compiler:    mpicc
 * linker:      mpicc
----------------------------------------------------------------------

[Drew Parsons]

We’ve now got a client program that can be built against PETSc with either real or complex number support. Dolfinx is the next-generation version of Dolfin for the FEniCS suite (https://github.com/FEniCS/dolfinx). It can be built for either real or complex numbers.

Ideally we’d want any system to be able to choose the real or complex number build for dolfinx without having to uninstall/reinstall the petsc4py packages. PETSc itself enables that via prefix installations and PETSC_DIR. It could be a good time therefore to revisit this Issue to see if we can do the same with petsc4py (controlled via PETSC_DIR rather than PETSC_ARCH).

[Lisandro Dalcin]

This should be rather trivial to implement. What I’m not sure is the “interface” to ask setup.py to do it. Right now, you build for multiply arches with PETSC_ARCH=arch-a:arch-b:arch-c. We could also support PETSC_DIR=prefix-a:prefix-b:prefix-c. But then, how do we let users select the version they want to use at runtime?

[Drew Parsons]

I think it's reasonable to expect users to set PETSC_DIR as an environment variable if they want to select the non-default installation at runtime. That makes it the same kind of handling from the user’s perspective as what you’ve currently got in place with PETSC_ARCH

Another option is to provide a switch inside petsc4py itself allowing progammatic change of build, something like

import petsc4py
petsc4py.petsc_dir='/usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-complex/'

where the value of petsc4py.petsc_dir would have to be one of the available builds, perhaps accessible by petsc4py.list_available_petsc_dir() But perhaps this second option is not so viable, since I imagine multiple PETSc inits would be a problem.

[Drew Parsons]

Unless (for that second option), you can adapt the mechanism you suggested with arch, to use something like

import petsc4py
petsc4py.init(petsc_dir="/usr/lib/petscdir/petsc3.10/x86_64-linux-gnu-complex/")

instead of

import petsc4py
petsc4py.init(arch="arch-complex")

‌

‌

[Lisandro Dalcin]

The more I think about this, the less I like it. Your request is definitely non-standard. Multiple PETSC_ARCH is something that PETSc supports and has been carefully designed for, so it is natural for petsc4py to support it. But multiple prefix installs are a different beast. This should be best handled with environment modules (as done in Fedora), but IIRC that’s not an available option on Debian/Ubuntu.

In the same way that you install PETSc in two different prefixes, petsc4py should be installed in two different places (maybe inside each PETSC_DIR?), and then install a *.pth file in python site-packages to implement the logic of appending to sys.path using the value of the PETSC_DIR environ var. And most likely you will have to do something similar for Dolfinx.

[Drew Parsons]

We can make something like that work with a petsc4py_real, petsc4py_complex installation, alongside a petsc4py dummy module which selects the one needed according to the value of PETSC_DIR. I’ve been working on a selective import with a similar mechanism for h5py, selecting between serial and mpi build (cf. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944769 )

Installing actually inside the PETSC_DIR is an interesting proposition. Not sure if it makes it too messy updating PYTHONPATH or keeping the list of available petsc4pys up to date to make it workable, but can look into it. Maybe a symlink from PYTHONPATH/petsc4py/<some_petsc_build> to the appropriate subdir in PETSC_DIR can make it work.

‌

[Lisandro Dalcin]

I hate these dummy modules. IMHO, *.pth files are a far better option.

Try the following bash script on your side (minimal PETSc build, no MPI, no Fortran, assumes blas/lapack installed on the system, just to make it quick to build). As you can see, this is quite trivial to implement, it is robust and follow Python standard practice, it does not require changes to petsc4py, just some boring work on the packaging side (which I’m not interested in contributing to, sorry). Same approach could be implemented for any other Python package like Dolfinx wanting to support multiple PETSc builds. Now I’m done for good!

Disclaimer: Only tested on Fedora 31 with PETSc/petsc4py 3.12.

#!/bin/bash
set -eu

# For testing purposes, install stuff at $HOME
ROOTDIR=$HOME


# Download, build and install PETSc real/complex variants

wget -q http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-3.12.4.tar.gz
tar -zxf petsc-lite-3.12.4.tar.gz
cd petsc-3.12.4

VARIANT=real
export PETSC_DIR=$PWD
export PETSC_ARCH=arch-$VARIANT
./configure --prefix=$ROOTDIR/petscdir/petsc3.12/x86_64-linux-gnu-$VARIANT --with-fc=0 --with-mpi=0 --with-scalar-type=$VARIANT
make
make install
unset PETSC_DIR
unset PETSC_ARCH

VARIANT=complex
export PETSC_DIR=$PWD
export PETSC_ARCH=arch-$VARIANT
./configure --prefix=$ROOTDIR/petscdir/petsc3.12/x86_64-linux-gnu-$VARIANT --with-fc=0 --with-mpi=0 --with-scalar-type=$VARIANT
make
make install
unset PETSC_DIR
unset PETSC_ARCH

cd ..


# Download, build and install petsc4py real/complex variants

wget -q https://bitbucket.org/petsc/petsc4py/downloads/petsc4py-3.12.0.tar.gz
tar -xzf petsc4py-3.12.0.tar.gz
cd petsc4py-3.12.0

VARIANT=real
export PETSC_DIR=$ROOTDIR/petscdir/petsc3.12/x86_64-linux-gnu-$VARIANT
python3 setup.py clean --all
python3 setup.py install --install-lib=$PETSC_DIR/lib
unset PETSC_DIR

VARIANT=complex
export PETSC_DIR=$ROOTDIR/petscdir/petsc3.12/x86_64-linux-gnu-$VARIANT
python3 setup.py clean --all
python3 setup.py install --install-lib=$PETSC_DIR/lib
unset PETSC_DIR

cd ..


# Install petsc4py.pth file in user site-packages (~/.local/...)

site_packages=$(python3 -m site --user-site)
petsc4py_pth=$site_packages/petsc4py.pth
mkdir -p $site_packages
printf "import sys, os; " > $petsc4py_pth
printf "p = '%s'; " $ROOTDIR/petscdir/petsc3.12/x86_64-linux-gnu-real >> $petsc4py_pth
printf "p = os.path.join(os.getenv('PETSC_DIR') or p, 'lib'); " >> $petsc4py_pth
printf "p in sys.path or sys.path.append(p);\n" >> $petsc4py_pth


# Test petsc4py installations

# PETSC_DIR not set
python3 -c "from petsc4py import PETSc; print(PETSc); print(PETSc.ScalarType)"

# PETSC_DIR set to real variant
VARIANT=real
export PETSC_DIR=$ROOTDIR/petscdir/petsc3.12/x86_64-linux-gnu-$VARIANT
python3 -c "from petsc4py import PETSc; print(PETSc); print(PETSc.ScalarType)"

# PETSC_DIR set to complex variant
VARIANT=complex
export PETSC_DIR=$ROOTDIR/petscdir/petsc3.12/x86_64-linux-gnu-$VARIANT
python3 -c "from petsc4py import PETSc; print(PETSc); print(PETSc.ScalarType)"


# Remember to cleanup your user site-package after testing!
# rm $(python3 -m site --user-site)/petsc4py.pth

‌

[Lisandro Dalcin]

Note that the petsc4py.pth file need to have a single line with semicolon-separated statements!

[Drew Parsons]

Thanks Lisandro. I’m not familiar with .pth, I’ll take a close look.

[Drew Parsons]

Seems to be working fine, working now in Debian petsc4py 3.12.0-6. Debian already defines /usr/lib/petsc as the default PETSc, so I used

import sys, os; petsc_dir = '/usr/lib/petsc'; petsc_dir = os.path.join(os.getenv('PETSC_DIR') or petsc_dir, 'lib/python3/dist-packages'); petsc_dir in sys.path or sys.path.append(petsc_dir);

Likewise for slepc4py.

I guess we can close this Issue, unless it’s worth working petsc4py.pth and installation under PETSC_DIR into the main source code.

[Lisandro Dalcin]

I don’t think it is worth it to add these pth files to petsc4py, they are more or less trivial, and its contents will depend on specific packaging details of Linux distros.