When compiled with petsc4py, passing options to petsc via --petsc.pc_type whatever doesn't work

[Patrick Farrell]

Dolfin passes options to PETSc by parsing them into an argc and an argv and passing these to PetscInitialize in SubSystemsManager::init_petsc.

Unfortunately, when dolfin is compiled with petsc4py support, this particular call to PetscInitialize does nothing, as PetscInitialize has already been called when petsc4py is imported, without the right arguments.

[Patrick Farrell]

Also note that the PPA packages are compiled against petsc4py.

[Prof Garth Wells]

We had problems in the past with something similar with PyTrilinos (the order in which MPI was initialised).

We should make sure that DOLFIN initialises PETSc before petsc4py, and check that petsc4py calls PetscInitialized to avoid calling PetscInitialize twice.

[Lawrence Mitchell]

petsc4py does, I believe, check PetscInitialized. I'm not sure where in the swigifying one can call dolfin::init_petsc appropriately. Here's a python way that fixes things for Patrick, however, it's not robust in that it imports petsc4py even if dolfin was compiled without petsc4py support. There's not way of checking if that support was compiled in without already import petsc4py.

diff --git a/site-packages/dolfin/__init__.py b/site-packages/dolfin/__init__.py
index c5ccc6d..c0dbf0a 100644
--- a/site-packages/dolfin/__init__.py
+++ b/site-packages/dolfin/__init__.py
@@ -12,6 +12,26 @@ PROGRESS = 16
 # Remove some unnecessary imports from UFL
 del set_handler, set_level

+# Ensure petsc4py is initialised with DOLFIN command line arguments.
+try:
+    import petsc4py
+    import sys
+    args = sys.argv[1:]
+    petsc_args = []
+    add = False
+    for arg in args:
+        if add and not arg.startswith('--petsc.'):
+            petsc_args.append(arg)
+            add = False
+        if arg.startswith('--petsc.'):
+            add = True
+            petsc_args.append('-' + arg[8:])
+
+    petsc4py.init(['dolfin'] + petsc_args)
+    del petsc4py, sys, args, petsc_args, add
+except ImportError:
+    pass
+
 # Import names from the compiled cpp modules
 import cpp
 from dolfin.cpp import *

[Johan Hake]

We could generate some optional import code during configuration. If dolfin is configured with petscy4py we add the above code if not we do not add anything. We already generate python code for the default parameters.

[Johan Hake]

So the real problem is that we only allow setting PETSc command line options through PetscInitialize. So when PETSc is initialized the command line options are silently ignored.

I suggest we update the logic of parsing PETSc options so it calls PetscOptionsInsert if PETSc already is initialized. Logic for this should be possible to add to Parameters::parse_petsc with something like:

// Pass options to PETSc
if (SubSystemsManager::responsible_petsc())
{
  #ifdef HAS_PETSC
  PetscOptionsInsert(&argc, &argv, PETSC_NULL);
  #endif
}
else
{
  SubSystemsManager::init_petsc(argc, argv);
}

That raised a question of how one through the DOLFIN interface access if PETSc is already initialized, and I could only find SubSystemsManager::responsible_petsc(). Is this the only one way? It seems a bit convoluted as that method should answer who is responsible for initializing MPI, which btw is not that obvious from the name of the method...

[Johan Hake]

Fix issue 196.

If PETSc is already initialised (e.g. from importing petsc4py), insert the command-line arguments via PetscOptionsInsert.

→ <<cset 33dc5ba86915>>

[Prof Garth Wells]

Removing milestone: 1.4 (automated comment)