1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

fix misisng initializating and incorrect locking in TATPETSc

Issue #2421 resolved
Roland Haas created an issue

TATPETSc contains code that fails to run and makes the test Poisson/charge_tatelliptic_petsc from CactusTest fail.

Comments (10)

  2. Roland Haas reporter

    Please see also the discussion in https://github.com/EinsteinToolkit/ExternalLibraries-PETSc/pull/1 where the error message is shown:

    Right now with 3.12.5 (but not the version in the release) it fails the charge_tatelliptic_petsc test of CactusExamples/Poisson which uses TATelliptic and TATPETSc but not EllPETSc (which seems to be not used by any par or ccl file in the ET). On 2 MPI ranks the error produced bycharge_tatelliptic_petsc is about differing scalar multipliers and an aXPlusY or so vector call, while on a single MPI rank the error is (the maybe more helpful):

    [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Operation done in wrong order
[0]PETSC ERROR: You should call DMSetUp() first
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020
[0]PETSC ERROR: Cactus on a  named ekohaes8 by rhaas Sun Jun 21 15:27:28 2020
[0]PETSC ERROR: Configure options --LDFLAGS="-rdynamic  -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/usr/lib/gcc/x86_64-linux-gnu/10/ -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10/ -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/gcc/x86_64-linux-gnu/10/ -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10/ -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10 -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10 -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10" --LIBS=" -lm -lm" --doCleanup=0 --prefix=/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc --with-cpp=cpp --CPPFLAGS="  -fopenmp -Wall " --with-cc=gcc-10 --CFLAGS="-g -std=gnu99 -O2 -fopenmp -Wall -rdynamic  -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/usr/lib/gcc/x86_64-linux-gnu/10/ -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10/ -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/gcc/x86_64-linux-gnu/10/ -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10/ -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10 -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10 -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10" --with-cxx=g++-10 --CXXFLAGS="-pipe -std=c++0x -O2 -fopenmp -Wall -rdynamic  -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/usr/lib/gcc/x86_64-linux-gnu/10/ -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10/ -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/gcc/x86_64-linux-gnu/10/ -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10/ -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10 -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10 -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/PETSc/lib -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/LAPACK -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/external/BLAS -L/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/10" --with-fc=0 --with-ar=ar --AR_FLAGS=rcs --with-shared-libraries=0 --with-mpi=yes --with-mpi-include=/usr/lib/x86_64-linux-gnu/openmpi/include --with-mpi-lib="[/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so,/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so]" --with-mpi-compilers=no --with-mpiexec=false --with-ssl=no --with-x=no --with-blas-lib="[lapack,gfortran,blas,gfortran]" --with-lapack-lib="[lapack,gfortran,blas,gfortran]" --known-64-bit-blas-indices=yes MAKE="make -j1"
[0]PETSC ERROR: #1 DMDASetFieldName() line 68 in /data/rhaas/postdoc/gr/cactus/ET_trunk/configs/poison/scratch/build/PETSc/petsc-3.12.5/src/dm/impls/da/dacorn.c
[0]PETSC ERROR: #2 TATPETSc_solve() line 355 in /data/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/CactusElliptic/TATPETSc/src/solve.c
WARNING level 1 from host ekohaes8 process 0
  while executing schedule bin CCTK_POSTINITIAL, routine Poisson::Poisson_solve
  in thorn Poisson, file /data/rhaas/postdoc/gr/cactus/ET_trunk/arrangements/CactusExamples/Poisson/src/uniform_charge.c:72:
  -> Failed to solve elliptic equation

    I am not familiar with PETSc but it would seem to me that initializing things in the wrong order should have always been wrong, with it not triggering an error in previous PETSc versions not being an indication for it being correct but more that PETSc just did not check.

  6. Roland Haas reporter

    The pull request makes use of PETSc functions VecGetArrayRead and VecRestoreArrayRead which have existed since PETSc 3.2 so should be present in all versions of PETSc we are likely to encounter on clusters (it is from Thu Sep 8 12:03:44 2011 -0500).

  8. Roland Haas reporter

    Turns out that this same failure now happens with PETSc 3.2 as well. No idea why they were not triggered before.

  11. Log in to comment
Assignee
Type
bug
Priority
major
Status
resolved
Component
EinsteinToolkit thorn
Milestone
–
Version
–
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!