- changed title to test failure in Fedora rawhide
test failure in Fedora rawhide
Issue #138
resolved
Similar to to #127, but now more architectures are failing, with different tests:
x86_64, aarch64, ppc64le:
BUILDSTDERR: ======================================================================
BUILDSTDERR: FAIL: testCompareAndSwap (test_rma.TestRMASelf)
BUILDSTDERR: ----------------------------------------------------------------------
BUILDSTDERR: Traceback (most recent call last):
BUILDSTDERR: File "test/test_rma.py", line 228, in testCompareAndSwap
BUILDSTDERR: self.assertEqual(rbuf[1], -1)
BUILDSTDERR: AssertionError: 0 != -1
BUILDSTDERR: ======================================================================
BUILDSTDERR: FAIL: testFetchAndOp (test_rma.TestRMASelf)
BUILDSTDERR: ----------------------------------------------------------------------
BUILDSTDERR: Traceback (most recent call last):
BUILDSTDERR: File "test/test_rma.py", line 190, in testFetchAndOp
BUILDSTDERR: self.assertEqual(rbuf[1], -1)
BUILDSTDERR: AssertionError: -34 != -1
BUILDSTDERR: ======================================================================
BUILDSTDERR: FAIL: testCompareAndSwap (test_rma.TestRMAWorld)
BUILDSTDERR: ----------------------------------------------------------------------
BUILDSTDERR: Traceback (most recent call last):
BUILDSTDERR: File "test/test_rma.py", line 228, in testCompareAndSwap
BUILDSTDERR: self.assertEqual(rbuf[1], -1)
BUILDSTDERR: AssertionError: 0 != -1
BUILDSTDERR: ======================================================================
BUILDSTDERR: FAIL: testFetchAndOp (test_rma.TestRMAWorld)
BUILDSTDERR: ----------------------------------------------------------------------
BUILDSTDERR: Traceback (most recent call last):
BUILDSTDERR: File "test/test_rma.py", line 190, in testFetchAndOp
BUILDSTDERR: self.assertEqual(rbuf[1], -1)
BUILDSTDERR: AssertionError: 123 != -1
BUILDSTDERR: ----------------------------------------------------------------------
BUILDSTDERR: Ran 1102 tests in 4.957s
BUILDSTDERR: FAILED (failures=4, skipped=46)
Seems entirely reproducible, always the same four tests fail, and on ppc64le, in addition the two tests from #127.
For refrence:
+ ompi_info
Package: Open MPI mockbuild@buildhw-10.phx2.fedoraproject.org Distribution
Open MPI: 4.0.1
Open MPI repo revision: v4.0.1
Open MPI release date: Mar 26, 2019
Open RTE: 4.0.1
Open RTE repo revision: v4.0.1
Open RTE release date: Mar 26, 2019
OPAL: 4.0.1
OPAL repo revision: v4.0.1
OPAL release date: Mar 26, 2019
MPI API: 3.1.0
Ident string: 4.0.1
Prefix: /usr/lib64/openmpi
Configured architecture: x86_64-unknown-linux-gnu
Configure host: buildhw-10.phx2.fedoraproject.org
Configured by: mockbuild
Configured on: Fri Jun 21 04:36:41 UTC 2019
Configure host: buildhw-10.phx2.fedoraproject.org
Configure command line: '--prefix=/usr/lib64/openmpi' '--mandir=/usr/share/man/openmpi-x86_64' '--includedir=/usr/include/openmpi-x86_64' '--sysconfdir=/etc/openmpi-x86_64' '--disable-silent-rules' '--enable-builtin-atomics' '--enable-mpi-cxx' '--enable-mpi-java' '--enable-mpi1-compatibility' '--with-sge' '--with-valgrind' '--enable-memchecker' '--with-hwloc=/usr' '--with-libevent=external' '--with-pmix=external' 'CC=gcc' 'CXX=g++' 'LDFLAGS=-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CFLAGS= -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'CXXFLAGS= -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'FC=gfortran' 'FCFLAGS= -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
Built by: mockbuild
Built on: Fri Jun 21 04:42:02 UTC 2019
Built host: buildhw-10.phx2.fedoraproject.org
C bindings: yes
C++ bindings: yes
Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
Fort mpi_f08 compliance: The mpi_f08 module is available, but due to limitations in the gfortran compiler and/or Open MPI, does not support the following: array subsections, direct passthru (where possible) to underlying Open MPI's C functionality
Fort mpi_f08 subarrays: no
Java bindings: yes
Wrapper compiler rpath: runpath
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C compiler family name: GNU
C compiler version: 9.1.1
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fort compiler: gfortran
Fort compiler abs: /usr/bin/gfortran
Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
Fort 08 assumed shape: yes
Fort optional args: yes
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: yes
Fort BIND(C) (all): yes
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): yes
Fort TYPE,BIND(C): yes
Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
Fort PROTECTED: yes
Fort ABSTRACT: yes
Fort ASYNCHRONOUS: yes
Fort PROCEDURE: yes
Fort USE...ONLY: yes
Fort C_FUNLOC: yes
Fort f08 using wrappers: yes
Fort MPI_SIZEOF: yes
C profiling: yes
C++ profiling: yes
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes, OMPI progress: no, ORTE progress: yes, Event lib: yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
IPv6 support: no
MPI1 compatibility: yes
MPI extensions: affinity, cuda, pcollreq
FT Checkpoint support: no (checkpoint thread: no)
C/R Enabled Debugging: no
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.0.1)
MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.0.1)
MCA btl: uct (MCA v2.1.0, API v3.1.0, Component v4.0.1)
MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.0.1)
MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.0.1)
MCA btl: usnic (MCA v2.1.0, API v3.1.0, Component v4.0.1)
MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA crs: none (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA event: external (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA memchecker: valgrind (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA pmix: ext3x (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v4.0.1)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA dfs: app (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA dfs: test (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA dfs: orted (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component v4.0.1)
BUILDSTDERR: MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v+ cp .__init__openmpi.py src/mpi4py/__init__.py
4.0.1)
MCA ess: tm (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA notifier: syslog (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA plm: tm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA routed: debruijn (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v4.0.1)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA coll: monitoring (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fbtl: pvfs2 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fs: pvfs2 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v4.0.1)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.0.1)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v4.0.1)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component v4.0.1)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component v4.0.1)
Failing build: https://koji.fedoraproject.org/koji/taskinfo?taskID=36679907.
Comments (4)
-
reporter
-
Unless someone can prove that my testing is bogus, there is nothing I can do about it. You will have to report the issue upstream with the Open MPI Fedora maintainers and eventually the developers. For the time being, maybe you should just patch the sources to skip these tests, just add the decorator
@unittest.skipMPI('openmpi'). -
In issue 137, I found a test bug leading to memory corruption that may affect RMA calls , could you please patch using 7de7b5c?
-
- changed status to resolved
- Log in to comment
