I met the “Accuracy test failed”
problem when I try to test the sample “precession” provided
in magic-sph(https://github.com/magic-sph/magic).
The testing is done on two systems: one is a workstation whose CPU is dual Intel(R) Xeon(R) Gold 5218 CPU
, and the OS is Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-111-generic x86_64)
. The other one is a cluster, whose CPU is dual Intel(R) Xeon(R) Gold 6248 CPU
and the OS is CentOS Linux release 7.7.1908 (Core)
. Both the compiler and math library are intel parallel studio 2020 cluster edition.
What confused me is that the problem is only happened on the first system.
I do a little research and try to compile SHTns without openmp support and set verbose value to 3, like this: “export CC=icc; export FC=ifort; ./configure --enable-mkl --enable-ishioka --enable-magic-layout --prefix=/home/shen/local --enable-verbose=3 --disable-openmp”
. I compiled magic-sph also without openmp, like this: “cmake .. -DUSE_SHTNS=yes -DUSE_OMP=no
“.
I test the “precession” sample three times with only 1 mpi processes. each time it gave me different SPH value. here are the parts of the log:
[SHTns 3.4.2] built for MagIC Jul 21 2020, 22:58:48, id: avx512,ishioka
Lmax=42, Mmax*Mres=42, Mres=1, Nlm=946 [1 threads, no Condon-Shortley phase, Robert form, orthonormalized]
> Condon-Shortley phase = 0, normalization = 0
=> using FFTW : Mmax=42, Nphi=128, Nlat=64 (data layout : phi_inc=72, theta_inc=1, phi_embed=128)
=> using Gauss nodes
Gauss quadrature for 3/2.x^2 = 1 (should be 1.0) error = -1.55431e-15
+ polar optimization threshold = 1.0e-10
finding optimal algorithm*****************
+ SHT accuracy = 0.241
ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
=> SHTns is ready.
! SHTns uses theta padding with nlat_padded= 72
**************forrtl: severe (154): array index out of bounds
[SHTns 3.4.2] built for MagIC Jul 21 2020, 22:58:48, id: avx512,ishioka
Lmax=42, Mmax*Mres=42, Mres=1, Nlm=946 [1 threads, no Condon-Shortley phase, Robert form, orthonormalized]
> Condon-Shortley phase = 0, normalization = 0
=> using FFTW : Mmax=42, Nphi=128, Nlat=64 (data layout : phi_inc=72, theta_inc=1, phi_embed=128)
=> using Gauss nodes
Gauss quadrature for 3/2.x^2 = 1 (should be 1.0) error = -1.55431e-15
+ polar optimization threshold = 1.0e-10
finding optimal algorithm*****************
+ SHT accuracy = 0.683
ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
=> SHTns is ready.
! SHTns uses theta padding with nlat_padded= 72
**************forrtl: severe (154): array index out of bounds
[SHTns 3.4.2] built for MagIC Jul 21 2020, 22:58:48, id: avx512,ishioka
Lmax=42, Mmax*Mres=42, Mres=1, Nlm=946 [1 threads, no Condon-Shortley phase, Robert form, orthonormalized]
> Condon-Shortley phase = 0, normalization = 0
=> using FFTW : Mmax=42, Nphi=128, Nlat=64 (data layout : phi_inc=72, theta_inc=1, phi_embed=128)
=> using Gauss nodes
Gauss quadrature for 3/2.x^2 = 1 (should be 1.0) error = -1.55431e-15
+ polar optimization threshold = 1.0e-10
finding optimal algorithm*****************
+ SHT accuracy = 0.467
ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
=> SHTns is ready.
! SHTns uses theta padding with nlat_padded= 72
**************ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
forrtl: severe (154): array index out of bounds
Please Notice that the SHT accuracy
value are different, which are 0.241, 0.683 and 0.467 respectively.
I also tried to run the test with 4 MPI processes, the parts of logs are:
[SHTns 3.4.2] built for MagIC Jul 21 2020, 22:58:48, id: avx512,ishioka
Lmax=42, Mmax*Mres=42, Mres=1, Nlm=946 [1 threads, no Condon-Shortley phase, Robert form, orthonormalized]
> Condon-Shortley phase = 0, normalization = 0
=> using FFTW : Mmax=42, Nphi=128, Nlat=64 (data layout : phi_inc=72, theta_inc=1, phi_embed=128)
=> using Gauss nodes
Gauss quadrature for 3/2.x^2 = 1 (should be 1.0) error = -1.55431e-15
+ polar optimization threshold = 1.0e-10
finding optimal algorithm************************************************************ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
*
+ SHT accuracy = 8.18e-14
=> SHTns is ready.
! SHTns uses theta padding with nlat_padded= 72
***********************************************************ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
**
! MPI transpose strategy for 5 fields
! isend/irecv/waitall communicator= 1.675E-03 s
! alltoallv communicator = 2.730E-03 s
! alltoallw communicator = 1.491E-03 s
...
! STARTING TIME INTEGRATION AT:
start_time = 0.0000000000E+00
step no = 0
start dt = 1.0000E-05
! Starting time integration!
! Building matrices at time step: 1 1.000000E-05
! COURANT: dt= 1.0000E-05 > dt_r= 0.0000E+00 and dt_h= 0.0000E+00
! Time step too small, dt= 0.0000E+00
! I thus stop the run !
! Something went wrong, MagIC will stop now
! See below the error message:
Stop run in steptime!
[SHTns 3.4.2] built for MagIC Jul 21 2020, 22:58:48, id: avx512,ishioka
Lmax=42, Mmax*Mres=42, Mres=1, Nlm=946 [1 threads, no Condon-Shortley phase, Robert form, orthonormalized]
> Condon-Shortley phase = 0, normalization = 0
=> using FFTW : Mmax=42, Nphi=128, Nlat=64 (data layout : phi_inc=72, theta_inc=1, phi_embed=128)
=> using Gauss nodes
Gauss quadrature for 3/2.x^2 = 1 (should be 1.0) error = -1.55431e-15
+ polar optimization threshold = 1.0e-10
finding optimal algorithm***************************************************************
+ SHT accuracy = 9.03e-14
=> SHTns is ready.
! SHTns uses theta padding with nlat_padded= 72
***********************************************************
! MPI transpose strategy for 5 fields
! isend/irecv/waitall communicator= 8.008E-04 s
! alltoallv communicator = 1.152E-03 s
! alltoallw communicator = 7.503E-04 s
! -> I pack some fields for the MPI transposes
! -> I choose alltoallw
...
! STOPPING TIME INTEGRATION AT:
stop time = 2.0000000000E-03
stop step = 201
steps gone= 200
!!! regular end of program MagIC !!!
!***********************************!
!---- THANK YOU FOR USING MAGIC ----!
!---- ALWAYS HAPPY TO PLEASE YOU ---!
!-------- call BACK AGAIN ---------!
!- GET YOUR NEXT DYNAMO WITH MAGIC -!
!***********************************!
JW
[SHTns 3.4.2] built for MagIC Jul 21 2020, 22:58:48, id: avx512,ishioka
Lmax=42, Mmax*Mres=42, Mres=1, Nlm=946 [1 threads, no Condon-Shortley phase, Robert form, orthonormalized]
> Condon-Shortley phase = 0, normalization = 0
=> using FFTW : Mmax=42, Nphi=128, Nlat=64 (data layout : phi_inc=72, theta_inc=1, phi_embed=128)
=> using Gauss nodes
Gauss quadrature for 3/2.x^2 = 1 (should be 1.0) error = -1.55431e-15
+ polar optimization threshold = 1.0e-10
finding optimal algorithm*************************************************************ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
+ SHT accuracy = 5.51e-14
=> SHTns is ready.
! SHTns uses theta padding with nlat_padded= 72
ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
**********************************************************ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
ESC[93m Accuracy test failed. Please file a bug report at https://bitbucket.org/nschaeff/shtns/issues ESC[0m
! MPI transpose strategy for 5 fields
! isend/irecv/waitall communicator= 8.255E-04 s
! alltoallv communicator = 1.142E-03 s
! alltoallw communicator = 7.339E-04 s
! -> I pack some fields for the MPI transposes
! -> I choose alltoallw
...
! Something went wrong, MagIC will stop now
! See below the error message:
Stop run in steptime!
! COURANT: dt= 1.0000E-05 > dt_r= 0.0000E+00 and dt_h= 0.0000E+00
! Time step too small, dt= 0.0000E+00
! I thus stop the run !
! Something went wrong, MagIC will stop now
! See below the error message:
Stop run in steptime!
Abort(32) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 32) - process 1
Abort(32) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 32) - process 0
These results are more confusing: the second try ended normally, and both the first and third tries failed. It is more weird that the SHT accuracy
` value in all three tests are very small: 8.18e-14
, 9.03e-14
, and 5.51e-14
. I checked the source file sht_init.c
, the critical value of the “accuracy” is 1e-6
at line 1439. I don’t understand why the program still said “Accuracy test failed”.
In summary, I met the SHT accuracy
problem when I try to do the test in a magic’s sample. Because this problem only happens on the ubuntu workstation. I guess maybe there are some hardware or OS related reasons.
Please give me some suggestion. Thank you and best wish.
PS: please see the sample and log in the attachment file.
The issue of systematic accuracy error (reported initially) has been tracked down to the gnu assembler “as” that is invoked by gcc to compile, and that produced wrong avx512 code in some cases.
See https://sourceware.org/bugzilla/show_bug.cgi?id=23465 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96512
To fix it on your system, you have to upgrade the “binutils” package to version 2.32 or more, and then recompile SHTns.
If you can’t upgrade the “binutils” package, you may try one of the following workarounds:
Makefile
and replace the first occurence of$(shtcc)
with$(shtcc) -fno-tree-pre
--enable-kernel-compiler=icc
Makefile
and replace all occurences ofnative
withskylake
effectively using avx2 instead of avx512.