Warnings found during Relion v3.0-beta-2 build with Intel 2017 compilers; warning #15552: loop was not vectorized with "simd"

[Frank Vago]

Greetings,

I’ve been struggling with a build issue using Intel compilers and was wondering if you could provide any additional information. I’ve been trying to compile the CPU-accelerated version (with AVX512) of Relion v3.0-beta-2 on a community cluster with Intel compilers and libraries. I followed the build instructions (Building with the Intel(R) Compiler 2.4.2) found in betaGuide.pdf using the following CMake command:

CC=mpiicc CXX=mpiicpc cmake -DCUDA=OFF -DCMAKE_INSTALL_PREFIX=$HOME/apps/relion/3.0_cpu-only_intel_2017_avx512 -DALTCPU=ON -DMKLFFT=ON -DTIFF_INCLUDE_DIR=/home/fvago/include -DTIFF_LIBRARY=/home/fvago/lib64/libtiff.so -DCMAKE_C_FLAGS="-O3 -ip -g -debug inline-debug-info -xCORE-AVX512 -restrict " -DCMAKE_CXX_FLAGS="-O3 -ip -g -debug inline-debug-info -xCORE-AVX512 -restrict " ..

After compiling with CMake, the only thing I found were missing paths to:

--MPI_C_COMPILER :

--MPI_CXX_COMPILER :

I’m not certain if this is a major issue, but configuring/compiling did not find any errors. Please see attached “CMakeOutput.txt” for the complete CMake output.

After “make”, I encountered the following warnings:

[ 27%] Building CXX object src/apps/CMakeFiles/relion_lib.dir//acc/cpu/cpu_kernels/helper.cpp.o /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/wavg.h(90): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/wavg.h(90): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/wavg.h(239): warning #15552: loop was not vectorized with "simd" [ 27%] Building CXX object src/apps/CMakeFiles/relion_lib.dir//Healpix_2.15a/cxxutils.cc.o /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/wavg.h(90): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/wavg.h(239): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/wavg.h(90): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(1006): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(853): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(853): warning #15552: loop was not vectorized with "simd" [ 28%] Building CXX object src/apps/CMakeFiles/relion_lib.dir//Healpix_2.15a/healpix_base.cc.o [ 28%] Building CXX object src/apps/CMakeFiles/relion_lib.dir//jaz/distribution_helper.cpp.o /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(445): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(445): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(445): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(445): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(375): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(1162): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(707): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(1329): warning #15552: loop was not vectorized with "simd" /home/fvago/apps/relion/relion-3.0_beta/src/acc/cpu/cpu_kernels/diff2.h(1162): warning #15552: loop was not vectorized with "simd"

The complete GNU Make output is attached (“MakeOutput.txt”). I was wondering, have you encountered similar warning messages during your builds? I tried to find solutions to the #15552 warning messages, but none were very helpful. Do you happen to know what the warning messages mean and is this normal to happen? Despite the warning messages, I installed and tested the build. All features/binaries of Relion 3 were functional without any other issue and can successfully process data from beginning to end. As for 2-D/3-D classification and 3-D refinement, I’m not certain if CPU-acceleration (or vectorisation) is fully implemented or not. In the “run.out” file from a 3-D refinement job, I see the following first 2 lines:

RELION version: 3.0-beta-2

Precision: BASE=double, VECTOR-ACC=single

Thank you for reading this post. I appreciate any information you can provide about the following observations I described earlier. If you need any more information, please don’t hesitate to ask. I appreciate all of the work and added features that has been done with Relion 3!

Cheers,

Frank

System: CentOS v7.5, CMake v3.5.2, GNU Make v3.82, GCC v4.8.5, Intel/impi v2017.1.132

[CharlesCongdon]

Frank: I'm sorry you have been struggling. A lot to unpack, but let me give it a shot.

• The blank MPI_C_COMPILER and MPI_CXX_COMPILER output. I see this too, but ultimately the correct compiler gets chosen. I did recently have a case where CMAKE was finding none of the MPI environment variables. Turned out that this was due to a too recent version of CMAKE (3.12 vs. 3.2). Using the older CMAKE everything worked fine. But I don't think this is what is happening for us - the right compiler is ultimately found.
• To clarify, here is how I set up my 2017.4 environment:

source /opt/intel/impi/2019.0.117/intel64/bin/mpivars.sh intel64
source /opt/intel/compilers_and_libraries_2019.0.117/linux/bin/compilervars.sh intel64
source /opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/bin/mklvars.sh intel64

• The warning #15552: loop was not vectorized with "simd": With 2017.4 and more recent compilers I only see this warning for diff2.h (we know why, but didn't want to destabilize things to fix it with a new algorithm). If I recall correctly the AVX512 support was pretty new in 2017.1, and there were still cases where the compiler struggled, including apparently the cases in wavg.h. If possible, please lobby your cluster's administrator to update to ICC 2018.3, which has much more robust AXV512 support and loads of optimizations.

• Are you running the optimized CPU code? Yes, if you run with --cpu and see "Precision: BASE=double, VECTOR-ACC=single" then, yes you are. What this message is saying that RELION is running mostly double-precision, except in the accelerated CPU code (ACC) for expectation, which runs single-precision like the accelerated GPU code.

In short, despite the alarming messages, it looks like everything is being built and run like it should. Sorry about the confusion. Now if only we could get you a more modern version of the compiler. :-)

Cheers, Charles

[Frank Vago]

Hi Charles,

It's an honor to receive feedback from you. I found the information and answers to my questions you provided to be very helpful.

• That's good to hear the correct compiler is found. I did experience the case with different versions of CMake. From my tests, v3.5.2, v3.6.3, and v3.7.2 work, but v3.13 does not. Through some of the feedback in Github and CCP4EM, this was also the case too for others when using a more recent CMake version. Version 3.9 appears to be the most recent version one can use. Versions from 3.10 and up seems to be the cutoff.

• Thank you for clarifying on setting up the environment. I also set it up as you did, except for mklvars.sh was sourced before compilervars.sh, but I believe the order doesn't matter.

• The cluster I use has Intel 2018.1.163 as well. Not too long ago, I built Relion 3 using this version of the compiler and now only see the 'warning #15552: loop was not vectorized with "simd"' for diff2.h as you did with recent compilers. I didn't think the 2017.1 compiler was too out-dated, but it sounds like that this was the case. I'll ask the cluster to update to a more recent one. That's good to hear there is a fix for diff2.h. Perhaps the fix will be in a near future release of Relion 3.

• Thank you for clarifying this and good to know it's running the accelerated code. I do include --cpu in my command so that it is enabled.

Thank you, again, for your feedback. I truly appreciate it. I'm relatively new to building/compiling programs with the Intel compiler and didn't find any of this information confusing, so no worries on the confusion. It makes sense and I was able to understand it. I do see that having a more recent version of the compiler helps.

Have a good day.

Cheers, Frank

[CharlesCongdon]

Frank: Thanks for the information about the CMake cutoff. That's going to be a recurring pain. Wonder why it's happening...

I'm glad to hear the later compiler reduces you to just the one warning as well.   It's always frustrating when the user and the developer see different things, but not in this case!   As for a definitive fix, I can't promise it will be in a "near future release of RELION 3".

Let us know if we can close this issue, or if you want to keep it open.
Thanks,
Charles

[Frank Vago]

Hi Charles,

You're most welcome for the information on the CMake cutoff. I'm not sure what is the cause yet. Maybe there will be a fix to that one day.

The later compilers do help indeed. Those issues are truly frustrating, but I'm glad that we see the same build issues now. I believe the betaGuide also mentions the support of Intel 2018 Update 3 and later. Since I'm a graduate student, I was able to download the later compilers (e.g 2018.5 and 2019.0), install, and use the student license. I stumbled on this and was happy to see it's available for higher-education students. I was curious on their availability and how one could obtain them. Version 2018.5 (Update 4) works well and the only issue that results is the simd issue in diff2.h(445). Perhaps not in a near future release, I'm sure the fix to this will be applied when ready.

Charles, you have been a great help and the minor issues observed during builds is understood. This issue can be closed. Thank you again.

Cheers,

Frank