supermike II fails to compile

[Roland Haas]

The current simfactory files fail to compile the ET on SuperMike II at LSU HPC. One gets:

In file included from /home/rhaas/ET_vanilla/arrangements/CactusUtils/Vectors/src/vectors.cc(1):
/home/rhaas/ET_vanilla/arrangements/CactusUtils/Vectors/src/vectors.h(877): error: invalid type for defaulted constructor
    vmask(vmask &&) = default;
    ^

Keyword:

[Zach Etienne]

This error also occurs on WVU's Spruce Knob resource.

Using icc --version icc (ICC) 17.0.1 20161005 Copyright (C) 1985-2016 Intel Corporation. All rights reserved.

[Zach Etienne]

Similar error using icc --version icc (ICC) 15.0.2 20150121 Copyright (C) 1985-2015 Intel Corporation. All rights reserved.

My guess at this point is that whoever originally committed this was using a really new Intel compiler.

Start list of compile errors:

.../Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(877): error: invalid type for defaulted constructor vmask(vmask &&) = default; ^

In file included from .../Cactus/arrangements/CactusUtils/Vectors/src/test.cc(1): .../Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(879): error: invalid type for defaulted assignment operator vmask &operator=(vmask &&) = default; ^

In file included from .../Cactus/arrangements/CactusUtils/Vectors/src/vectors.cc(1): .../Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(877): error: invalid type for defaulted constructor vmask(vmask &&) = default; ^

In file included from .../Cactus/arrangements/CactusUtils/Vectors/src/vectors.cc(1): .../Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(879): error: invalid type for defaulted assignment operator vmask &operator=(vmask &&) = default; ^

[Zach Etienne]

Found the offending commit!

commit f9cbab7bd3fd534fec2d254665ce6228f6f3ef71 Author: Erik Schnetter [e-mail] Date: Tue Aug 22 09:08:37 2017 -0400

Vectors: Provide vectorized reduction operators

... is the first commit that yields the compiler error.

Therefore commit

commit 7b68551394d3748044362f47ac03d44ad7aede89 Author: Erik Schnetter [e-mail] Date: Mon Aug 21 16:28:53 2017 -0400

Vectors: New configure flags VECTORISE_ALIGN_FOR_CACHE and VECTORISE_ALIGN_INTERIOR

... works just fine.

[Steven R. Brandt]

Does that mean we have a fix?

[Roland Haas]

No, I would not immediately say that. The code added in that commit uses (as far as I know) only C++11 and we list C++11 (at least as subset of it) as a requirement for the ET. The compiler used on supermike II defaults to use the Intel 13 compiler which is very old by now and while we request gnu++11 that particular compiler does not yet handle defaulted move constructors correctly (see https://software.intel.com/en-us/articles/c0x-features-supported-by-intel-c-compiler in particular http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3053.html looks like it might be the only applicable but I am by no means sure given the complexity of the language in the proposals).

I think I tried to use newer compilers for a bit on supermike II and had no luck either though. I mostly dropped my efforts when realizing that I had confused supermike II and supermic (the former being LSU - HPC only while the latter is currently accessible through XSEDE).

We need / want to keep some of the commit f9cbab7bd3fd534fec2d254665ce6228f6f3ef71 namely the part that introduces the vectorized reduction operators. If that part can be nicely separated from the part that makes old compilers quit and if the part that makes compilers quit is not deemed to be performance relevant then we could / should / may want to remove only those bits.

[Roland Haas]

It also seems to work just fine with the intel 14 / 15 compiler on other clusters. So it would be good to find out why it fails on Spruce Knob and supermike II but works eg on queenbee (Intel 14) or cori (intel 15).

[Zach Etienne]

I have confirmed the issue appears with Intel 17.0.1 (20161005) on Spruce as well, which runs RHEL6 (Linux kernel 2.6.32-696.1.1.el6.x86_64). Note that CXXFLAGS are set to use C++11:

CXXFLAGS = -Ofast -qopenmp -xHost -align -std=c++11

Given what Roland has found, my suspicion at the moment is that this is not a compiler issue.

[Zach Etienne]

I found a fix! It turns out that the struct that is triggering the compilation errors on Intel 17.0.1 (and 15.0.2) on RHEL6 is never used except by functions that are never called:

vstore_masked() and vstorea_masked()

I commented out the vmask and vstore()/vstorea() declarations and the Vectors thorn (as well as the rest of the Toolkit) compiled just fine.

Doesn't this mean that the vmask struct, vstore_masked(), and vstorea_masked() are dead code and should be removed?

[Roland Haas]

The functions vstore_masked() and vstorea_masked() are used in HydroOpenMPToy. Right now the mike.ini file in simfactory uses shelob-mvapich.cfg to build which uses parallel_studio_xe_2018.0 to compile which is Intel 18.0 and which seems to work fine (on shelob). My suspicion (with anything Intel compiler and C++ related) is that there is an incompatible version of g++ (and its C++ library) being used with Intel 17 (see https://software.intel.com/en-us/articles/intel-c-compiler-170-for-linux-release-notes-for-intel-parallel-studio-xe-2017). Intel 17 supports gcc versions 4.3 - 6 .

[Roland Haas]

@Zach Etienne we can currently compile on supermike II (the mike.ini file in simfactory, if the Spruce Knob cluster still around and still affected by the issue? If not true on both grounds I would close this ticket with wontfix.

[Zach Etienne]

@Roland Haas I still have to comment out all vmask stuff in the Vectors thorn to get ETK compiled on Spruce Knob. Given that everything works fine if it’s commented out, it seems that this is dead code that should be removed?

[Roland Haas]

Ok, it having been retired would have been too easy.

Unfortunately it is not dead code and is used (though not in the ET right now), please see https://bitbucket.org/einsteintoolkit/tickets/issues/2115/supermike-ii-fails-to-compile#comment-52788391 .

Given that this compiles on quite a number of other clusters (with Intel compilers) I wonder what makes it fail on Spruce Knob and the previous Mike setup.

I did a quick test on BlueWaters with the intel 16 compiler there. Since the failing code is:

class foo {
  foo(foo&&) = default;
};

I created such a file and tried compiling it with icpc. All is well if I use the default settings:

icpc -c -std=c++11 foo.cc

However if instead I force icpc to use an old gcc version (4.3 in my case) then I get:

icpc -c -std=c++11 -gcc-name=/usr/bin/gcc-4.3 foo.cc
foo.cc(2): error: invalid type for defaulted constructor
    foo(foo&&) = default;
    ^

compilation aborted for foo.cc (code 2)

Namely exactly the error that you observed. So my suspicion is that a for c++11 a minimum version of gcc is required. I can compile the fragment above with:

icpc -c -std=c++11 -gcc-name=/usr/bin/gcc-4.7 foo.cc

So I would like to ask if you could try and see if Spruce Knob has a gcc version of at least 4.7 available (it really should) and try adding -gcc-name=PATH-TO-NEWER-GCC to your CXXFLAGS.

If this fixes the issue (and in particular if it works with the gcc in /usr/bin/gcc), and you have enough clout, you may want to suggest to the cluster admins that they should make icpc use a newer gcc by default.

[Zach Etienne]

Thank you for getting to the bottom of this! I didn’t realize that icc depended on gcc in this way. Indeed the gcc version on Spruce Knob is 4.4.7 (i.e., very old). A newer version of gcc is installed (8.2.0; accessible via a `module load`); I’ll try module load'ing icc & gcc 8.2.0 and see if it compiles now.

[Zach Etienne]

Peter says that the latest ETK compiles fine on Supermike ii. As for Spruce Knob, modern gcc is incompatible with the latest ICC. Given that I have a fine workaround, and Spruce Knob will be decommissioned within a couple years, let’s just close this ticket as WONTFIX.

[Roland Haas]

The issue turned out to be the use of too old g++ in combination with icpc.