1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

Possible issue between ExternalLibraries/FFTW3 thorn and the Cactus/lib/sbin/strip-incdirs.sh script

Issue #2428 resolved
Federico Cipolletta created an issue

While I was attempting the compilation of one GRMHD code with the ETK which requires FFTW 3.3.7 libraries (those libraries seem not to be straightforwardly backward compatible), I was getting one issue because the FFTW3 thorn was not able to catch my local installed libraries. In particular, on my Ubuntu 16.04 VirtualBox, I have the FFTW 3.3.4 libraries already installed on /usr/ and I also installed the 3.3.7 version (which is the one required by the code) on /usr/local/. If on the .cfg file I don’t indicate FFTW3_DIR, the 3.3.4 version is found automatically (look at the file Cactus/arrangements/ExternalLibraries/FFTW3/src/detect.sh line 23), but if I set FFTW3_DIR = /usr/local (so pointing at the 3.3.7 required version), then the include and lib locations are stripped off by the script Cactus/lib/sbin/strip-incdirs.sh (called at lines 143 and 144 of Cactus/arrangements/ExternalLibraries/FFTW3/src/detect.sh). In particular, in the Cactus/configs/<config>/bindings/Configuration/Capabilities/make.FFTW3.defn the following is reported:

FFTW3_DIR = /usr/local
FFTW3_INC_DIRS =
FFTW3_LIB_DIRS =

In this way, the required FFTW3 library version could not be found. A workaround has been to install the required version of FFTW3 in a different location (e.g. /home/) and to point to this location (which is not stripped off by the Cactus/lib/sbin/strip-incdirs.sh script) in the .cfg file and everything works like a charm. The Cactus/configs/<config>/bindings/Configuration/Capabilities/make.FFTW3.defn, thus, reports:

FFTW3_DIR = /home/federico
FFTW3_INC_DIRS = /home/federico/include
FFTW3_LIB_DIRS = /home/federico/lib

which is correct. Maybe, it could be useful to suggest a location for the local installed version of FFTW3 libraries as a comment in the ExternalLibrary/FFTW3/README file.

Regards,

Federico Cipolletta

Comments (20)

  1. Roland Haas

    Are you doing this on a Mac by any chance? In that case see: https://bitbucket.org/einsteintoolkit/tickets/issues/2306/set-up-cpath-and-library_path-in for reasons why “/usr/local” is always stripped and for workarounds for compilers (such as some of the MacPorts and HomeBrew) that do not search in places we expect them to search by setting LIBRARY_PATH and CPATH. There’s also a branch https://bitbucket.org/cactuscode/cactus/branch/rhaas/strip-dirs of the flesh where I try to provide a better solution, but frankly it does not really work either.

  3. Federico Cipolletta reporter

    No, my laptop has Windows 10 on it and I am working on one Ubuntu 16.04 Virtual Box.

  4. Roland Haas

    Hmm. In that case /usr/local should be searched for as part of the default set of search locations (which is part of why we remove them from LIB_DIRS). It should also be searched before /usr so should pick that library in case there is an FFTW3 in both /usr and /usr/local.

    Can you provide the output of gcc -xc -E -v - please (see https://stackoverflow.com/questions/4980819/what-are-the-gcc-default-include-directories)? On my Debian Bullseye box this returns:

    $ gcc -xc -E -v -
Using built-in specs.
COLLECT_GCC=gcc
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 9.3.0-13' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-F9gimE/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-mutex
Thread model: posix
gcc version 9.3.0 (Debian 9.3.0-13)
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/9/cc1 -E -quiet -v -imultiarch x86_64-linux-gnu - -mtune=generic -march=x86-64 -fasynchronous-unwind-tables
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/9/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/home/rhaas/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/9/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

    showing for example that /usr/local/include is searched for include files.

    Note that the reason we strip /usr/local and a couple of other directories out of XXX_LIB_DIRS is basically a variant of the issue you are seeing. Assume the following situation:

    • someone has FFTW 3.8 installed in /usr/
    • they set HDF5_DIR to /usr
    • the install FFTW 3.3 into their $HOME

    Now what will happen (without stripping) is that Cactus will end up generating -L options like this:

    -L /usr/lib  # due to HDF5_DIR=/usr
-L $HOME/lib # due to FFTW3_DIR=$HOME

    however since there is a libfftw3.a in /usr/lib and the linker will use the library in the first location found, this will use fftw3 from /usr and not from $HOME.

    The current setup with stripping basically means that “non-standard locations always win” while the behaviour without stripping means that things depend in subtle ways on the order in which -L options are generated.

    There is not really good way to fix this with -L as long as the there are multiple versions of a library in -L locations and / or system locations.

    The only way to fix this for good is to instruct the linker to link against a particular file explicitly instead of against a library name and let the linker find the library. Eg this one:

    gcc main.c /usr/local/lib/libfftw3.a

    will always link against /usr/local/lib/libfftw3.a no matter what. Note however that this requires that (on top of my mind):

    1. Cactus has to find the exact location of the library ie walk the XXX_LIB_DIRS paths and possible all “system paths” (which are compiler, OS and compiler version dependent)
    2. it needs to know whether to link against a .a or .so or .dylib library file

    Doing this may be the nicest solution but so far no one has spent the time working on it.

  5. Roland Haas

    Note that you say “While I was attempting the compilation of one GRMHD code with the ETK which requires FFTW 3.3.7 libraries (those libraries seem not to be straightforwardly backward compatible),” and have FFTW 3.3.4 already installed. So it seems less like a case of FFTW 3.3.7 being non-backwards compatible (ie being callable by code that only knows about older versions of FFTW) and more like your GRMHD code requires a newer version of FFTW (which I would have expected apt-get install libfft3-dev would install but maybe Ubuntu 16.04 is too old by now).

    Since the issue seems to be code requiring a newer FFTW3 and not too-new FFTW3 and since version 3.3.3 (included in the ET as a tarball) is from 11/25/12 we may well want to update to 3.3.8 from 5/28/18 which we already use eg on comet so know that ET compiles are runs with it.

    If that would be a possible solution for you, please create an enhancement ticket asking for the included FFTW3 tarball to be updated to 3.3.8 ideally with at statement that you tried this and it works.

    Updating requires forking https://github.com/einsteintoolkit/ExternalLibraries-FFTW3 editing https://github.com/EinsteinToolkit/ExternalLibraries-FFTW3/blob/master/src/build.sh to update the version number that shows up there and (obviously) updating the actual tarball file.

    You can instruct Cactus to build it by adding FFTW3_DIR=BUILD to your option list (and most likely building from scratch).

    ‌

  6. Federico Cipolletta reporter

    With the command that you suggesting me I am obtaining the following:

    Using built-in specs.
    COLLECT_GCC=gcc
    Target: x86_64-linux-gnu
    Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.12' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
    Thread model: posix
    gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)
    COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
    /usr/lib/gcc/x86_64-linux-gnu/5/cc1 -E -quiet -v -imultiarch x86_64-linux-gnu - -mtune=generic -march=x86-64 -fstack-protector-strong -Wformat -Wformat-security
    ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
    ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/include"
    #include "..." search starts here:
    #include <...> search starts here:
    /usr/lib/gcc/x86_64-linux-gnu/5/include
    /usr/local/include
    /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed
    /usr/include/x86_64-linux-gnu
    /usr/include
    End of search list.

    indeed, it seems that usr/local/ include is searched also in my case for include files.

    I see your argumentation on trying to solve this issue and how time-consuming it would be (in contrast of just searching in another place). Indeed, I was proposing one simple comment on the README file, just to make people be aware of this behavior.

  7. Federico Cipolletta reporter

    Apologize me for my error in using the terminology “backward compatible”. The problem is that the GRMHD code I was compiling is working correctly with FFTW 3.3.7 (with which it was developed), but it does not seem to work with 3.3.4.

    I did not try to build the library from scratch because I already had it installed.

    I already reached a solution, by installing FFTW 3.3.7 on another location (/home/federico instead of /usr/local) but it took some time for me to figure out what was going on. My proposal for adding one comment was meant for un-experienced people (like me ^_^).

  8. Roland Haas

    Not sure where the best place to add such a comment is.

    The README file is usually more to describe the thorn. Technically this should go into the thorn documentation (documentation.tex) and also into the Cactus UsersGuide http://einsteintoolkit.org/usersguide/UsersGuide.html#x1-21000B2.2 which describes ExternalLibraries though how likely this is to be found there by unexperienced users (really anyone who has not run into this rather tricky issue before) is not immediately obvious to me.

  9. Federico Cipolletta reporter

    I see. Probably the thorn documentation would be the best place since I only found this issue related to this particular thorn and I am not sure if it can occur also with other External Libraries. Indeed, inserting such a comment in the Cactus UserGuide may be too dispersive, as the Cactus UserGuide is already very extended and is full of precious details. On the other hand, if we were sure that this issue may be found also with other External Libraries, then the Cactus UserGuide may be the best. Let me know if I can help to edit the thorn documentation or if it would worth making further trials with other External Libraries.

  10. Roland Haas

    This will happen with every ExternalLibrary. It does not hurt though to repeat the information in each thorn’s documentation.tex (and the UsersGuide). The easier it is for this to actually be found (rather than pointed out after the fact) the better.

  11. Federico Cipolletta reporter

    Ok, then. Shall I proceed to add some notes to the ExternalLibraries thorns' documentations and to the Cactus User Guide and create a pull request?

  15. Federico Cipolletta reporter

    I would like to add one NOTE like the following in each of the documentation (e.g. at section B2.2 of Cactus User Guide):

    ‌

    “NOTE: each thorn contained in Cactus/arrangements/ExternalLibraries will automatically adopt the library version contained in the Cactus/arrangements/<library>/dist folder. If another location is specified via the THORN_DIR variable in the machine.cfg file at compilation, then the Cactus/lib/sbin/strip-incdirs.sh script will automatically strip away (for safety reasons) the locations:

    1. /include

    2. /usr/include

    3. /usr/local/include

    locations. Therefore, if there is any need for using one already installed version of one external library, the aforementioned location should be avoided (e.g. indicating /home as the THORN_DIR will work with no problems if the required library is installed there).”

    ‌

    Do you agree with that?

    Can I simply edit the checkout file from github, commit it and push (as I have been added to the ET developers)?

  16. Roland Haas

    Hello @Federico Cipolletta ,

    last question first: yes you are welcome to directly push documentation, no review required.

    The note may require some clarification though:

    • the tarball in “dist” is only used if either THORN_DIR is set to BUILD or is left empty and no precompiled copy of the library (any version, for the most part anyway) is found. Ie the FFTW thorn, while it includes 3.3.3 tarball, will accept any installed version (both newer and older)
    • the directories are stripped from THORN_INC_DIRS which default to THORN_DIR/include

    • the same stripping happens to THORN_LIB_DIRS in lib/sbin/strip-libdirs.sh which a larger list of directories:

      1. /lib
      2. /usr/lib
      3. /usr/local/lib
      4. /lib64
      5. /usr/lib64
      6. /usr/local/lib64

    You may want to add that “since the compiler is assumed to search them automatically at the end of all explicitly passed locations. This avoids picking up other libraries installed in these system locations.”.

  17. Roland Haas

    Note that the general rule here is that we’d rather have incomplete documentation than none, so please feel free to commit text in the current state (or any other state) as well.

  18. Federico Cipolletta reporter

    Ok, I just pushed the doc for the ExternalLibraries/FFTW3 and made a pull request for the Cactus UserGuide with the note that I was talking about. I think that the documentation added is very general and could be applied to all the other external libraries.

  19. Roland Haas

    I copied the text into the documentation files for:

    ‌

  21. Log in to comment
Assignee
–
Type
bug
Priority
minor
Status
resolved
Component
EinsteinToolkit thorn
Milestone
–
Version
–
Votes
0
Watchers
1
Jira Software: the preferred issue tracker for Bitbucket. Join the team!