How to link an externally compiled library during ET compilation

[[e-mail]]

I have compiled the COCAL code separately from the ET, generating a library for which I have the .a file. I’m trying to link to that library when compiling the ET on Expanse. When I try to link it without copying all the source files in the thorn that uses it into the src directory I get “undefined reference to” errors for anything in a subdirectory. I tried moving all the files in subdirectories into the src directory as a work-around, but then I started getting errors that look like this:

‌

‌

```

‌

/lib/../lib64/crt1.o: In function `_start':

(.text+0x24): undefined reference to `main'

collect2: error: ld returned 1 exit status

make[3]: *** [<builtin>: include_modulefiles_MRNS.f90] Error 1

make[2]: *** [/home/mmutz/Cactus/lib/make/make.thornlib:113: make.checked] Error 2

make[1]: *** [/home/mmutz/Cactus/lib/make/make.configuration:179: /home/mmutz/Cactus/configs/sim17/lib/libthorn_Cocalthorn.a] Error 2

```

‌

What is the proper way to link the library during compilation?

I’ve attached a copy of my config file.

‌

[Roland Haas]

This is rather strange. The optioin list looks ok to me to be honest. Adding -lcocalmrns to LDFLAGS is ok.

The error message about missing main is strange since main is provided by the Cactus flesh. Is it possible that the COCAL library defines its own main and that that one interferes? It would be good to see all the error messages and not just the last couple. Would you mind taking a look at http://einsteintoolkit.org/support.html#general-guidelines-for-questions and provide the files described in there, please?

Since COCAL is not part of the Einstein Toolkit (and, at least until the recent past, had an unusual build system), have you contacted the COCAL author (Antonios Toskaros), who may be able to provide advise as well?

[[e-mail]]

  <div class="preview-container wiki-content"><!-- loaded via ajax --></div>
  <div class="mask"></div>
</div>

</div> </form>

[Roland Haas]

Hello Maria, I had a look at make.log.pdf (a text file is sufficient, there is no need to convert to PDF). Unfortunately, I am still not sure what is going on. The error message is produced by the linker (ld) but the compilation does not yet have reached the final linking stage and seems to be just compiling include_modulesfiles_MRNS.f90 which should only compile and not attempt to link.

There may be “confusing” files present in the src directory (or your environment may contain strange variables) is my best guess. Is the file that produces the error (include_modulesfiles_MRNS.f90) the first Fortran 90 file ever compiled?

Could you please use the command

 VERBOSE=1 simfactory/bin/sim build -j1 2>&1 | tee make.log  

indicated on http://einsteintoolkit.org/support.html#general-guidelines-for-questions to ensure that make outputs the actual commands it executes (this is what VERBOSE=yes does) and compilation happens serially (-j1) so that one is sure what command produces the error?

Also if you could run env >env.log and attach the file env.log this may help.

Ideally also if possible, could you inlcude the source file MRNS/Cocalthorn/src/include_modulefiles_MRNS.f90 (via private mail to [e-mail] if needed)?

[[e-mail]]

I did use VERBOSE=1 and -j1. The whole command was:

VERBOSE=1 simfactory/bin/sim build sim18 --thornlist ./thornlists/cocal_mrns.th --optionlist ./simfactory/mdb/optionlists/expanse-gnu.cfg -j1 2>&1 | tee make.log

‌

[Roland Haas]

Thank you. Indeed, VERBOSE=1, as shown on the webiste, does not work. The correct values is VERBOSE=yes. I have just corrected the website. If you could re-run with VERBOSE=yes and send the produced make.log file, that would be great. Sorry about the incorrect information on the website. Your env.log looks fine to me, so still no idea what is going on.

[Roland Haas]

Thank you. Ok so the line that triggers the error is the last executed one:

gcc -L/cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen2/gcc-10.2.0/numactl-2.0.12-brsvuiqgwwnn2otgnu3pyemvgp6dal2n/lib/ -lm -lgfortran -L/home/mmutz/Cactus/arrangements/MRNS/Cocalthorn/src/Cocal_MRNS/Main_utility/ -lcocalmrns  include_modulefiles_MRNS.f90.o   -o include_modulefiles_MRNS.f90

which indeed is full compile and link (no -c flag). This line should not be there, in particular since the lines above:

1673483420.23084 Compiling /home/mmutz/Cactus/arrangements/MRNS/Cocalthorn/src/include_modulefiles_MRNS.f90 current_wd=`pwd` ; cd /home/mmutz/Cactus/configs/sim18/scratch ; gfortran -m128bit-long-double -fcray-pointer -ffixed-line-length-none -fno-range-check -march=znver2 -fomit-frame-pointer -lgfortran -I/home/mmutz/Cactus/arrangements/MRNS/Cocalthorn/src/Cocal_MRNS/Include_file/ -O2 -mavx2 -mfma -fopenmp -I"/home/mmutz/Cactus/arrangements/MRNS/Cocalthorn/src" -I"/home/mmutz/Cactus/configs/sim18/config-data" -I"/home/mmutz/Cactus/configs/sim18/bindings/include" -I"/home/mmutz/Cactus/src/include" -I"/home/mmutz/Cactus/arrangements" -I"/home/mmutz/Cactus/configs/sim18/bindings/Configuration/Thorns" -I"/home/mmutz/Cactus/configs/sim18/bindings/include/Cocalthorn" -I"/home/mmutz/Cactus/arrangements/MRNS/Cocalthorn/src" -I"/home/mmutz/Cactus/configs/sim18/bindings/include/Cocalthorn" -c -o $current_wd/include_modulefiles_MRNS.f90.o $current_wd/include_modulefiles_MRNS.f90
1673483422.67902 Postprocessing /home/mmutz/Cactus/arrangements/MRNS/Cocalthorn/src/include_modulefiles_MRNS.f90

already compiled that same file to an object file (note the -c option in the long gfortran line).

The only reason I could think of why this may happen is that there are rogue Makefile recipes present.

In your case could you attach any and all files that have names like “Makefile”, “make.code.defn”, “make.code.deps”, “make.configuration.defn”, “make.configuration.deps” in the source code directory /home/mmutz/Cactus/arrangements/MRNS/Cocalthorn/src/ that may help.

My guess would be that there is somthing that COCAL added. Namely the link line uses gcc but Cactus would use g++ to link. Also the error producing file is very wrong in itself already, since it uses include_modulefiles_MRNS.f90 as the output that is writes to it (writes the linked executable there if it succeeded). The normal direction would be for include_modulefiles_MRNS.f90 to be the input and include_modulefiles_MRNS.f90.o (the object file) be the output. That is what is in the “Compiling” line above.

Unfortunately I am mostly guessing here, sorry.

[[e-mail]]

There’s just a make.code.defn file in src, and the whole thing is:

# Main make.code.defn file for thorn Cocalthorn  

# Source files in this directory
SRCS = coc2pri_mrs.f90

# Subdirectories containing source files
SUBDIRS = 

‌

[Roland Haas]

Sorry, I forgot to respond to this. Looking at the make.code.defn file I do not understand why it even would ever try to compile include_modulefiles_MRNS.f90 at all since it is not even listed in the SRCS line. So I am very confused. Is there any chance that I could get to see the whole source code?

[[e-mail]]

Update: got it to compile! The include files being inside the src directory or its subdirectories was causing the problem described. Once I moved the library and everything except the one file listed in the makefile completely outside of Cactus, both this and the undefined reference errors went away.

‌