Linking static cuda/gpu build with cmake on cori using UPCXX::upcxx library interface

[Rob Egan]

In MHMXX we are encountering the following problem with static builds on cori, but the shared build works fine. The build also works okay with cmake 3.10.2 (that does not well support library interfaces so our build does not use the UPCXX::upcxx library interface), but not the default module 3.14.2 that does support it well and our build does use it.

‌

I’m not sure if this is a problem with CMake’s cuda support or UPC++'s CMake support…

‌

[ 56%] Linking CUDA device code CMakeFiles/mhmxx.dir/cmake_device_link.o
cd /global/homes/r/regan/workspace-shared/mhmxx/build-gnu-gpu/src && /global/common/sw/cray/cnl7/haswell/cmake/3.14.4/gcc/8.2.0/2hef55n/bin/cmake -E cmake_link_script CMakeFiles/mhmxx.dir/dlink.txt --verbose=1
/usr/common/software/cuda/10.2.89/bin/nvcc  -g -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink CMakeFiles/mhmxx.dir/main.cpp.o CMakeFiles/mhmxx.dir/merge_reads.cpp.o CMakeFiles/mhmxx.dir/kcount.cpp.o CMakeFiles/mhmxx.dir/dbjg_traversal.cpp.o CMakeFiles/mhmxx.dir/hash_funcs.c.o CMakeFiles/mhmxx.dir/klign.cpp.o CMakeFiles/mhmxx.dir/cgraph.cpp.o CMakeFiles/mhmxx.dir/build_ctg_graph.cpp.o CMakeFiles/mhmxx.dir/walk_ctg_graph.cpp.o CMakeFiles/mhmxx.dir/spanner.cpp.o CMakeFiles/mhmxx.dir/splinter.cpp.o CMakeFiles/mhmxx.dir/localassm.cpp.o CMakeFiles/mhmxx.dir/histogrammer.cpp.o CMakeFiles/mhmxx.dir/aln_depths.cpp.o -o CMakeFiles/mhmxx.dir/cmake_device_link.o  -L/usr/common/software/cuda/10.2.89/targets/x86_64-linux/lib/stubs  -L/usr/common/software/cuda/10.2.89/targets/x86_64-linux/lib ssw/libSSW_LIBRARY.a  ../libMHMXX_VERSION_LIB.a adept-sw/libADEPT_SW_LIBRARY_static.a ../upcxx-utils/src/libUPCXX_UTILS_LIBRARY.a -lpthread -L/usr/common/ftg/upcxx/2020.3.2/craype-none/gpu/gnu/gcc-8.3.0/upcxx.debug.gasnet_seq.ibv/lib -lupcxx -L/usr/common/ftg/upcxx/2020.3.2/craype-none/gpu/gnu/gcc-8.3.0/gasnet.debug/lib -lgasnet-ibv-seq -L/global/homes/h/hargrove/lib -libverbs   -L/opt/esslurm/lib64 -lpmi2  -lpthread  -lrt -L/opt/gcc/8.3.0/snos/lib/gcc/x86_64-suse-linux/8.3.0 -lgcc -lm  -Wl,--start-group -L/usr/common/software/cuda/10.2.89/bin/../targets/x86_64-linux/lib/stubs -L/usr/common/software/cuda/10.2.89/bin/../targets/x86_64-linux/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -lcuda -lcudadevrt -lcudart_static -lrt -ldl
nvcc fatal   : Unknown option '-Wl,--start-group'
make[2]: *** [src/CMakeFiles/mhmxx.dir/build.make:286: src/CMakeFiles/mhmxx.dir/cmake_device_link.o] Error 1
make[2]: Leaving directory '/global/u2/r/regan/workspace-shared/mhmxx/build-gnu-gpu'
make[1]: *** [CMakeFiles/Makefile2:870: src/CMakeFiles/mhmxx.dir/all] Error 2
make[1]: Leaving directory '/global/u2/r/regan/workspace-shared/mhmxx/build-gnu-gpu'

I believe the --start-group / --end-group commands are being introduced by the cuda language support in CMake.

‌

I tracked down the erroneous dependency that CMake picks up from the UPCXX::upcxx library interface which then makes its way into the above cuda device link dependency somehow.

‌

UPCXX::upcxx library interface: UPCXX::upcxx;-L/usr/common/ftg/upcxx/2020.3.2/craype-none/gpu/gnu/gcc-8.3.0/upcxx.debug.gasnet_seq.ibv/lib -lupcxx -L/usr/common/ftg/upcxx/2020.3.2/craype-none/gpu/gnu/gcc-8.3.0/gasnet.debug/lib -lgasnet-ibv-seq -L/global/homes/h/hargrove/lib -libverbs   -L/opt/esslurm/lib64 -lpmi2  -lpthread  -lrt -L/opt/gcc/8.3.0/snos/lib/gcc/x86_64-suse-linux/8.3.0 -lgcc -lm  -Wl,--start-group -L/usr/common/software/cuda/10.2.89/bin/../targets/x86_64-linux/lib/stubs -L/usr/common/software/cuda/10.2.89/bin/../targets/x86_64-linux/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -lcuda

‌

‌

Normally having extra libs on the line is fine, but the issue is that during linking “-Wl,--start-group” and “-Wl,--end-group” is breaking. replacing the argument text with “-Xlinker --start-group” and “-Xlinker --end-group” works okay. So does removing everything between these arguments.

When I remove the offending arguments from the above nvcc command, it links fine and when restarting the build, it completes successfully.

In fact the only part of the cmake_device_link.o build command above which is required is the actual library file which is built for the gpu:

/usr/common/software/cuda/10.2.89/bin/nvcc  -g -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink  -o CMakeFiles/mhmxx.dir/cmake_device_link.o  adept-sw/libADEPT_SW_LIBRARY_static.a

‌

… so this may be an issue with how CMake is applying linking dependencies to the cmake_device_link.o target that do not have a dependency when a library interface is involved in the build.

‌

This is the CMakeLists.txt for the only subdirectory that requires nvcc.

if(NOT ENABLE_CUDA)
    message(FATAL_ERROR "Trying to build ADEPT-SW but CUDA is not enabled")
endif()

find_package(OpenMP REQUIRED)

set(CMAKE_POSITION_INDEPENDENT_CODE ON)
set(CMAKE_CUDA_SEPARABLE_COMPILATION ON)
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -Xcompiler=-Wall,${OpenMP_CXX_FLAGS} -gencode arch=compute_70,code=sm_70")
message(STATUS "CMAKE_CUDA_FLAGS=${CMAKE_CUDA_FLAGS}")

option(ADEPT_SW_SHARED "Adept-SW shared library" ON)
option(ADEPT_SW_STATIC "Adept-SW static library" ON)

if (ADEPT_SW_SHARED)
  add_library(ADEPT_SW_LIBRARY_shared SHARED driver.cpp kernel.cpp gpu_alns.cpp utils_gpu.cpp)
  target_link_libraries(ADEPT_SW_LIBRARY_shared INTERFACE OpenMP::OpenMP_CXX) 
  install(TARGETS ADEPT_SW_LIBRARY_shared LIBRARY DESTINATION lib)
  if (NOT ADEPT_SW_STATIC)
    add_library(ADEPT_SW_LIBRARY ALIAS ADEPT_SW_LIBRARY_shared)
  endif()
endif()
if (ADEPT_SW_STATIC)
  add_library(ADEPT_SW_LIBRARY_static STATIC driver.cpp kernel.cpp gpu_alns.cpp utils_gpu.cpp)
  target_link_libraries(ADEPT_SW_LIBRARY_static INTERFACE OpenMP::OpenMP_CXX)
  install(TARGETS ADEPT_SW_LIBRARY_static ARCHIVE DESTINATION lib)
  add_library(ADEPT_SW_LIBRARY ALIAS ADEPT_SW_LIBRARY_static)
endif()


set_source_files_properties(driver.cpp kernel.cpp gpu_alns.cpp utils_gpu.cpp PROPERTIES LANGUAGE CUDA LINKER_LANGUAGE CUDA)

And I should note that the cmake_device_link.o file for this library builds just fine, because it does not have any dependencies.

‌

And this is from the parent dir that has all the dependencies listed and I cannot figure out why it generates dependencies on the cmake_device_link.o file with all the other build targets:

if(ENABLE_CUDA)
    include_directories("adept-sw")
    add_subdirectory(adept-sw)
endif()

add_executable(mhmxx
    main.cpp merge_reads.cpp kcount.cpp dbjg_traversal.cpp hash_funcs.c klign.cpp cgraph.cpp build_ctg_graph.cpp
    walk_ctg_graph.cpp spanner.cpp splinter.cpp localassm.cpp histogrammer.cpp aln_depths.cpp)


if(ENABLE_CUDA)
    set_property(TARGET mhmxx PROPERTY CUDA_SEPARABLE_COMPILATION ON)
    target_link_libraries(mhmxx
        Threads::Threads
        SSW_LIBRARY
        ${ZLIB_LIBRARIES}
        ${UPCXX_LIBRARIES}
        ${UPCXX_UTILS_LIBRARIES}
        MHMXX_VERSION_LIB
        ADEPT_SW_LIBRARY)

else()
    target_link_libraries(mhmxx
        Threads::Threads
        SSW_LIBRARY
        ${ZLIB_LIBRARIES}
        ${UPCXX_LIBRARIES}
        ${UPCXX_UTILS_LIBRARIES}
        MHMXX_VERSION_LIB)
endif()

This is the state of modules loaded when building:

Currently Loaded Modulefiles:
  1) modules/3.2.11.4                                 11) ugni/6.0.14.0-7.0.1.1_7.33__ge78e5b0.ari         21) PrgEnv-gnu/6.0.5
  2) altd/2.0                                         12) pmi/5.0.14                                       22) craype-hugepages2M
  3) darshan/3.1.7                                    13) dmapp/7.1.1-7.0.1.1_4.48__g38cf134.ari           23) craype-network-aries
  4) cray-mpich/7.7.10                                14) gni-headers/5.0.12.0-7.0.1.1_6.28__g3b1768f.ari  24) craype-x86-skylake
  5) git/2.21.0                                       15) xpmem/2.2.20-7.0.1.1_4.10__g0475745.ari          25) jdk/1.8.0_202
  6) esslurm                                          16) job/2.2.4-7.0.1.1_3.36__g36b56f4.ari             26) cudnn/7.6.5
  7) gcc/8.3.0                                        17) dvs/2.12_2.2.156-7.0.1.1_8.9__g5aab709e          27) cuda/10.2.89
  8) craype/2.6.2                                     18) alps/6.6.58-7.0.1.1_6.4__g437d88db.ari           28) cmake/3.14.4
  9) cray-libsci/19.06.1                              19) rca/2.2.20-7.0.1.1_4.46__g8e3fb5b.ari            29) upcxx-gpu/2020.3.2
 10) udreg/2.3.2-7.0.1.1_3.31__g8175d3d.ari           20) atp/2.1.3

‌

‌

[Dan Bonachea]

@Rob Egan - All of the flags provided by UPC++ (including the linker flags) are formatted for consumption by the configured C++ compiler (in this case g++) and not the nvcc wrapper which has a different and crippled syntax for many options (some of which you are not encountering here). The UPC++ link flags should include the needed cruft to link in the CUDA libraries, extracted by UPC++ configure from nvcc.

I suggest trying to link with upcxx or g++ instead of nvcc, although I'm not sure how to force CMake to do that..

[Rob Egan]

That’s the problem yes. But the Cuda language support in cmake is preventing me from doing that. As far as I can tell, the cmake_device_link.o target which gets built for any executable or library target that links with a cuda object is something embedded well within the set of routines that get generated by the enable_language(CUDA) directive.

‌

Unfortunately the dependency calculations include every target_link_library and object associated with a target, so all of the UPC++ object code and library flags/objects are swept into the nvcc command that generates the device link object. It is normally okay because the linkers (nvcc or c++) systematically ignore anything it does not understand. Unfortunately nvcc does not also ignore flags it does not understand.

‌

I eventually found a hack of a work-around to build the cmake_device_link.o for the static library and also prohibit its generation for the executable, but then manually including the static library’s device link object.

When building the static library of GPU code:

add_library(ADEPT_SW_LIBRARY_obj OBJECT driver.cu kernel.cu gpu_alns.cu utils_gpu.cu)

set_property(TARGET ADEPT_SW_LIBRARY_static PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON)
add_library(ADEPT_SW_LIBRARY_static STATIC $<TARGET_OBJECTS:ADEPT_SW_LIBRARY_obj>)
set(ADEPT_SW_BROKEN_DEVICE_LINK_HACK ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/ADEPT_SW_LIBRARY_static.dir/cmake_device_link.o PARENT_SCOPE)

Then when creating the executable in the parent directory

#
executable(mhmxx ....)
set_property(TARGET mhmxx PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS OFF)

target_link_libraries(mhmxx
...
${ADEPT_SW_BROKEN_DEVICE_LINK_HACK}
ADEPT_SW_LIBRARY_static)

‌

‌

So for this ticket regarding upc++ support of a GPU build, is it possible to qualify the language to just CXX (and not CUDA) for these required linker flags? i.e. can the link objects / flags be qualified by the generator variable $<LINK_LANGUAGE:languages>

[Jonathan Madsen]

I think this is happening because Rob is using ${UPCXX_LIBRARIES} instead of UPCXX::upcxx because the interface library uses a generator expression but the raw variable does not:

set_target_properties( UPCXX::upcxx PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES "${UPCXX_INCLUDE_DIRS}"
    INTERFACE_LINK_LIBRARIES      "${UPCXX_LIBRARIES}"
    INTERFACE_COMPILE_DEFINITIONS "${UPCXX_DEFINITIONS}"
    # generators ensure that potentially proprietary compiler options 
    # are only added when invoking the CXX compiler frontend:
    INTERFACE_LINK_OPTIONS        "$<$<COMPILE_LANGUAGE:CXX>:${UPCXX_LINK_OPTIONS}>"
    INTERFACE_COMPILE_OPTIONS     "$<$<COMPILE_LANGUAGE:CXX>:${UPCXX_OPTIONS}>"
    )

If this is causing unresolved symbols, try this:

string(REPLACE "-Wl,--start-group" "-Xlinker=-Wl,--start-group" UPCXX_CUDA_LINK_OPTIONS "${UPCXX_LINK_OPTIONS}")
string(REPLACE "-Wl,--end-group" "-Xlinker=-Wl,--end-group" UPCXX_CUDA_LINK_OPTIONS "${UPCXX_CUDA_LINK_OPTIONS}")
set_target_properties( UPCXX::upcxx PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES "${UPCXX_INCLUDE_DIRS}"
    INTERFACE_LINK_LIBRARIES      "${UPCXX_LIBRARIES}"
    INTERFACE_COMPILE_DEFINITIONS "${UPCXX_DEFINITIONS}"
    # generators ensure that potentially proprietary compiler options 
    # are only added when invoking the CXX compiler frontend:
    INTERFACE_LINK_OPTIONS        "$<$<COMPILE_LANGUAGE:CXX>:${UPCXX_LINK_OPTIONS}>" "$<$<COMPILE_LANGUAGE:CUDA>:${UPCXX_CUDA_LINK_OPTIONS}>"
    INTERFACE_COMPILE_OPTIONS     "$<$<COMPILE_LANGUAGE:CXX>:${UPCXX_OPTIONS}>"
    )

‌

[Rob Egan]

So swapping the variable with the contents does not fix the build problem, and I verified that the variable ${UPCXX_LIBRARIES} == “UPCXX::upcxx”

+++ b/src/CMakeLists.txt
@@ -79,15 +79,16 @@ add_executable(mhmxx
 if(ENABLE_CUDA)
     include_directories("adept-sw")
     add_subdirectory(adept-sw)
-    set_property(TARGET mhmxx PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS OFF) # necessary to turn off cmake_device_link for static build
+    message(STATUS "UPCXX_LIBRARIES=${UPCXX_LIBRARIES}")
+    set_property(TARGET mhmxx PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) # necessary to turn off cmake_device_link for static build
     target_link_libraries(mhmxx
         Threads::Threads
         SSW_LIBRARY
         ${ZLIB_LIBRARIES}
-        ${UPCXX_LIBRARIES}
+        UPCXX::upcxx
         ${UPCXX_UTILS_LIBRARIES}
         MHMXX_VERSION_LIB
-        ${ADEPT_SW_BROKEN_DEVICE_LINK_HACK} # necessary for cmake_device_link static build
+#        ${ADEPT_SW_BROKEN_DEVICE_LINK_HACK} # necessary for cmake_device_link static build
         ADEPT_SW_LIBRARY_static)

‌

Perhaps the UPCXX::upcxx properties needs this change to associate the linker options with the linker language, not the compile language?

INTERFACE_LINK_OPTIONS        "$<$<LINKER_LANGUAGE:CXX>:${UPCXX_LINK_OPTIONS}>" "$<$<LINKER_LANGUAGE:CUDA>:${UPCXX_CUDA_LINK_OPTIONS}>"

‌

‌

[Dan Bonachea]

So the root problem here is that nvcc has a different command-line syntax than the g++/clang++/icpc backend C++ compiler that it wraps. In addition, nvcc silently adds its own CUDA-related linker options that are not automatically added by the backend compiler.

This means that in order to provide correct link-stage flags, we need to know which linker wrapper is being used. Currently upcxx and upcxx-meta (and the CMake package wrapping them) assume the backend C++ compiler is being invoked to wrap the system linker (ie that's what upcxx invokes for linking and upcxx-meta CXX is that linker). So we provide flags in the syntax the C++ compiler expects and explicitly add the CUDA-related linker options that are needed. However passing these same flags to nvcc for linkage breaks because it has a different (and IMHO horribly broken/restricted) input syntax for flags.

Perhaps the UPCXX::upcxx properties needs this change to associate the linker options with the linker language, not the compile language?

INTERFACE_LINK_OPTIONS        "$<$<LINKER_LANGUAGE:CXX>:${UPCXX_LINK_OPTIONS}>" "$<$<LINKER_LANGUAGE:CUDA>:${UPCXX_CUDA_LINK_OPTIONS}>"

I believe that an approach like this could work (at least in recent CMake versions). However our scripts that generate the linker flags don't currently generate both versions - IOW I currently have the value of ${UPCXX_LINK_OPTIONS} available, but no automated way to get a valid value for ${UPCXX_CUDA_LINK_OPTIONS} needed above (at least not in the general case). In order to support this approach portably/robustly, we'd need to add logic to our configure script to build the second set of linker flags and export it via upcxx-meta for consumption in the CMake package.

TL;DR: we can solve this, but it requires some hacking in our build infrastructure before we can deploy such a solution in a robust/general way. If your current workaround is sufficient, I'd suggest sticking with that for now and hopefully we can deploy something better by next release.

[Rob Egan]

Thanks Dan.

I want to emphasize, after reading what little documentation there is about this, that all the upc++ linker options are superfluous for the nvcc linking command to build the necessary device link object: cmake_device_link.o

The CUDA language support in CMake is applying all the link libraries for a target on the off chance that there is some GPU code that needs to be extracted and put into that object file with the other cuda libs for the target device – all the CPU objects in those libraries are ignored.

So, I think it would be sufficient that the UPCXX_CUDA_LINK_OPTIONS variable is empty (or only contains *.cu based objects in the upcxx-gpu build).

[Jonathan Madsen]

This may be useful. I compiled a C++ file and set the LINKER_LANGUAGE to CUDA, and I compiled a CUDA file and set the linker language to C++.

$ cat ../CMakeLists.txt

cmake_minimum_required (VERSION 3.12 FATAL_ERROR)
project (Options LANGUAGES CXX CUDA VERSION 0.0.1)

add_library(options INTERFACE)

target_compile_options(options INTERFACE $<$<COMPILE_LANGUAGE:CXX>:-Wl,-Map=compile_options.cxx.compile>)
target_link_options   (options INTERFACE $<$<COMPILE_LANGUAGE:CXX>:-Wl,-Map=link_options.cxx.compile>)
target_link_options   (options INTERFACE    $<$<LINK_LANGUAGE:CXX>:-Wl,-Map=link_options.cxx.link>)

target_compile_options(options INTERFACE $<$<COMPILE_LANGUAGE:CUDA>:-Xlinker=-Wl,-Map=compile_options.cuda.compile>)
target_link_options   (options INTERFACE $<$<COMPILE_LANGUAGE:CUDA>:-Wl,-Map=link_options.cuda.compile>)
target_link_options   (options INTERFACE    $<$<LINK_LANGUAGE:CUDA>:-Wl,-Map=link_options.cuda.link>)

add_executable(regex-cxx regex.cpp)
target_link_libraries(regex-cxx PRIVATE options)
set_property(TARGET regex-cxx PROPERTY LINKER_LANGUAGE    CUDA)
set_property(TARGET regex-cxx PROPERTY CUDA_ARCHITECTURES OFF)

add_executable(regex-cuda regex.cu)
target_link_libraries(regex-cuda PRIVATE options)
set_property(TARGET regex-cuda PROPERTY LINKER_LANGUAGE    CXX)
set_property(TARGET regex-cuda PROPERTY CUDA_ARCHITECTURES OFF)


$ cmake .. -G Ninja
-- The CXX compiler identification is GNU 8.4.0
-- The CUDA compiler identification is NVIDIA 10.2.89
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jrmadsen/devel/c++/scratch/options/build

Basically, compile_options.cxx.compile encodes the command (e.g. target_compile_options), the language used in the generator expression (e.g. CUDA), and whether the generator was COMPILE_LANGUAGE or LINK_LANGUAGE.

C++ Target

$ ninja -v -j1 regex-cxx
[1/2] /usr/bin/c++   -Wl,-Map=compile_options.cxx.compile -MD -MT CMakeFiles/regex-cxx.dir/regex.cpp.o -MF CMakeFiles/regex-cxx.dir/regex.cpp.o.d -o CMakeFiles/regex-cxx.dir/regex.cpp.o -c ../regex.cpp
[2/2] /usr/bin/g++ -Wl,-Map=link_options.cuda.compile -Wl,-Map=link_options.cuda.link CMakeFiles/regex-cxx.dir/regex.cpp.o -o regex-cxx   -L"/usr/local/cuda/targets/x86_64-linux/lib/stubs" -L"/usr/local/cuda/targets/x86_64-linux/lib"

CUDA Target

$ ninja -v -j1 regex-cuda
[1/2] /usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler   -Xlinker=-Wl,-Map=compile_options.cuda.compile -std=c++14 -MD -MT CMakeFiles/regex-cuda.dir/regex.cu.o -MF CMakeFiles/regex-cuda.dir/regex.cu.o.d -x cu -c ../regex.cu -o CMakeFiles/regex-cuda.dir/regex.cu.o
[2/2] /usr/bin/c++  -Wl,-Map=link_options.cxx.compile -Wl,-Map=link_options.cxx.link CMakeFiles/regex-cuda.dir/regex.cu.o -o regex-cuda -L/usr/local/cuda/targets/x86_64-linux/lib/stubs   -L/usr/local/cuda/targets/x86_64-linux/lib -lcudadevrt  -lcudart_static  -lrt  -lpthread  -ldl 

Summary

• CXX language + CUDA linker

  • When CUDA is the LINKER_LANGUAGE, it invokes g++ not CMAKE_CXX_COMPILER when linking
  • INTERFACE_LINK_OPTIONS from $<COMPILE_LANGUAGE:CUDA> and $<LINK_LANGUAGE:CUDA> get passed during linking
  • INTERFACE_COMPILE_OPTIONS for $<COMPILE_LANGUAGE:CXX> get passed during compilation

• CUDA language + CXX linker

  • INTERFACE_LINK_OPTIONS from $<COMPILE_LANGUAGE:CXX> and $<LINK_LANGUAGE:CXX> get passed during linking
  • INTERFACE_COMPILE_OPTIONS for $<COMPILE_LANGUAGE:CUDA> get passed during the compilation

This is basically what is expected but it may help as a reference for figuring out correct linker language and generator expression settings to that the proper flags get included/excluded from your target.

‌

[Jonathan Madsen]

Ah, I said this is what is expected but that didn’t quite ring true. I couldn’t put my finger on it but here it is. The slightly strange quirk is that $<COMPILE_LANGUAGE:...> is true for the linker-language in INTERFACE_LINK_OPTIONS

[Jonathan Madsen]

It appears that this is the solution:

‌

INTERFACE_LINK_OPTIONS        "$<$<LINK_LANGUAGE:CXX>:${UPCXX_LINK_OPTIONS}>" "$<$<LINK_LANGUAGE:CUDA>:${UPCXX_LINK_OPTIONS}>"
INTERFACE_COMPILE_OPTIONS     "$<$<COMPILE_LANGUAGE:CXX>:${UPCXX_OPTIONS}>"

‌

Bc since the linking is done by g++, you don’t need to adding -Xlinker to the flags unless they get added to INTERFACE_COMPILE_OPTIONS.

[Rob Egan]

I think I have a much easier fix to the issue:

set_property(TARGET UPCXX::upcxx PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS OFF)

This should prevent that cuda_device_link.o from picking up this target as one of the dependencies, which is the culprit in my testing.

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

Mass roll-over of open issues to next release milestone

[Dan Bonachea]

Rob has reported he has a workaround for this issue, so no action to be taken in the near future