Build error with CUDA: fatal error: cuda_profiler_api.h: No such file or directory

[Bernd Waibel]

I’m working on packaging the library for Gentoo Linux, when it failed to build sparse/blas/magma_sparse_generated_zgeisai_batched32.cu.o with the above error.

The output is as

[2591/3376] cd /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas && /usr/bin/cmake -E make_directory /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/. && /usr/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=RelWithDebInfo -D generated_file:STRING=/var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/./magma_sparse_generated_zgeisai_batched32.cu.o -D generated_cubin_file:STRING=/var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/./magma_sparse_generated_zgeisai_batched32.cu.o.cubin.txt -P /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/magma_sparse_generated_zgeisai_batched32.cu.o.RelWithDebInfo.cmake
FAILED: CMakeFiles/magma_sparse.dir/sparse/blas/magma_sparse_generated_zgeisai_batched32.cu.o /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/magma_sparse_generated_zgeisai_batched32.cu.o
cd /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas && /usr/bin/cmake -E make_directory /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/. && /usr/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=RelWithDebInfo -D generated_file:STRING=/var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/./magma_sparse_generated_zgeisai_batched32.cu.o -D generated_cubin_file:STRING=/var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/./magma_sparse_generated_zgeisai_batched32.cu.o.cubin.txt -P /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/magma_sparse_generated_zgeisai_batched32.cu.o.RelWithDebInfo.cmake
/var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2/sparse/blas/zgeisai_batched32.cu:16:10: fatal error: cuda_profiler_api.h: No such file or directory
16 | #include <cuda_profiler_api.h>
|          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMake Error at magma_sparse_generated_zgeisai_batched32.cu.o.RelWithDebInfo.cmake:220 (message):
Error generating
/var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/CMakeFiles/magma_sparse.dir/sparse/blas/./magma_sparse_generated_zgeisai_batched32.cu.o

‌

I’m using CUDA 11.6.2 and the file cuda_profiler_api.h is present, but it seems like the cuda include path isn’t passed to this specific file, although it’s present in the summary report at the end of cmake configuration run within the INCLUDE_DIRECTORIES variable.

Please see the attached build log for more details.

[Stan Tomov]

That’s strange indeed. We can try to reproduce it. For what GPU are you trying to do the build and what cmake version are you using? I see that you didn’t specify one and by default you get:

-- Found CUDA 11.6
--     CUDA_INCLUDE_DIRS:   /opt/cuda/include
--     CUDA_CUDART_LIBRARY: /opt/cuda/lib64/libcudart.so
--     compile for CUDA arch 7.5 (Turing)
-- Define -DMAGMA_HAVE_CUDA -DMAGMA_CUDA_ARCH_MIN=750

Just to be on the safe side about support for old CUDA versions, can you add to cmake option for the GPU target, e.g., for Volta will be “-DGPU_TARGET=Volta”. If that’s not the problem I can try to reproduce it.

‌

[Bernd Waibel]

Trying to build for Turing target (GeForce GTX 1660 Ti), which also get’s passed to cmake v3.23.1.

CMake is called as

cmake -C /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/gentoo_common_config.cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DBLA_VENDOR=Generic -DBUILD_SHARED_LIBS=ON -DGPU_TARGET=Turing -DMAGMA_ENABLE_CUDA=yes -DMAGMA_ENABLE_HIP=no -DUSE_FORTRAN=ON -Dverbose=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2_build/gentoo_toolchain.cmake /var/tmp/portage-ondisk/portage/sci-libs/magma-2.6.2-r1/work/magma-2.6.2

which you can also see in the build log and includes -DGPU_TARGET=Turing

[Bernd Waibel]

My bad. I’m testing in a chroot sandbox and had CUDA installed without the profiler option being enabled. After reinstalling CUDA with the profiler option enabled the package compiles fine and the error is gone.

I’m very sorry for the noise.

[Bernd Waibel]

Misconfiguration

[Heyang Qin]

I want to add something as this is the very first result from Google search. If you do have cuda_profiler_api.h in CUDA directory but nvcc cannot find it, check the location of your nvcc. In my case, the recent change of conda repo introduces its own nvcc that becomes the default compiler. After switching back to system nvcc, the error is gone.