Minimum compute capability supported by PyCA is 1.3

[Tian Cao]

I want to setup pyca on cluster. I can compile it successfully, however I got the following error when I run example ROFTV.py:

iteration 0... Traceback (most recent call last): File "ROFTV.py", line 320, in <module> RunTest() File "ROFTV.py", line 309, in RunTest Mask=Mask) File "ROFTV.py", line 207, in RunROFTV en = calcEnergy(I, Data, DataFidC, TVC, TVPow, Mask) File "ROFTV.py", line 74, in calcEnergy imEn = (DataFidC/2.0)Sum2(scratchI) File "/home/tiancao/python/bass/site-packages/PyCA/Core.py", line 5257, in Sum2 return _Core.Sum2(args) RuntimeError: PyCA::Opers::Sum2(float &,PyCA::Image3D const &): PyCAException : From /home/tiancao/src/PyCA/Code/Cxx/src/base/GReduce.cu:44 : -- Cuda last error: invalid device function

I compiled pyca with: swig:1.3.40 python:2.6.6 cmake:2.8.7 gcc:4.4.7 cuda: 4.2

Any suggestions? Thanks.

[Sam Preston]

the "invalid device function" errors usually have to do with the compute capability of the cuda device on which you're trying to run the kernel. What is the compute capability of the devices on the cluster node you're using?

[Tian Cao]

there are three gpus on this node: Quadro FX 5600 (compute capability 1.0) Quadro FX 5600 (compute capability 1.0) GeForce 9800 GX2 (compute capability 1.1, default)

[Sam Preston]

Yes, that's your issue. By default there's a CUDA_ARCH_VERSION cmake variable that's set to '13' (compute capability 1.3). I know you won't be able to run PyCA on compute capability 1.0 since it requires atomic instructions introduced in 1.1. You can try setting CUDA_ARCH_VERSION to 11 and running on the 9800, I think it should work but I'm not totally sure since I've never compiled on anything lower than 1.3. Post here and let us know if it does or does not, though, since it would be good information for others running on older cards.

[Tian Cao]

Thanks, Sam. I have tried CUDA_ARCH_VERSION with 11, and I cannot compile it successfully.

[Sam Preston]

Okay, then I'd guess that we're using some functionality introduced in 1.3. Just out of curiosity, what compile errors do you get?

[Tian Cao]

Here are the errors:

Building NVCC (Device) object Cxx/src/CMakeFiles/PyCA.dir/alg/./PyCA_generated_GFluidKernelFFTKernels.cu.o
/home/tiancao/src/PyCA/Code/Cxx/inc/base/gcache.h(231): error: identifier "__hiloint2double" is undefined

/home/tiancao/src/PyCA/Code/Cxx/inc/base/gcache.h(306): error: identifier "__hiloint2double" is undefined

/home/tiancao/src/PyCA/Code/Cxx/inc/base/gcache.h(358): error: identifier "__hiloint2double" is undefined

/home/tiancao/src/PyCA/Code/Cxx/inc/base/gcache.h(436): error: identifier "__hiloint2double" is undefined

4 errors detected in the compilation of "/tmp/tmpxft_000055c7_00000000-4_GGaussUtils.cpp1.ii".
CMake Error at PyCA_generated_GGaussUtils.cu.o.cmake:256 (message):
  Error generating file
  /home/tiancao/src/pyca-bin/Cxx/src/CMakeFiles/PyCA.dir/alg/./PyCA_generated_GGaussUtils.cu.o

[Sam Preston]

I think it would be possible to remove some of those references and get PyCA to compile under compute capability 1.1, but those cards are pretty old and I'm not sure it's worth the effort. Is this a priority for you, or can you find machines with newer cards to run on?

[Tian Cao]

Thanks, Sam. I found another cluster with tesla gpu card, so it's not a problem now.

[Sam Preston]

Changing the title of this issue to reflect the actual problem

[Sam Preston]

we're not going to support cards older than compute capability 1.3