Limitation to 255 threads using smp

[Jérémie Lagravière]

Dear UPC++ Users,

At my lab we have a Knight's Landing Intel Processor on a machine. This CPU offers the possibility to run a program on 272 logical cores.

The problem is that when I try to run a UPC++ program on this amount of cores (272), I get this error:

$ make runLegit THREADS=272 BLOCKSIZE=65536
export UPCXX_GASNET_CONDUIT=smp &&  export UPCXX_SEGMENT_MB=167  && export GASNET_MAX_SEGSIZE=100000MB && export GASNET_PSHM_NODES=272 && export LD_LIBRARY_PATH=/home/jeremie/gccForUpcxx/gcc_6_4_0_release/installed/lib64 && /home/jeremie/myRepo/compilers/BUPC++_GCC/installed/bin/upcxx-run -n 272 upcxxProgram/upcxxSpmv ../dataset/D67MPI3Dheart.55 100 65536
*** FATAL ERROR: Nodes requested (272) > maximum (255)

Is there a way to bypass this limitation? Is this limitation done on purpose? Can UPC++ smp network conduit be improved to support more than 255 threads ?

For info: I think I remember having the same issue on UPC, GASNET was the reason of the limitation. UPC offers the possibility to pack threads together in pthreads or something like that...Is there an equivalent for UPC++

Thank you in advance for your help

[Dan Bonachea]

Hi @jeremieLagraviere

You can bypass this limitation by installing UPC++ with the following setting:

export GASNET_CONFIGURE_ARGS="--enable-large-pshm"
./install <whereever>

This raises the smp-conduit process limit from 255 to 45k.

The same configure option also works for Berkeley UPC.

PS - you should no longer need to explicitly set GASNET_MAX_SEGSIZE - recent releases have much "smarter" defaulting for that knob.