77af90c·Author: Paul Hargrove·Closed by: Paul Hargrove·2017-08-06
Description
This commit updates ibv-conduit's handling of GASNET_PHYSMEM_MAX as follows:
Configuring using --with-ibv-physmem-max=VAL can replace the "default of defaults" (which is still 2/3).
GASNET_PHYSMEM_NOPROBE is replaced by GASNET_PHYSMEM_PROBE
Default for GASNET_PHYSMEM_PROBE is ON only in the case that neither the configure option nor environment variable has been used to set GASNET_PHYSMEM_MAX, and is OFF otherwise
ibv-conduit/README updated to reflect the changes above
When the probe does run for a size larger than 2GB, it is preceded by a warning, and followed by instructions for avoiding the probe in the future.
Example run on Dirac nodes 5 and 6:
$ time ./contrib/gasnetrun_ibv -n2 ./testhello
WARNING: Beginning a potentially slow probe of max pinnable memory...
WARNING: Probe of max pinnable memory has yielded '15 GB'.
WARNING: To avoid this probe in the future either reconfigure using
WARNING: --with-ibv-physmem-max='15 GB' or run with environment
WARNING: variable GASNET_PHYSMEM_MAX='15 GB'.
Hello from node 0 of 2
Hello from node 1 of 2
real 0m4.904s
user 0m0.107s
sys 0m3.838s
It is definitely worth noting that the probe is taking under 5 seconds, and not the multiple minutes of the original bug report.
This commit updates ibv-conduit's handling of GASNET_PHYSMEM_MAX as follows:
Configuring using --with-ibv-physmem-max=VAL can replace the "default of defaults" (which is still 2/3).
GASNET_PHYSMEM_NOPROBE is replaced by GASNET_PHYSMEM_PROBE
Default for GASNET_PHYSMEM_PROBE is ON only in the case that neither the configure option nor environment variable has been used to set GASNET_PHYSMEM_MAX, and is OFF otherwise
ibv-conduit/README updated to reflect the changes above
When the probe does run for a size larger than 2GB, it is preceded by a warning, and followed by instructions for avoiding the probe in the future.
Example run on Dirac nodes 5 and 6:
$ time ./contrib/gasnetrun_ibv -n2 ./testhello WARNING: Beginning a potentially slow probe of max pinnable memory... WARNING: Probe of max pinnable memory has yielded '15 GB'. WARNING: To avoid this probe in the future either reconfigure using WARNING: --with-ibv-physmem-max='15 GB' or run with environment WARNING: variable GASNET_PHYSMEM_MAX='15 GB'. Hello from node 0 of 2 Hello from node 1 of 2 real 0m4.904s user 0m0.107s sys 0m3.838sIt is definitely worth noting that the probe is taking under 5 seconds, and not the multiple minutes of the original bug report.