Compile error when using mpi conduit

[Tianchen Liu]

Hello, I’m new to upc++ and distributed computing. I was trying to compile hello_gasnet.cpp with mpi conduit on my server (with 2 nodes).

‌

upcxx -g -network=mpi  hello_gasnet.cpp

There are a bunch of errors like this:

upcxx-install/gasnet.debug/lib/libammpi.a(ammpi_ep.o): In function `MPI_ErrorName':
upcxx/bld/GASNet-2020.3.0/other/ammpi/ammpi_internal.h:344: undefined reference to `MPI_Error_string'
/upcxx-install/gasnet.debug/lib/libammpi.a(ammpi_ep.o): In function `AMMPI_AllocateEndpointResource':

Seems like ammpi couldn’t find mpi’s path.

But when I installing upcxx, it shows:

And I already added ./configure --with-mpi-cc=mpicc --with-mpirun-cmd=mpirunwhen configuring gasnet.

Is it possible to provide a more specific instruction or tutorial about how to configure the mpi conduit?

[Dan Bonachea]

Hello - I think most likely what you are missing is: ./configure CXX=mpicxx

You should not set --with-mpirun-cmd=mpirun, that's the wrong syntax for that option and will cause problems.

Also note the output above indicates the InfiniBand drivers were found (possibly as a default part of your distro). If you happen to have real InfiniBand network hardware (ie from Mellanox) then -network=ibv is recommended. Otherwise if your nodes are connected via Ethernet, then -network=udp is the recommended choice.

Finally, note there was a 2020.3.2 patch release last week with a number of bug fixes, so you might want to pick up that download when starting over.

Hope this helps!

[Tianchen Liu]

@Dan Bonachea Is it

./configure --with-cxx=mpicxx

in gasnet path?

[Dan Bonachea]

./configure --with-cxx=mpicxx

Either spelling should work, provided you have mpicxx in your PATH. Some MPI implementations spell it mpic++ but the former is more common. This requests UPC++ to build using the MPI-wrapped C++ compiler, which ensures the MPI libraries get linked in properly at application build time. These instructions are documented here for ibv-conduit with MPI spawning, but the same applies to mpi-conduit (which again, is generally not recommended because it's basically never the best performance choice for any platform).

[Tianchen Liu]

@Dan Bonachea

Already passed make check NETWORKS='mpi' tests. I think it should be work.

UDP is a much simpler way to use, I’ll try both.

Much appreciate!

[Dan Bonachea]

Already passed make check NETWORKS='mpi' tests. I think it should be work.

Sounds good! Thanks for reaching out.

UDP is a much simpler way to use, I’ll try both.

If you're only interested in functional correctness, then either should work. However if you're looking for the best performance, you should choose the conduit that is the best match to your network hardware. For Ethernet hardware, that is currently udp-conduit.

[Tianchen Liu]

@Dan Bonachea

One more question, is it possible to assign multiple processes to one host?

If yes, how to configure it?

[Dan Bonachea]

is it possible to assign multiple processes to one host? If yes, how to configure it?

Yes it's possible but the details differ depending on which network and job spawner you are using.

If you are using udp or ibv, the easiest way is just to list the host multiple times, eg:

$ upcxx-run -n 4 -ssh-servers 'pcp-d-5 pcp-d-5 pcp-d-6 pcp-d-6' ./a.out

You can also pass -vv to activate some status lines (near the end of initial output) that show where processes are running:

$ upcxx-run -n 4 -ssh-servers 'pcp-d-5 pcp-d-5 pcp-d-6 pcp-d-6'  -vv ./a.out
...
UPCXX: Process 0/4 (local_team: 0/1) on pcp-d-5 (16 processors)
UPCXX: Process 3/4 (local_team: 0/1) on pcp-d-5 (16 processors)
UPCXX: Process 1/4 (local_team: 0/2) on pcp-d-6 (16 processors)
UPCXX: Process 2/4 (local_team: 1/2) on pcp-d-6 (16 processors)
Hello from 1 of 4
Hello from 2 of 4
Hello from 0 of 4
Hello from 3 of 4