Improve test driver script

[Dan Bonachea]

@sbbaden asked me to write this up:

The UPCXX implementation should include a single, short, documented command that builds and runs a set of correctness tests (probably on smp-conduit) that are expected to pass, and ideally validates their success. ie the equivalent of "make check" if we were using automake.

This is important so that an ECP auditor who has no knowledge of UPC++ or GASNet can easily verify that our software passes its own tests, as a demonstration of milestone criteria.

This could initially be as simple as a "run-tests" shell script that loads up nobs and builds and runs each test and outputs "pass" or "fail" on each.

[Steven Hofmeyr]

I've added a run-tests script. Dan: if it passes muster, please mark this issue as resolved.

[Dan Bonachea]

Thanks Steve - the new script is certainly better than no script.

However I think it could still use some improvements (in rough priority order):

1. Documentation: the README needs to explain how to run this script, in particular how to properly setup variables that affect its operation (notably $CXX, $CC, $CROSS, $GASNET_CONDUIT, $OPTLEV, $DBGSYM)

2. Validate system requirements. If an auditor tries to run this script using an unsupported C++ compiler (eg g++ 4.8.5 which is the default on many Linux distros), he currently gets a long and confusing error message. Ideally we should start by validating system requirements and fail gracefully with an understandable error.

3. Output system information. If something goes wrong (or possibly even if not), the output log should record and include enough pertinent information about the platform and configuration to help us reproduce the problem when the user emails us a copy.

4. Default to parallel testing: Execution (nobs run) currently assumes smp-conduit. Unless the user has set GASNET_PSHM_NODES, run-tests will only run with a single rank, making for a pretty weak test. The run test script should probably set that to something reasonable like 4 if it's not already set.

5. Distributed execution support: We'll eventually need a way to allow an auditor to run tests on distributed-memory conduits to validate upcoming milestones. So we may need a feature to invoke the correct gasnetrun_* script (or a user-provided run command like "srun")

I recently added logic to the install script to handle points 2 and 3, we should discuss whether to use, clone or factor that logic to get similar effect.

[Steven Hofmeyr]

Dan, I'm not sure that the README.md is the right place for instructions to run this script. Currently, the README.md has no instructions for building, installing or compiling. It just refers to the programmer's guide. I intend to put instructions for running the tests in the programmers guide anyway. Perhaps it's sufficient for the README.md to just point to the guide (as it currently does). Otherwise, I'd suggest we put full build/install/compile instructions in the README.md, not just for run-tests.sh

[Dan Bonachea]

I agree it probably makes sense for README.md to contain either full build/install/compile instructions, or maybe a reference to a text-format INSTALL.md file that contains them.

The problem with only placing install instructions in the guide is it means someone who downloads our release to their favorite supercomputer needs a PDF reader to view the install instructions, which might entail additional steps for the end user. Textual instructions seem preferable in the source tarball.

Perhaps an INSTALL.md in the top-level of the implementation repo that contains the all build/install/compile instructions, and is also #included as a chapter in the guide?

[Steven Hofmeyr]

The new script together with some other changes now addresses all of Dan's suggested improvements:

1. README.md has a link to docs/testing.md, which explains how to run the tests.
2. run-tests uses a common script in utils to check the system
3. the same common script is used to output system info
4. testing is now parallel - it automatically detects the number of hardware threads and runs using a reasonable number of them, but not too many.
5. we have distributed execution support through certain conduits, e.g aries