Parallelize regression test running

[Martin Sandve Alnæs]

Demos are currently run through sequentially. This can be speed up with task parallelism. Done properly, the framework can be more robust w.r.t single demos hanging through timeouts of the demo processes.

I have written code for running a process with Popen with timeout.

[Martin Sandve Alnæs]

The same approach should be applied to regression tests in ffc while at it.

[Martin Sandve Alnæs]

Attached a file with tests and code using Popen and threading to run a list of commands with single-job and total timeouts, a possible solution.

A possibly better solution than using the full call_in_parallel function in here, is to only use the get_future_call function to run each single demo with a timeout from a pytest framework parallelized with xdist.

[Prof Garth Wells]

How about we run the Python regression tests through pytest, and the C++ regression tests through gtest. Each has a way to run in parallel.

[Martin Sandve Alnæs]

The demos need to run as separate processes. I don't see why pytest can't be used to launch those processes.

[Prof Garth Wells]

With gtest, all cpp demos could be wrapped into one binary. We can run multiple version of the binary with flags indicating which demo(s) each instance should run.

[Martin Sandve Alnæs]

Not without rewriting the demos. They all have their own main function.

[Prof Garth Wells]

Python unit tests can now in parallel with pytest-xdist (used on CircleCI). C++ unit tests are very fast, so little to gain from parallel execution.

Big win would be running regression tests in parallel.