- changed component to tests
Parallelize regression test running
Demos are currently run through sequentially. This can be speed up with task parallelism. Done properly, the framework can be more robust w.r.t single demos hanging through timeouts of the demo processes.
I have written code for running a process with Popen with timeout.
Comments (11)
-
reporter -
reporter -
reporter The same approach should be applied to regression tests in ffc while at it.
-
reporter - attached test_running_process_in_threads.py
Attached a file with tests and code using Popen and threading to run a list of commands with single-job and total timeouts, a possible solution.
A possibly better solution than using the full
call_in_parallel
function in here, is to only use theget_future_call
function to run each single demo with a timeout from a pytest framework parallelized with xdist. -
reporter - removed responsible
- removed milestone
-
How about we run the Python regression tests through pytest, and the C++ regression tests through gtest. Each has a way to run in parallel.
-
reporter The demos need to run as separate processes. I don't see why pytest can't be used to launch those processes.
-
With gtest, all cpp demos could be wrapped into one binary. We can run multiple version of the binary with flags indicating which demo(s) each instance should run.
-
reporter Not without rewriting the demos. They all have their own main function.
-
- changed milestone to 2018.1
-
Python unit tests can now in parallel with pytest-xdist (used on CircleCI). C++ unit tests are very fast, so little to gain from parallel execution.
Big win would be running regression tests in parallel.
- Log in to comment