RFE: mechanisms for "known failures" on tests

[Paul Hargrove]

We have identified a gap in the capabilities of our make check and make dist-check logic with respect to tests which are known to fail. Currently, the use of TESTS and NO_TEST (with CI_ prefixed in GitLab CI) is our only mechanism for excluding a given test.

Not only can this gap complicate testing, but it has so far deterred us from adding reproducers for unresolved issues to our repo. In particular, we may avoid adding tests that might only fail for one compiler family but which would be beneficial to test with others.

This issue is a feature request for a mechanism to indicate that a test should be compiled and run normally, but have any failure reported as "KNOWN" instead of "FAILED".

The proposed mechanism is based on the pattern set by the existing TEST_ENV_* and TEST_ARGS_* environment variables. My proposal is to add TEST_KCF_* and TEST_KRF_* for, respectively, "Known Compile Failure" and "Known Run Failure".

The semantics would be that a non-empty value would lead to replacement of "FAILED" with "KNOWN" (in the event of failure) followed by the variables value as additional explanatory text. This is perceived to be of particular value when the test name does not identify (or could missidentify) the problem. Dan noted a good example in which the issue478 test is a reproducer for issue 536: TEST_KRF_ISSUE478='issue 536: Unfulfilled promise leaks memory if it has a dependent future created by then() or when_all()'

This choice of per-test environment variables means that we can easily set values in bld/tests.mak for tests that fail "everywhere" or only for certain compiler(s). Platform-specific known failures could be added in the same place (possibly with some work to add make variables for appropriate conditionals) or add them to the gitlab-ci.yml.

The "non-empty" aspect of the specified semantics allows a user or the .yml to simply override a default by setting it to an empty value. For manual runs this can be done in the environment or on the make command line. When using GitLab CI, this can be done as pipeline variables.

[Paul Hargrove]

Before I put any effort into implementation, I'd appreciate feedback.

[Paul Hargrove]

Proposed implementation appears in pull request #493

[Paul Hargrove]

Add known failure handling to Makefile.tests

This commit resolves issue #610 by adding a family of environment variables (with the opportunity to provide defaults in tests.mak) which annotate tests known to fail either to compile or run.

→ <<cset 1a43ef5c42f4>>