[4.4.0] Integrating Clover with JUnit 5 Parameterized Tests

Issue #92 resolved
Ramu Gudelli created an issue

Clover is not able to display test results properly if the tests are run with new Junit Platform (Junit 5) even though the tests are using Junit 4 Parameterized Runner class.

Comments (21)

  1. Ramu Gudelli reporter

    Not sure if the comments is the right place to ask this question but do is there any Junit 5 Test Execution Listener similar to Junit4 TestRun listner (com.atlassian.clover.recorder.junit.JUnitTestRunnerInterceptor)?

    Currently, if I run tests using Junit 5 Parameterized Tests, then the links on the Test Results tab not working. I am getting the following error when I click on the link. See below attached screen shots.

    Image 03-21-2019 001.png

    Image 03-21-2019 002.png

  2. James Spagnola

    Background: I only helped submit a PR for handling JUnit5's package private visibility changes which clover previously required everything to be public.

    Looking around: The JUnit5 Parameterized Test and JUnit5 Display Name annotations need to be integrated as well. I have a sketch of the name extractors for those method, but I'm not sure how to plum that through to the Interceptor or a Sniffer.

  3. Marek Parfianowicz

    Hi James, thank you very much for looking into this topic.

    How test name sniffers work is indeed non trivial, I had to scratch my head a lot when I was implementing the solution :)

    It works this way:

    1) When OpenClover instruments a source file (it works for both Java and Groovy), it adds the

    public static final TestNameSniffer __CLRx_y_z_TEST_NAME_SNIFFER

    field to every top-level class, where x_y_z is OpenClover's version number.

    2) Depending on the type of the test class detected, this field gets initialised with one of:

    =new com_atlassian_clover.JUnitParameterizedTestSniffer();

    =new com_atlassian_clover.SpockFeatureNameSniffer();

    =com_atlassian_clover.TestNameSniffer.NULL_INSTANCE;

    See: com.atlassian.clover.instr.java.RecorderInstrEmitter#generateTestSnifferField(com.atlassian.clover.recorder.pertest.SnifferType) method

    com.atlassian.clover.instr.groovy.Grover#createTestNameSnifferField(org.codehaus.groovy.ast.ClassNode, com.atlassian.clover.recorder.pertest.SnifferType)

    3) Every test method in a test class is being wrapped into a try-finally block (more precisely - the original code of the method is being rewritten to another method, the OpenClover's wrapper gets the original method name). In the finally block (so at exit of the test), new per-test code coverage file is written to disk. The code calls __CLRx_y_z_TEST_NAME_SNIFFER.getTestName() method to get the name of the currently executed test ('null' if unknown).

    So this is half of the work - how to store the test name in a per-test coverage file.

    Now we have to figure out what was the name of the test. Here a test interceptor comes in.

    4) All of JUnit3, JUnit4 and Spock test frameworks offer an API to read name of the currently executed test, it also gives a reference to a test class.

    What you have to do is:

    • find a __CLRx_y_z_TEST_NAME_SNIFFER field in the test class via reflections
    • figure out class type of the sniffer
    • call a proper setter method to store test name

    Example: com.atlassian.clover.recorder.junit.JUnitTestRunnerInterceptor#testStarted(org.junit.runner.Description) + com.atlassian.clover.recorder.junit.JUnitTestRunnerInterceptor#lookupTestSnifferField

    5) You have to execute test framework with the proper test interceptor.

    In case of JUnitTestRunnerInterceptor you can use proper maven-surefire-plugin option or enable it programatically.

    In case of CloverSpockIterationInterceptor it's being done via CloverSpockGlobalExtension which happens automatically thanks to META-INF/services.

    And now all should work:

    • test framework executes an interceptor
    • interceptor looks up the sniffer field and stores a test name
    • finally block in a test method reads the test name as stores in a per-test coverage file
  4. Marek Parfianowicz

    I assume that JUnit5 also offers an API to read name of the test.

    I suggest to enhance existing com.atlassian.clover.recorder.junit.JUnitTestRunnerInterceptor to handle JUnit5 (yet another interface to implement?)

    In case it turns out to be impossible, then you could create a separate class (but this would bring a problem how to distinguish it from JUnit4), e.g. JUnit5TestRunnerInterceptor. Next you'd have to inject this class into TEST_NAME_SNIFFER field in RecorderInstrEmitter#generateTestSnifferField and Grover#createTestNameSnifferField.

  5. James Spagnola

    Thanks for the info @marek-parfianowicz. I'm trying to understand some more things.

    1. TestNameExtractor vs TestNameSniffer.

    Maybe the distinction isn't that big of a deal. I've done the pieces to implement TestNameExtractor. Looking at JUnit5 changes, for JUnitTestRunnerInterceptor and TestNameSniffer we can implement TestExecutionListener. https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom

    1. That brings me to how would I add junit5 as a dependency to the build?

    One thing that may be tricky is JUnit5 is now split into 3 pieces, From the junit docs `JUnit 5 = JUnit Platform + JUnit Jupiter + JUnit Vintage.

    1. The next would be testing it all. Consider the clover-antresources/parameterized-junit4. My thought is that it's you would want to have exactly the same resources under test but for JUnit 5 vintage. Then have another set of resources that use the Junit5 equivalent annotations.
  6. Marek Parfianowicz

    Some test frameworks allow to have a name of the test different from the name of the method.

    For example in Spock you can type {{def "this is my test"}}.

    In JUnit you can have a name template, e.g. {{@ Parameterized(name = "this is {0} test")}}

    The TestNameExtractor class is meant to extract this information and store it in the model. AFAIR it can be later used for reporting.

  7. Marek Parfianowicz

    Testing - having a {{clover-ant/resources/parameterized-junit5}} is a good idea.

    Add junit5 as a dependency to the build - I have to check it. Definitely we'd have to avoid a conflict with JUnit4.

  8. Marek Parfianowicz
    • changed status to open

    In progress... I've already merged the branch and now testing for regressions and missed corner cases.

  9. Marek Parfianowicz

    From https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom:

    “6.1.4. Plugging in your own Test Execution Listener

    In addition to the public Launcher API method for registering test execution listeners programmatically, by default custom TestExecutionListener implementations will be discovered at runtime via Java’s java.util.ServiceLoadermechanism and automatically registered with the Launcher created via the LauncherFactory. For example, an example.TestInfoPrinter class implementing TestExecutionListener and declared within the /META-INF/services/org.junit.platform.launcher.TestExecutionListener file is loaded and registered automatically.”

    I will add the META-INF/services too.

  10. James Spagnola

    Hi @Marek Parfianowicz . Awesome! Thanks a lot for addressing this feature. I had a couple of questions.

    1. Do these changes handle the annotations @DisplayName and @ParameterizedTest#name?
      -The PersonTest for JUnit5 doesn't use the name property like the JUnit4 test.
      @Parameterized.Parameters(name = "{0} is a {1} [{index}]")
      -I couldn’t find a unit test for DisplayName which allows overriding the test name. It may be out of scope but seems closely related.
    2. When will 4.4.0 be released?
  11. Marek Parfianowicz

    ad 1. Good question … I just hooked into JUnit5 listener and it provided me display name of the test. I think it does not read the static test name - only the original method name is shown in the HTML report. I’ll double-check it.

    ad 2. I have only one issue left - #18 (deprecation of Maven 2) - issue is already in progress, I implemented most of the changes, however I got stuck on some weird bug that Maven components are not being injected to the MOJO class. Thus it’s hard for me to tell at the moment how long it will take (I hope not much). But feel free to build OpenClover from sources - changes are on the main (‘default’) branch already.

  12. Log in to comment