Pipelines show 2 times the number of tests reported in junit.xml

Issue #15590 resolved
Joaquin Cuenca Abela created an issue

If you have a test suite that contains multiple (embedded) tests suites, Bitbucket Pipelines will count the tests executed and failed twice. See for example https://bitbucket.org/graphicresources/tutpad_web/addon/pipelines/home#!/results/1706

The xml is something like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="CodeIgniter Application Test Suite" tests="260" assertions="1110" failures="0" errors="9" time="51.481371">
        <testsuite name="Admin_course_test" file="/code/application/tests/unit/controllers/Admin_course_test.php" tests="7" assertions="45" failures="0" errors="3" time="3.380968">
            <testcase name="test_admin_create_course" class="Admin_course_test" file="/code/application/tests/unit/controllers/Admin_course_test.php" line="52" assertions="6" time="1.127683"/>
            [...]
      </testsuite>
    </testsuite>
</testsuites>

PHPUnit creates a global testsuite (named "CodeIgniter Application Test Suite" in our example) and puts all the tests suites inside this tag.

Also, for some reason, if I remove this external testsuite, leaving a flat list of testsuites, Bitbucket Pipelines does not parse my junit.xml, see for example this run:

https://bitbucket.org/graphicresources/tutpad_web/addon/pipelines/home#!/results/1707

The junit.xml is the same as before, except in this run I deleted the third line and the penultimate line with sed -i '3d;$!x' junit.xml. Pipelines did not found any tests in this xml, but it displayed no errors. I verified this file with verify-junit-xml (npm module) and it seems it does not have any errors.

Comments (6)

  1. Adam Robertson

    Only twice, ha, we get 3 times!

    e.g. we only have 51 tests, results show 24 / 153 tests failed (yeah not the best test run!)

    Running PHPUnit on a Laravel codebase, in a PHP 7.0.27 container, if that helps.

    vendor/bin/phpunit --verbose --log-junit ./test-reports/junit.xml

  2. Raul Gomis staff

    Hey!

    Good news! We have just deployed a fix for this issue. Now, the number of tests reported should match the ones in the xml file.

    Please, let me know if you have any more questions or concerns.

    Regards! Raul

  3. Thomas Van Durme

    This is not yet fixed. See screenshot below. Date of execution is March 13 2019. The number of tests run is 84, whereas reported is 252. Codebase is JAVA.

    Screenshot 2019-03-13 at 15.36.57.png

  4. Log in to comment