Track Tests in Pipelines - Unit Tests

Issue #12738 resolved
Ben Tatham
created an issue

One of the great features of Bamboo was the tracking the history of JUnit (xUnit) tests - and the quarantining, etc. We need that in Pipelines too.

Official response

  • Philip Hodder staff

    Hi everyone,

    We've shipped test report parsing to all Pipelines users now.

    We will automatically parse any jUnit-style XML files found in the following directories (up to 3 levels of nesting from the base directory of your build):

    • surefire-reports

    • test-reports

    • test-results

    • failsafe-reports

    For some tools (such as Maven) your build will automatically produce these. For others, you can configure your test library to generate the XML reports into the right directory.

    Have a look at our documentation for additional information and examples.

    Thanks,

    Phil

Comments (56)

  1. Andrew Goodchild

    Bamboo allows us to collect junit test reports and clover coverage reports. Failing tests can be used to fail a build. Similarly, if you script it up, you can also fail builds without sufficient test coverage and you can script bamboo to fail builds on lint errors. Also you can graph coverage and lines of code over time. All of these features need to be in pipeline

  2. Michael Pridemore

    I think it's implied here, but specifically it would be great to have link to the test reports available from the pipelines screen. Similar to the way the Jenkins JUnit plugin supports "Publish JUnit test result report".

  3. Michaël van de Giessen

    nr of tests green; failed(yellow/orange); ignored are imho important extra indicators of the build status

    the junit result format is a good starting point for parsing test results

    coverage and lines-of-code are sensible next steps from there

  4. Thomas Turrell-Croft

    Sonar might be a solution for you. You would have to stop failing tests from falling the build though.

    However I've changed my mind don't think Pipelines should parse and track tests. It should be relatively simple for a developer to write a plugin for Pipelines which calls the Pipelines API.

  5. Ben Tatham reporter

    @Thomas Turrell-Croft However I've changed my mind don't think Pipelines should parse and track tests.

    Just curious why not? Bamboo does a great job of it, and while it certainly can be a plugin, I see no reason why Pipelines should not do it natively - other than it might be a while until they get to it so perhaps a separate company/OSS group would be a faster approach. In order for a plugin to work, we would have to push the xUnit xml file(s) to an outside service manually, which of course is doable too, but not available on the Pipelines API.

  6. Thomas Turrell-Croft

    Pipelines could do lots of things but it would quickly become an unmaintainable mess.

    For instance Pipelines could lint code, check dependencies for vulnerabilities, check dependency licensing, display code coverage, track tests, store artefacts, deploy artefacts, etc. However I’d prefer see Pipelines implement multiple steps and a final step, that way I can simply use third party solutions for all of these things. And as you mention it would be awesome if the Pipelines logs were exposed to authorised third party plugins.

    I really liked Bamboo but I didn’t morn it when Atlassian killed the cloud version. The Bamboo elastic agent didn’t have the software I needed, the artefact storage concept didn’t quite work, coverage only worked with clover, I couldn’t deploy to my particular cloud provider, etc. The promise of an out of the box solution for all of my requirements was nearly their but it never quite delivered.

    I think GitHub has been successful because it does one thing really well. Gaps in GitHub are plugged by third parties. I hope that Atlassian do the same thing with their products.

    This is just my opinion, I hope Atlassian take a balanced view of all of their customers opinions. Sorry for being off topic.

  7. Andrew Simpson

    I can certainly understand concerns about feature creep, but given that the point of a CI product is to build and run tests, a facility to break down the results of those tests is really not an unreasonable feature. This is something which is available on CircleCI, for example, which I use for some of my builds.

  8. Matt Ryall staff

    Thanks for your interest in this issue. We'd like to talk to a few people who want test reporting in Bitbucket Pipelines to understand your use case in more detail.

    If you can spare an hour for a quick chat about this, please email the team at pipelines-feedback@atlassian.com, and we'll try to line up a time in the next few weeks. Please include in your email:

    • your Bitbucket username
    • what you use Pipelines for
    • your tech stack and test library/tool you want results for
    • your location/time zone.

    Thanks,
    Matt

  9. Ben Tatham reporter

    Looks like progress is being made on this...at least in parsing output for maven/surefire: test.jpg

    Great to see progress, and look forward to more as it comes...

  10. Matt Ryall staff

    Well spotted, @Ben Tatham. We're still ironing out some kinks in the feature, but it's now running for customers in the Pipelines Alpha program.

    Our test reporting feature will automatically pick up jUnit-style XML test output in the following directories in your build (up to 3 levels of nesting):

    • surefire-reports
    • test-reports
    • test-results

    This is designed to work automatically for many tools, like Maven. For others, you simply configure your test library to generate the XML reports in the right place.

    Please let us know what you think.

  11. Philip Hodder staff

    Hi everyone,

    We've shipped test report parsing to all Pipelines users now.

    We will automatically parse any jUnit-style XML files found in the following directories (up to 3 levels of nesting from the base directory of your build):

    • surefire-reports

    • test-reports

    • test-results

    • failsafe-reports

    For some tools (such as Maven) your build will automatically produce these. For others, you can configure your test library to generate the XML reports into the right directory.

    Have a look at our documentation for additional information and examples.

    Thanks,

    Phil

  12. Ben Tatham reporter

    @Philip Hodder @Matt Ryall This is a great first step - being able to see test failures quickly on an individual pipeline run...I hope the next feature in this area is to "track tests", as this issue title suggests. This means seeing a history on a per-test basis, the way Bamboo Cloud used to do it (with I'm sure a much simpler UI). This allowed us to quickly look for tests that failed sporadically, how often they failed, etc. (The Quarantine feature was nice to (which meant pass the build anyhow - but that requires more knowledge of the build process/tooling itself), but I suspect that would be difficult to do in pipelines and is not strictly necessary.)

    I created a new issue to vote for this enhancement...https://bitbucket.org/site/master/issues/15328/track-test-history-in-pipelines.

  13. Ben Tatham reporter

    Where do the 3 levels of nesting start from? For multi-module (aka reactor) builds, this is likely not good enough. /target/ steals a level in all cases on maven. I can't image that searching just for directory names even 6 levels deep is too much load on the system - especially if you only search on build failures. (or perhaps I underestimate the number of directories people have in their builds). Alternatively, let us configure a set of specific test result directories in the yml - either by pattern or even explicit definitions.

    For example, I have a failed test reported in /opt/atlassian/pipelines/agent/build/network-parent/http-client/target/surefire-reports that does not generate anything on the pipelines view (not even success). Assuming you start from ../build/, that is likely too deep.

  14. Philip Hodder staff

    Hello,

    Charles - Sorry, that was an oversight on my part. I'll chase up getting that added.

    Ben - The 3 levels of nesting starts on /opt/atlassian/pipelines/agent/build/. We are currently in the process of measuring the time of increases to searching a larger depth. I'll update this ticket again once we have the results of this, and any increases to search depth. I'll leave discussing the tracking aspect of this feature for @Matt Ryall to comment on (possibly on the new ticket you've created).

    Igor - The feature is globally enabled. The parsing logs are on their way. They got held up in the pipeline. Optimistically, they should be shipped by next week.

    On making search paths configurable: For now you can work around this by setting your build tool to output to a directory automatically scanned. Hopefully the depth size increases will sufficiently solve this problem, otherwise you can create a new issue for adding YAML configuration support.

    I'll post an update on these improvements next week.

    Thanks, Phil

  15. Ade Miller

    I'm still not entirely clear how your nesting limitation is applied. I was unable to get Pipelines to find tests in ./build/test-results/.xml and had to move them into ./test-results/.xml for them to be picked up. While I understand how you would want to limit search depth I would be quite OK with providing a configuration setting to setup where to look for test results files.

    There is no log output so it is hard to figure out why test results are not displayed. It seems like they are only added if the build fails. Is that the case? If so that seems odd, I would expect them to be viewable even if the build is green.

    Right now there is barely feature parity with the basic Bamboo feature set.

  16. Philip Hodder staff

    Hello all,

    Just another update on the remaining changes.

    We've bumped the search depth so that submodules should now be recognisable. I've also opened this ticket for people who need YAML configuration: Configure Test Reporting in YAML

    Parsing logs have been shipped now. They will appear in the "Build Teardown" section of your logs. We'll be updating the documentation later today to clarify the behaviour as well.

    Showing successful tests in the UI is on the way. It should ship in a week or two, the holiday season makes it tough to give a more precise estimate.

    We've also got some improvements scheduled for integrating the test results into existing pipelines notifications.

    I'll be back after the new year with another update.

    Happy holidays!

  17. Benj Kamm

    I think I may be missing something obvious, but I can't get this to work. (I'm on Pipelines Alpha, so should have access to the feature.)

    When my tests fail, the test runner executes with a non-0 exit code, so the pipeline stops immediately - and the "build teardown" is never run. This has been my expected approach until now, because if tests fail the build should stop.

    How are others doing this?

  18. xqliu

    Dear,

    I also have the same issue like @Benj Kamm the test runner executes with a non-0 exit code, so the pipeline stops immediately - and the "build teardown" is never run. , I am also on pipeline Alpha,

    Could you please help to look into this issue?

    Thanks.

  19. Dominique Busser

    Hello @Philip Hodder,

    I'm receiving this in the (successful) log of my pipelines step that generates coverage reports:

    Build teardown
    7s
    Cache "node": Skipping upload for existing cache
    Searching for test report files in directories named [test-results, failsafe-reports, test-reports, surefire-reports] down to a depth of 4
    Found matching test report file /opt/atlassian/pipelines/agent/build/test-reports/clover.xml
    Found matching test report file /opt/atlassian/pipelines/agent/build/test-reports/junit.xml
    Finished scanning for test reports. Found 2 test report files.
    All test reports aggregated into 778 test cases
    Uploading test results
    Finished uploading test results.
    

    So it seems the coverage information is uploaded correctly somewhere, the number of test cases looks correct also, so I'll assume the parsing of the .xml files works as well.

    However, I'm not seeing any reports anywhere? In the pipelines log, I only have the "Log" view, nothing like the coverage reports as displayed here: https://blog.bitbucket.org/wp-content/blogs.dir/10/files/2017/12/bitbucket-pipelines-test-reporting-1024x576.png

    // edit

    I just saw that this apparently has not shipped yet?

  20. Philip Hodder staff

    Hi @Justin Kamerman,

    Currently we're only storing the individual results on the failed tests. So they will be available to view when they fail.

    Is there anything in particular that you're interested in seeing from the successful tests? Or were you just trying to verify you had set everything up correctly? :)

    If you had particular ideas in mind, I'd encourage you to open a new issue here so that we can track that as a new body of work.

    Thanks, Phil

  21. Justin Crown

    Hey @Philip Hodder thanks for your excellent work on this feature.

    For the record, I would like the ability to view full test coverage report results regardless of pass/fail (similar to something like coveralls or codecov.io). This is very helpful for determining regressions in code coverage.

    Also, is there a way to fail the build if the code coverage drops below a certain threshold. Did not see anything about this in the docs here.

  22. Philip Hodder staff

    @Justin Crown, @Justin Kamerman,

    Test code coverage reports will need to be a separate body of work, as they're a separate format to parse (and we may want modified designs for that). But it's a perfectly valid feature request. :)

    I suggest you open a new feature request for Code Coverage reports. I couldn't find any existing ones from my brief search.

    Thanks,
    Phil

  23. Marco Massenzio

    I am probably missing something here, but I have failing tests, but can't see anywhere in the web UI where to look at the reports (and, most importantly, the full stacktrace of the error):

    There were failing tests. See the report at: file:///opt/atlassian/pipelines/agent/build/ingestion/build/reports/tests/test/index.html

    Obviously that file is inaccessible, however, as this is a Gradle build, it creates the correct XML reports in the correct place: where can I go look at them?

    All I see in the console logs is this:

    com.....RestControllerTest > postFunction FAILED
        java.lang.IllegalStateException
            Caused by: org.springframework.context.ApplicationContextException
                Caused by: org.apache.kafka.common.KafkaException
                    Caused by: org.apache.kafka.common.config.ConfigException
    

    what I'd like to be able to look at is the full stacktrace (the test, obviously, pass happily locally - so I'm kinda of at a loss why they fail here).

    I've spent a few hours re-creating the test environment for the pipelines (creating an "equivalent" Dockerfile & then running the pipeline step) so I can see the stacktrace now (the failure mode still eludes me) but I would rather avoid all this work, and just look at the test results in the web UI.

    If there's a button/link in the pipelines dashboard, I sure have missed it! Thanks.

  24. Matt Ryall staff

    @Marco Massenzio - if Pipelines is showing "12 tests failed" or similar against a failed step in your pipeline, it has detected the failing tests and you can switch to see them by using the "Logs" dropdown at the top of the logs pane. We're in the process of redesigning this screen to make this navigation more obvious.

    If no failed tests are reported, then Pipelines has not detected your XML test output. You can the output of our directory scanning process in the "Build teardown" section, which might help you see why it is not finding them.

    If nothing there helps, please raise a support ticket, and one of our engineers can have a look at your builds to see what's going on.

  25. Marco Massenzio

    @Matt Ryall thanks for follow up - I think I may have expressed myself incorrectly, so in the spirit of "a picture and all that" here's what I mean.

    When tests fail in pipelines, this is all I see in the logs: Screen Shot 2018-03-05 at 11.18.36 AM.png

    while, if the actual test reports are uploaded and parsed, this is what I'd really like to see: Screen Shot 2018-03-05 at 11.24.31 AM.png

    i.e., a navigable report, where I can further click-through and see the full logs of the errors: particularly with Java apps, the stack traces can be really deep and confusing and convoluted - where the real failure reason is buried several levels deep down (Spring, anyone?).

    BTW - Pipelines are a super-useful tool, I cannot stress enough how much I like them as a feature!

  26. Matt Ryall staff

    @Marco Massenzio - thanks for the clarification. That build output you've shown is not Pipelines test reporting - it's just your log output. If Pipelines could find your XML test output, you'd see a list of failed tests with the stack traces for each underneath, like this screenshot from our blog:

    Test reporting

    (Notice the "Test results" header on the right panel -- this indicates you're seeing our formatted test output, not your logs.)

    It could be that your test output is buried too deep in your directory hierarchy for us to find it. Or, if by random change you happen to use Gradle 3+, we just recently fixed our scanning of test reports to pick those up.

    In terms of full reports like that, two quick points:

    • If there's a report produced by your tool in HTML or text that looks like this, attaching it to your Pipelines build is something we'll consider adding in the future as a separate feature request, as @Philip Hodder mentioned above.
    • Based on our chats with customers, we decided to focus this feature on debugging failed tests, highlighting them and their output so your team can get back to green sooner. We've decided not to bake in a big report of all the passed tests, like your example, because while they're sometimes useful, it's only relatively rarely. We're really trying to avoid adding things like this to Pipelines that you only need 1% of the time, so the product stays lean and easy to use.

    In either case, if it's something you'd like in Pipelines, please raise an enhancement request and we'll leave it open to gauge the level of interest in it.

  27. Tales Santos

    Hi @Matt Ryall, thanks for this nice addition on Bitbucket Pipelines. I've just begin with this feature and may be very useful the capability to see the code coverage aside the failed/passed tests count. Is that possible?

  28. Adam Barnwell

    I realise this thread hasn't been posted on in a while but @Matt Ryall I'm interested to know what your parser looks for in the junit xml file to work out if their are any successes/failures etc...?

    I'm using tslint which allows for me to export the report as a junit XML file but it doesn't add any attributes to the nodes (such as failures, successes, etc...) it simply adds the nodes themselves e.g. <testsuites><testcase><failure>. According to BitBucket pipeline logs, the file is being found and BitBucket Pipelines is finding the correct amount of test suites, however it simply skips over the failure nodes and I'm assuming it's because it's missing the attributes.

  29. Log in to comment