Quarantining tests still fail a build

[Marcus Heese]

Hi,

I have a job which is parsing mocha test results using the "Mocha Test Parser". It is parsing test results that were previously created from the mocha-bamboo-reporter from the resulting mocha.json file. It is working completely fine as long as all tests pass.

However, now my developers have a few tests written that fail, and they want to ignore these tests. Bamboo has the possibility to quarantine these build results. Normally when I rerun previously failed jobs, these tests fail again, but they will be shown as Quarantined now and they won't fail a build because they are quarantined.

In my build the tests are now showing no errors, just 3 quarantined errors, however, the build still fails and I get the notice in my build summary: "No failed tests found, a possible compilation error occurred."

Now there are no further tasks in this build job and the mocha test parser is definitely the last task in the job. In my logs I can see the following:

simple  17-Jun-2014 23:17:24    Starting task 'Parse Test results' of type 'com.atlassian.bamboo.plugins.bamboo-nodejs-plugin:task.reporter.mocha'
simple  17-Jun-2014 23:17:24    Parsing test results...
simple  17-Jun-2014 23:17:24    Failing task since 3 failing test cases were found.
simple  17-Jun-2014 23:17:24    Finished task 'Parse Test results'

To me that suggests that the task is failing because it found failed tests. However, I always thought that the task should only fail if it is not able to parse tests at all. Parsing failed tests should still be a successful task. Then if there are failed tests, that should be up to bamboo to say that this is a failing build, or - as in my case - is not a failing build because the failing tests have been quarantined.

I would consider this a bug, however, I might be mistaken and this is expected behaviour. However, then I would need to know how I am supposed to use the Quarantine feature in bamboo.

Thanks a lot in advance to look this!

Best regards Marcus

[Marcin Oles]

Hi Marcus,

I took a look at it and this doesn't seem to be a problem with the "Mocha Test Parser", but rather the "Mocha Test Runner" (or the raw Node.js script that executes Mocha - if that's the way you execute your tests). The problem is that when there are failing tests, the exit code returned by Mocha is different than 0, so Bamboo treats the whole job as failed.

One possible solution is to make Mocha with mocha-bamboo-reporter always exit with exit code 0 (unless of course there is another problem with it's execution - then expect a different code), so then Bamboo could parse test results, apply quarantine rules and decide if the job has failed or not.

I've created a pull request that fixes this for the mocha-bamboo-reporter.

[Marcus Heese]

Hi Marcin,

please carefully review that. My test scripts always return 0 because of exactly that problem, so I am sure that this is not the problem. This is a standard "problem" for continuous integration software and I am aware of that.

Also, have a look at my posted log again which already shows the nature of the problem IMO.

Anyway, thank you for taking care of the problem. I really appreciate it. Please let me know if there is any way I can help you to fix this issue.

Regards Marcus

[Marcin Oles]

Hi Marcus,

you are right, I was too quick with my previous solution. There were actually two problems with quarantining tests, one related to the exit code after executing tests, and another related to parsing test results (as the parsing task was not "quarantine aware").

I've fixed the problem with "Mocha Test Parser" now, so you can expect it to work properly in the next version of the plugin.

Below is the log message you should expect now:

simple  22-Jul-2014 11:58:22    Starting task 'Parse Mocha Test Results' of type 'com.atlassian.bamboo.plugins.bamboo-nodejs-plugin:task.reporter.mocha'
simple  22-Jul-2014 11:58:22    Parsing test results...
simple  22-Jul-2014 11:58:22    Failing task since 1 failing test cases were found.
simple  22-Jul-2014 11:58:22    Changing Task Result to SUCCESS as all failed tests were quarantined.
simple  22-Jul-2014 11:58:22    Finished task 'Parse Mocha Test Results'

Best regards, Marcin

[Marcus Heese]

thank you @m_oles! I appreciate it!