Automated testing

[Joseph Parker]

We should automatically run the tests on PRs. I have an old branch that set up pipelines for pfunit tests. I'll modify this to run the existing test suit.

[David Dickinson]

We have a limited number of pipeline minutes per month. I think we should explore having a light weight pipeline for general branches (e.g. compilation and cheap unit tests) and a more sophisticated pipeline for merges to master/next. Alternatively could explore hooking into another CI system that has more capacity.

[Joseph Parker]

Good point! One option is Anvil, which I believe is some kind of "forge" pun.

[Peter Hill]

I'd not heard of Anvil, that looks very useful!

[Stephen Biggs-Fox]

Anvil uses Jenkins as the CI server software. I've seen Jenkins in action before and it's fairly intuitive to use, plenty of customisable options, seems to be one of the industry standards. So, at a glance, Anvil looks like a good option.

[David Dickinson]

Will anvil be accessible to all international developers?

[Peter Hill]

PR #12 is in, it turns out we can't run tests automatically just on PRs and not every push, so they currently just run on commits to next.

Not all the tests pass -- do we want to wait till they do before resolving this issue?

[Joseph Parker]

Yes, unless we can identify which tests are not passing. I put a 10 minute limit on the pipeline (the one that works finished within 8 mins), but next times out. We could run it for longer, but I suspect one of the tests has stalled.

[David Dickinson]

Can we have a "does it compile" stage for all branches and a run the tests stage if commit is in next?

If our workflow involves "click the button that syncs a branch with next" prior to approving/merging the PR then this will ensure the branch actually contains what will be next. We could then manually trigger the test pipeline (or another custom pipeline) prior to approval/merging to ensure tests pass in the effectively merged branch.

[Peter Hill]

+1 for "does it compile" that's run on everything

Manually triggering the pipeline might be our best bet for now.

[Stephen Biggs-Fox]

I've seen David's suggestion in action before; it works well. In fact, it is good practice for developers to do this themselves anyway before submitting a PR so they can: (a) run the tests themselves on the effectively-merged code-base to check that it's all good and fix any issues if not before submitting the PR; (b) do it as a rebase rather than a merge to keep the revision history more linear (hence neater, easier to follow, etc.). This should probably be in the contributing document (either now or in due course).

[Peter Hill]

Once the test suite is a bit more robust and useful, we will be encouraging people to run the tests as they develop. Asking people to run the tests on bitbucket pipelines is a bit more dubious, as we currently have limited build minutes.

I would be very wary about asking people to rebase. If people know what it is and how to use it, that's fine, but I'm not keen on recommending people do it by default.

[Stephen Biggs-Fox]

OK, I meant for devs to run the tests on their own machine, not using a bitbucket pipeline.

Fair point about rebase. Those in the know should do it thought!

As for limited build minutes, what about David's idea of another CI system that has more capacity?

[Joseph Parker]

I've signed up the Gyrokinetics Project for an account on Anvil. These are the Terms of Service - I can't see limits on build minutes etc. Anyone with a Shibboleth account can be added to the project to administer the Anvil account; but once it's set-up, it can be made to run on anyone's PRs.