1. Brian Searle Avatar Brian Searle
  2. Proteomics
  3. EncyclopeDIA

Wiki

Clone wiki

EncyclopeDIA / Bitbucket_Pipelines

View History

Bitbucket Pipelines

EncyclopeDIA uses Bitbucket Pipelines for builds and testing of the project's code.

Current status

Pipelines builds are now used for most parts of the development, testing and deployment process of the EncyclopeDIA project. See Implementation Steps below for more information about the process of setting up Pipelines features.

Goals

In using Pipelines to build and test EncyclopeDIA we hope to (eventually) provide all the following benefits:

  • Automatic build and testing whenever code is pushed to the project's repository.
  • Integration with the development process to ensure that contributions are fully tested before being merged into the project.
  • Allow for both quick (unit-type) testing, as well as more involved builds that run long-running (integration-type) tests.
  • Allow easy access to build results from anywhere.
  • Simplify and automate the process of testing and producing releases.
  • Provide automatic deployment of EncyclopeDIA releases, both as executable (for end users) and dependency (for use as a software dependency) packages.

Using Bitbucket Pipelines

Depending on how you use or contribute to EncyclopeDIA, you will use Pipelines differently. This is roughly divided into three categories: users of the software, developers who contribute code, and maintainers, who are responsible for the administration and overall operation of the open source project.

For users

Users of EncyclopeDIA will not need to interact directly with the Pipelines feature of the project's Bitbucket site. They will continue to access executable JARs through the project's homepage, or the Bitbucket "Downloads" page.

For contributors

Pipelines builds will run on all commits to the repository, including those in pull requests. After creating a PR, you'll be able to see the results of building your code from pull request interface or from the "Commits" tab.

If a build with your changes fails any unit tests, you will see that the Pipeline indicates failure, and a "Tests" tab will appear for on the Pipeline's page. This will allow quick inspection of the test failures reported by Maven. For more information, see this page on test reporting for Pipelines.

In addition to the default build that runs unit tests on your branch, any pull request you create will trigger a pipeline that merges your code to the destination branch and executes a build with both unit and integration tests. PRs are configured so that both types of builds must run successfully before the PR is merged. Note that because additional (integration/regression) tests are run for PRs that aren't run on every other commit, your PR may begin failing tests despite passing branch-triggered builds. It is possible to run integration tests against any commit manually by running the custom: integration-tests pipeline.

For maintainers

In addition to contributor-like interactions, maintainers are responsible for configuring the Bitbucket project and Pipelines builds, as well as reviewing and merging pull requests. Some maintainers also contribute directly to the codebase without using pull requests, meaning they will use Pipelines builds of their development branch(es) to test their code.

Since merging PR #12 Pipeline builds are also used for (parts of) the process of deploying a release version of EncyclopeDIA. The process is (for now) still largely the same as the fully-manual approach:

  1. Run increment_version.sh to update versions and create an appropriate tag.
  2. Run git push origin encyclopedia-<version> (or git push origin --tags) to push the tag to the repo.
  3. The deploy build will automatically trigger, executing a full build (including ITs) and will upload the resulting executable JAR (assuming build/test success) to the project's Downloads section.
  4. (Optionally) update the project landing page (wiki) and current_version.txt to point to the new version.

Pipelines are configured through the bitbucket-pipelines.yml file in the repository, which offers fairly complex and powerful features to configure builds; see the official documentation for more information. Because each detail is so powerful, our choices are very deliberate:

  • We use a very-recent Maven Docker container, specifically specifying the JDK type (OpenJDK) and version (8).
  • We use the predefined (by Bitbucket) maven cache type, which saves and reuses the local maven repository (with dependency JARs); this cuts down build time significantly (from ~8.5 min to ~2.5 min [when building all executable JARs]).
  • Prior to setting up Pipeline builds a significant portion of the build time was spent constructing executable jars (after all tests have passed) even if these files would be discarded. We've updated the POM to disable building these JARs by default, but allow builds to reenable them by running mvn -DbuildJars ... (or just build a specific set of JARs by activating appropriate profiles with e.g. mvn -PbuildEncyclopedia ...).
  • When deploying an executable JAR we use separate steps to build/test the code (and create the JAR) and upload it to the project, passing the file(s) between steps as an artifact. This improves the display of build results and simplifies debugging failures, while also providing temporary access (for 1 week) to the artifact(s) through the Pipeline results page, even if the upload step fails.

Regression testing

PR #29 adds regression testing to the execution of ITs. This leverages a combination of Pipeline configuration and the encyclopedia-maven-repo.

Regression testing is configured by changing a dependency version in the POM which determines the reference data artifact (stored in the encyclopedia-maven-repo) that's compared to the results of the current build.

Results of the build (when integration/regression tests are enabled by -DskipITs=false) are packaged and deployed to the maven repository from builds that execute the deploy goal; currently this is only done automatically for tagged versions. When results change due to development of the program it's possible to deploy results from any commit using the custom: deploy-maven-artifacts pipeline. Care must be taken when deploying updated reference data artifacts and/or when changing the reference data dependency version.

Implementation Steps

We plan to progressively implement build and testing features for the EncyclopeDIA project, allowing us to test and develop procedures for the basic elements of the system before moving on to more complex elements.

A rough ordering of steps to take:

  1. DONE -- Develop basic Pipelines configuration in a test repository, just to run tests each time new commits are pushed. Develop basic documentation (this page).
  2. DONE -- Copy basic Pipelines configuration and documentation to the public project repository.

  3. DONE -- Develop configuration to run builds for additional branches and/or pull requests (active by default) as well as further integrations. See for example documentation on merge checks that can prevent merging a PR without it first passing tests. Note that merge checks are set up at the same time as other branch permissions and can interfere with the normal development workflow unless care is taken to add appropriate users and groups to the various roles. Most of this has been implemented as of 2020-08-25. The final part, implemented 2020-10-08 adds a pull-request specific pipeline that executes integration tests and is required to pass before merging the PR.

  4. DONE -- Develop procedures for increased automation of releases, possibly including new branching/tagging approaches.

  5. DONE -- Implement automatic deployment of release-versioned executable and dependency artifacts (likely to the EncyclopeDIA project/repo's "Downloads" section and a public Maven repository, respectively).

  6. DONE -- Implement policies for integration testing in build configurations. This may require cloud storage of test data and cloud compute services for build/test execution.

    Initially the relevant policy (determined with Brian) will be:

    • Unit tests: always
      • Our default build (runs for all commits) runs mvn verify which would execute and check the results of integration tests, but the Maven POM will be configured to skip ITs by default.
    • Integration tests: manually on dev branch, all releases, all PRs
      • We will implement a manually-triggered pipeline that will allow a full build with ITs to be run on any commit.
      • We will modify the deployment pipeline to execute ITs
      • All PRs trigger a pipeline that merges the code and runs ITs. PRs require both the regular branch build (with only unit tests) and the merge-and-IT build to be successful.

    We may revise this as more complex tests become available.

Updated