HTTPS SSH

CIAO, Overview!

This is the documentation of the Databiology for Enterprise Continuous Integration / Application Onboarding (CIAO) platform.

See the following files for detailed documentation of the CIAOfile syntax and CIAO tools:

First Steps

Initial Setup

  1. Request developer account on the lab.databiology.net DBE instance. As part of the onboarding process, each CIAO application developer will receive:

    • credentials (username and password) for lab.databiology.net
    • credentials (username and password) for docs.databiology.net
    • a short org name that uniquely identifies their organization, usually an acronym or abbreviation of the organisations name. The org name must be used as part of the name of all applications developed by members of this organisation.
    • An invitation to gain read access to the app/dbio/cowsay CIAO application example repository on bitbucket.org.
  2. Login to lab.databiology.net and download the CIAO development tools package.
    The CIAO development tools package is named dbe_ciao-X.Y.Z-py2-none-any.whl, where X.Y.Z is the version number. The current version of the package can be found in the Comments section of Project 3 - CIAO Developer Project.

  3. Install the CIAO development tools package on developer workstations:
    pip install /path/to/dbe_ciao-X.Y.Z-py2-none-any.whl

Creating A New Application

The following describes the steps required to setup a new application. The application source code will be hosted in a bitbucket.org repository. The application docker image will be built on bitbucket.org pipelines and deployed to repo.lab.databiology.net.

  1. Decide on a name for the new application.
    The full name should be of the form app/org/name, where org is the organizations short name received during the onboarding process and name is a short and unique name for this application, eg app/tyrell/nexus6.

  2. Fork the CIAO application template repository.
    The CIAO application template repository is located at https://bitbucket.org/databiology/app-dbio-template. For repositories hosted on bitbucket.org, it is recommended to name the new repository the same as the full application name, ie app/org/name.

  3. Setup credentials for lab.databiology.net.
    Setup the following environment variables in the Environment Variables settings of the new application repository at bitbucket.org:

    • CIAO_REGISTRY: set to repo.lab.databiology.net
    • CIAO_USERNAME: set to the username received during the onboarding process
    • CIAO_PASSWORD: set to the password associated with the username
      • mark the Secured checkbox for this variable to hide the password from casual view
      • update this variable manually whenever the password on lab.databiology.net changes.
  4. Enable the Pipeline feature for the source code repository on bitbucket.org

  5. Clone the new application repository to the developer workstation:
    git clone ssh://git@bitbucket.org/account/app-org-name.git;

  6. Keep the following files verbatim from the upstream app/dbio/template repository:

    • Makefile
    • bitbucket-pipelines.yml
  7. Edit the following files to reflect the details of the new application:

    • CIAOfile:
      • edit the NAME, LABEL, AUTHOR and DESCRIPTION fields to describe the new application
      • add PARAMETERS section as required
    • Dockerfile:
      • add steps as required by the new application
    • README.md:
      • describe the new application
  8. Commit the changes
    git add Dockerfile CIAOfile; git commit -m 'initial commit';

  9. Tag the commit with an initial version:
    git tag 0.0.1;

  10. Push the initial commit and tags to bitbucket.org:
    git push --tags origin master;

Creating Complex Applications

An application is considered a complex application if any of the following conditions apply:

  • the application provides a user interface for interactive sessions
  • the application requires additional services, such as MySQL or PostgreSQL
  • the application uses clustering middleware such as SGE, LSF or Spark for parallel computation

To create a complex application, the following steps are required in addition to the steps described above:

  1. Create a compose.yml file in the root of your source code repository.

  2. Edit the compose.yml file to describe the requirements of the application.

  3. Add and commit the compose.yml file to the source code repository.

The CIAO Development Cycle

Overview

The CIAO development cycle consists of the following steps:

  1. edit application source code, commit to git
  2. build application image on bitbucket.org
  3. test application image on lab.databiology.net
  4. in the absence of errors, tag commit as release version
  5. otherwise go back to #1

Detailed Building Steps

In detail, the following steps are required:

  1. Edit Dockerfile, CIAOfile and/or any source code of the application

  2. Commit the changes to the local git working tree:
    git add Dockerfile CIAOfile src/*; git commit -m 'message describing the changes';

  3. Push the new commit with tags to bitbucket.org:
    git push --tags origin master

  4. Inspect the bitbucket.org pipeline of the applications source repository.
    The pipeline will automatically start building the new application image and deploy the image to repo.lab.databiology.net if successful. A successful build will contain the following line at the end of the output of the make deploy pipeline stage:

``` ### Deployed Image as repo.lab.databiology.net/app/org/name:1.0.0

```

  1. Login to lab.databiology.net and inspect the new application image version.
    Click on Workunits->Analyze->ApplicationName. Make sure that the Version field of the executable associated with the application contains the identifier that was deployed by the pipeline.

  2. Run a workunit with the new application image.
    Click on Run in the application details page, then select a fitting set of input resources, storage, submitter and parameter in the following screens, then click Save to start the workunit.

  3. Verify correct application behaviour.
    If no problems are found, the workunit should finish cleanly, indicated by Status available. The following services are available to investigate a running workunit in case of errors. The specific URLs are shown in the External URL section of the workunit details page.

    • Container Logs
      The output (stdout/stderr) of the workunit container is available under https://wuXXXjobYYY-shell.lab.databiology.io/
    • Container Statistics
      Usage statistics (cpu/memory/io) of the workunit container is available under https://wuXXXjobYYY-stats.lab.databiology.io/
    • Container Shell
      An interactive shell within the workunit container is available under https://wuXXXjobYYY-shell.lab.databiology.io/
  4. In the absence of errors, tag the commit as a release version.
    To mark the most recent commit as version X.Y.Z, run git tag X.Y.Z. Run make info to verify that the tag is recognized correctly, then push the new tag to bitbucket.org:
    git push --tags origin master;
    This will result in the same image being built and deployed once more, tagged with the X.Y.Z release version.

  5. repeat

Names and Versions

An application image is identified by concatenation of its name and version. The name is taken from the NAME field in the ABOUT section of the CIAOfile. The version is taken from the tag of the git commit. For commits that are not tagged directly, the git description is used as the docker image tag instead.

An image that should be released for distribution or use in scientific research must be tagged with a release version number. These images are identified by the following form:app/org/name:1.0.0

During the application development process, application images need to be built and run for debugging. These work-in-progress images should never be distributed or used for scientific research. They should not be tagged with a version number and are identified by the following form: app/org/name:1.0.0-1-g78257e2-000

The version string is displayed by these components:

  • The output of git describe --tags in the source code repository
  • All output of the CIAO build pipeline, especially the make info stage
  • The Version field of the DBE Executable Object on www.lab.databiology.net

The CIAO development tools ensure that each docker image is tagged with the same version that the corresponding git commit was tagged with. Hence, each application image can be mapped to the exact commit in the application source code repository that the image was built from. This allows developers, operators and researchers to identify the specific application version used in running or building, and to identify the specific git commit corresponding to it.

It is the responsibility of the application developer to tag commits with release version numbers, and to make sure that no duplicate or errorenous versions are used.

Application Repository Contents

Each Databiology CIAO Application source code repository must contain the following files in the top level directory:

  • CIAOfile
    The CIAOfile describes the CIAO application. It specificies all aspects of the application that need to be known at workunit run time, such as parameters displayed to the user and the references required by the application. See doc/Example.CIAOfile and doc/Annotated.CIAOfile for a description of the contents.

  • Dockerfile
    The Dockerfile describes the steps required to build the application docker image. See docker documentation for a description of the Dockerfile syntax. All CIAO application images should be based on one of the base application images provided by Databiology, eg: FROM app/dbio/base:1.0.0

  • Makefile
    The Makefile provides the required functionality to build and deploy the application. This file should be copied verbatim from the app/dbio/template repository.

  • bitbucket-pipelines.ylm
    The bitbucket-pipelines.yml file describes the steps to build and deploy the application using the pipeline service at bitbucket.org. This file should be copied verbatim from the app/dbio/template repository.

In addition to the above, each Complex CIAO Application source code repository must contain the following files in the top level directory: