This is the documentation of the Databiology for Enterprise Continuous Integration / Application Onboarding (CIAO) platform.
See the following files for detailed documentation of the CIAOfile syntax and CIAO tools:
- doc/Example.CIAOfile - Briefly showcases all elements supported within a CIAOfile.
- doc/Annotated.CIAOfile.md - Explains in detail each of the elements supported within a CIAOfile.
- _doc/Example.compose.yml - Briefly showcases all elements supported within a compose file.
- doc/Annotated.compose.yml.md - Explains in detail each of the elements supported within a compose file.
- doc/make.help - Describes the targets provided by the CIAO Master Makefile.
- doc/ciao.help - Describes the operations provided by the ciao command line tool.
Request developer account on the lab.databiology.net DBE instance. As part of the onboarding process, each CIAO application developer will receive:
- credentials (username and password) for lab.databiology.net
- credentials (username and password) for docs.databiology.net
- a short
orgname that uniquely identifies their organization, usually an acronym or abbreviation of the organisations name. The
orgname must be used as part of the name of all applications developed by members of this organisation.
- An invitation to gain read access to the
app/dbio/cowsayCIAO application example repository on bitbucket.org.
Login to lab.databiology.net and download the CIAO development tools package.
The CIAO development tools package is named
dbe_ciao-X.Y.Z-py2-none-any.whl, where X.Y.Z is the version number. The current version of the package can be found in the Comments section of Project 3 - CIAO Developer Project.
Install the CIAO development tools package on developer workstations:
pip install /path/to/dbe_ciao-X.Y.Z-py2-none-any.whl
Creating A New Application
The following describes the steps required to setup a new application. The application source code will be hosted in a bitbucket.org repository. The application docker image will be built on bitbucket.org pipelines and deployed to repo.lab.databiology.net.
Decide on a name for the new application.
The full name should be of the form
orgis the organizations short name received during the onboarding process and
nameis a short and unique name for this application, eg app/tyrell/nexus6.
Fork the CIAO application template repository.
The CIAO application template repository is located at
https://bitbucket.org/databiology/app-dbio-template. For repositories hosted on bitbucket.org, it is recommended to name the new repository the same as the full application name, ie
Setup credentials for lab.databiology.net.
Setup the following environment variables in the Environment Variables settings of the new application repository at bitbucket.org:
CIAO_REGISTRY: set to
CIAO_USERNAME: set to the username received during the onboarding process
CIAO_PASSWORD: set to the password associated with the username
- mark the Secured checkbox for this variable to hide the password from casual view
- update this variable manually whenever the password on lab.databiology.net changes.
Enable the Pipeline feature for the source code repository on bitbucket.org
Clone the new application repository to the developer workstation:
git clone ssh://firstname.lastname@example.org/account/app-org-name.git;
Keep the following files verbatim from the upstream
Edit the following files to reflect the details of the new application:
- edit the
DESCRIPTIONfields to describe the new application
PARAMETERSsection as required
- edit the
- add steps as required by the new application
- describe the new application
Commit the changes
git add Dockerfile CIAOfile; git commit -m 'initial commit';
Tag the commit with an initial version:
git tag 0.0.1;
Push the initial commit and tags to bitbucket.org:
git push --tags origin master;
Creating Complex Applications
An application is considered a complex application if any of the following conditions apply:
- the application provides a user interface for interactive sessions
- the application requires additional services, such as MySQL or PostgreSQL
- the application uses clustering middleware such as SGE, LSF or Spark for parallel computation
To create a complex application, the following steps are required in addition to the steps described above:
compose.ymlfile in the root of your source code repository.
compose.ymlfile to describe the requirements of the application.
Add and commit the
compose.ymlfile to the source code repository.
The CIAO Development Cycle
The CIAO development cycle consists of the following steps:
- edit application source code, commit to git
- build application image on bitbucket.org
- test application image on lab.databiology.net
- in the absence of errors, tag commit as release version
- otherwise go back to #1
Detailed Building Steps
In detail, the following steps are required:
CIAOfileand/or any source code of the application
Commit the changes to the local git working tree:
git add Dockerfile CIAOfile src/*; git commit -m 'message describing the changes';
Push the new commit with tags to bitbucket.org:
git push --tags origin master
Inspect the bitbucket.org pipeline of the applications source repository.
The pipeline will automatically start building the new application image and deploy the image to repo.lab.databiology.net if successful. A successful build will contain the following line at the end of the output of the
make deploypipeline stage:
``` ### Deployed Image as repo.lab.databiology.net/app/org/name:1.0.0
Login to lab.databiology.net and inspect the new application image version.
Click on Workunits->Analyze->ApplicationName. Make sure that the Version field of the executable associated with the application contains the identifier that was deployed by the pipeline.
Run a workunit with the new application image.
Click on Run in the application details page, then select a fitting set of input resources, storage, submitter and parameter in the following screens, then click Save to start the workunit.
Verify correct application behaviour.
If no problems are found, the workunit should finish cleanly, indicated by Status available. The following services are available to investigate a running workunit in case of errors. The specific URLs are shown in the External URL section of the workunit details page.
- Container Logs
The output (stdout/stderr) of the workunit container is available under
- Container Statistics
Usage statistics (cpu/memory/io) of the workunit container is available under
- Container Shell
An interactive shell within the workunit container is available under
- Container Logs
In the absence of errors, tag the commit as a release version.
To mark the most recent commit as version X.Y.Z, run
git tag X.Y.Z. Run
make infoto verify that the tag is recognized correctly, then push the new tag to bitbucket.org:
git push --tags origin master;
This will result in the same image being built and deployed once more, tagged with the X.Y.Z release version.
Names and Versions
An application image is identified by concatenation of its name and version. The name is taken from the
NAME field in the
ABOUT section of the CIAOfile. The version is taken from the tag of the git commit. For commits that are not tagged directly, the git description is used as the docker image tag instead.
An image that should be released for distribution or use in scientific research must be tagged with a release version number. These images are identified by the following form:
During the application development process, application images need to be built and run for debugging. These work-in-progress images should never be distributed or used for scientific research. They should not be tagged with a version number and are identified by the following form:
The version string is displayed by these components:
- The output of
git describe --tagsin the source code repository
- All output of the CIAO build pipeline, especially the
- The Version field of the DBE Executable Object on www.lab.databiology.net
The CIAO development tools ensure that each docker image is tagged with the same version that the corresponding git commit was tagged with. Hence, each application image can be mapped to the exact commit in the application source code repository that the image was built from. This allows developers, operators and researchers to identify the specific application version used in running or building, and to identify the specific git commit corresponding to it.
It is the responsibility of the application developer to tag commits with release version numbers, and to make sure that no duplicate or errorenous versions are used.
Application Repository Contents
Each Databiology CIAO Application source code repository must contain the following files in the top level directory:
CIAOfiledescribes the CIAO application. It specificies all aspects of the application that need to be known at workunit run time, such as parameters displayed to the user and the references required by the application. See doc/Example.CIAOfile and doc/Annotated.CIAOfile for a description of the contents.
Dockerfiledescribes the steps required to build the application docker image. See docker documentation for a description of the Dockerfile syntax. All CIAO application images should be based on one of the base application images provided by Databiology, eg:
Makefileprovides the required functionality to build and deploy the application. This file should be copied verbatim from the app/dbio/template repository.
bitbucket-pipelines.ymlfile describes the steps to build and deploy the application using the pipeline service at bitbucket.org. This file should be copied verbatim from the app/dbio/template repository.
In addition to the above, each Complex CIAO Application source code repository must contain the following files in the top level directory: