HTTPS SSH

CLuBBeR

Cluster Load Balancer for Bioinformatics e-Resources

With the advent of modern day high-throughput technologies, the bottleneck in biological discovery has shifted from the cost of doing experiments to that of analyzing their results. clubber is our automated cluster load balancing system developed for optimizing these “big data” analyses. Its plug-and-play framework encourages re-use of existing solutions for bioinformatics problems. clubber’s goals are to reduce computation times and to facilitate use of cluster computing. The first goal is achieved by automating the balance of parallel submissions across available high performance computing (HPC) resources. Notably, the latter can be added on demand, including cloud-based resources, and/or featuring heterogeneous environments. The second goal of making HPCs user-friendly is facilitated by an interactive web interface and a RESTful API allowing for job monitoring and result retrieval. We used clubber to speed up our pipeline for annotating molecular functionality of metagenomes. We analyzed the Deepwater Horizon oil-spill study data to quantitatively show that that the beach sands have not yet entirely recovered. Further, our analysis of the CAMI-challenge data revealed that microbiome taxonomic shifts do not necessarily correlate with functional shifts. These examples (21 metagenomes processed in 172 minutes) clearly illustrate the importance of clubber in the everyday computational biology environment.

Pre-Requirements

Standalone Python Installation

mandatory:

  • MySQL 5.x
  • Python 3.x

optional:

  • PHP 5.6.x

Docker Image

  • Docker 17.x

Instructions

Docker

CLuBBeR is publicly available on the docker cloud.

https://cloud.docker.com/swarm/bromberglab/repository/docker/bromberglab/clubber/general

Download it from the cloud or let docker pull it automatically by running the docker command:

docker run -d -p 80:80 bromberglab/clubber:latest

Additionally exposing port 3306 enables access to the MySQL database:

docker run -d -p 80:80 -p 3306:3306 bromberglab/clubber:latest

Visit http://localhost/ and follow the first-run instructions.

Docker - Create Image from source

Download the sources and use the docker build command:

wget https://bitbucket.org/bromberglab/bromberglab-clubber/get/master.zip
docker build -t clubber:current .

Standalone Python Installation

Download the sources or clone the github repository:

https://bitbucket.org/bromberglab/bromberglab-clubber/get/master.zip
git@bitbucket.org:bromberglab/bromberglab-clubber.git

Create the MySQL scheme in your database and edit the main config file, than start the daemon:

mysql -h<hostname> -u<user> -D<database> -p < build/clubber_scheme.sql
nano build/bromberglab_suite/python/config/main.conf
build/bromberglab_suite/run.sh bblsuite/hpc/daemon.py start

Setup

Independent on the chosen installation CLuBBeR requires some basic configuration determined throughout the installation process. Once the installation is finished the clubber daemon is automatically started and will run in the background from that point on.

Docker by default launches a webinterface offering several administration options. This webinterface is also available for the standalone isntallation but has to be made available manually using a local running webserver. Both installations offer a webinterface (docker: http://localhost/) and an interactive Terminal ("CluBBeR console") for monitoring and administration.

Registering a cluster

In order to register a cluster with clubber, a valid user:password combination (passwd) or identity file (idrsa) is required along with following parameters: cluster identifier/name cluster hostname job manager type [SGE|SLURM] project base directory scratch directory queuelimit

Usage

Docker Use the webinterface to configure and use clubber.

Standalone Python Installation Use the CluBBeR console to use clubber or add jobs manually to the connected database.