1. CCG, Murdoch University
  2. awesomeness
  3. yabi

Wiki

Clone wiki

yabi / Home

Yabi

Workflows made easy

A sophisticated online research environment for Grid, High Performance and Cloud computing

About

Yabi provides users with an intuitive, easy to use, abstraction of compute and data environments. Developed at the Centre for Comparative Genomics (CCG), Yabi has been deployed across a diverse set of scientific disciplines and high performance computing environments. The CCG has a demo account available.

Citation

Hunter AA, Macgregor AB, Szabo TO, Wellington CA and Bellgard MI, Yabi: An online research environment for Grid, High Performance and Cloud computing, Source Code for Biology and Medicine 2012, 7:1 doi:10.1186/1751-0473-7-1 Published: 15 February 2012 Source Code for Biology and Medicine

Screencasts

YouTube Yabi Screencasts

Licence

GNU Affero GPL v3. Please contact the Centre for Comparative Genomics if you require a licence other than GPL for legal or commercial reasons.

Latest releases

9.9.2 (8th April 2016)

  • Bug fix release.

9.9.1 (17th February 2016)

  • Custom command for adding yabi users.

9.8.5 (11th February 2016)

  • Bug fix release - fixing upstream bug in pykerberos.

9.8.4 (20th January 2016)

  • Bug fix release - fixing a few S3 related bugs.

9.8.3 (20th January 2016)

  • Fixes a problem in yabish that prevented using it via HTTPS.

9.8.2 (19th January 2016)

  • Upgraded Django (to include latest security fix) and other dependencies.

9.8.1 (16th November 2015)

  • Bug fix release - fixing workflow submit error introduced in 9.8.0.

9.8.0 (12th October 2015)

  • CSRF and Clickjacking protection.

9.7.0 (7th September 2015)

  • S3 upload and other improvements.

9.6.0 (17th August 2015)

  • Upgrade to latest Django (1.8.3) and other small improvements.

9.5.1 (7th July 2015)

  • Bugfix release to address a local recursive copy problem on SFTP Backends.

9.5.0 (26th June 2015)

  • Support for Univa Grid Engine and Kerberos. Sharing workflows between users on a single installation.

9.4.0 (3rd February 2015)

  • Support for SLURM execution backend

9.3.0 (15th January 2015)

  • Select multiple files to upload.

9.2.0 (19th November 2014)

  • Task per user limit on Backends and other improvements.

9.1.2 (24th September 2014)

  • Unicode filename support, other improvements and bug fixes.

9.0.1 (3rd September 2014)

  • Ability to use existing tools in multiple backends, OpenStack support for Dynamic Backends.

8.0.0 (14 August 2014)

  • Create backends dynamically prior to execution.

7.2.7 (31th July 2014)

  • Save workflows without submitting.

7.2.6 (26th June 2014)

  • Allow tool parameters to be marked as sensitive.

7.2.5 (20th June 2014)

  • Updated production puppet modules and bug fixes.

7.2.4 (30th May 2014)

  • Keep file arguments when Reusing Workflows.

7.2.3 (8th May 2014)

  • Download directories as tarball, credential administration improvements, tool parameter ordering in frontend.

7.2.2 (17th April 2014)

  • Swift Backend, improved logging, tools loading optimisation.

7.2.1 (3rd April 2014)

  • Cleanup and bug fixes.

7.2.0 (24th March 2014)

  • Python 2.7 in development and RPM, Celery upgrade 3.1.9, file operations on different celery queue and other improvements and fixes.

7.1.10 (28th February 2014)

  • Cleanup and bug fixes.

7.1.9 (14th February 2014)

  • Frontend improvements, upgrade of boto and paramiko and bug fixes.

7.1.8 (30th January 2014)

  • Optimise SFTP copying of large number of small files and bug fixes.

7.1.7 (16th January 2014)

  • Preserving the access and modification times of directories and files when copying and bug fixes.

7.1.6 (21st December 2013)

  • Security release for issues reported by Leif Nixon, National Supercomputer Centre - Swedish National Infrastructure for Computing.

7.1.5 (11th December 2013)

  • This release contains bug fixes and code clean-up.

7.1.4 (2nd December 2013)

  • This release adds support for Job Arrays and dynamic environment variables for command submission template.

7.1.3 (13th November 2013)

  • This release contains bug fixes and internal API changes.

7.1.2 (06th November 2013)

  • This release adds abort capability for all the supported execution Backends.

7.1.1 (17th October 2013)

  • This release changes the SSH Execution backend to detach when submitting a job and poll for completion. In previous releases job submission was waiting for the job to complete. LCopy and LocalLink functionality has been reimplemented for the new Celery based backend. Workflows are now abortable from the admin interface (Note: currently remote execution abort is supported only for SSH and Local Execution Backend. For the other Backends the Workflow can be aborted, but running jobs on servers won't be stopped. Support to abort jobs on all the other Backends will be added in a following release)

7.1.0 (26th September 2013)

  • A major Django upgrade (1.3 to 1.5), re-instate S3 backend which was removed to ship 7.0.0 and some minor bugs from 7.0.0. Existing installations will need to take care with any local Django settings due to upgrade to Django 1.5.

7.0.0 (18th September 2013)

  • This major release comprises a complete re-write of the Yabi backend. The previous Twisted code base has been replaced with a message queue based design using Celery and RabbitMQ. A number of significant issues affecting the stability and scalability of the previous code base have been addressed. The readability, maintainability and overall quality of the code is also much improved.
  • If you experience migration issues with Kombu migrations such as DatabaseError: (1050, "Table 'djkombu_queue' already exists"), it is most likely due to the introduction of South in newer versions of Kombu. The solution is to: sudo yabiadmin migrate kombu.transport.django 0001 --fake and then re-run migrations.

6.15.1 (18th June 2013)

  • Final release in the 6.x.x series. 7.x.x addresses stability problems with significant architectural changes.

Documentation

Documentation can be found at https://yabi.readthedocs.org/

Running Yabi for developers

See the readme.txt in the source for a guide to getting yabi running for developers.

Email Yabi Project

YabiProject

Updated