Tool Dependency Resolver Plugins (Revision 2)

#228 Declined
Repository
galaxy-central
Branch
default
Author
  1. John Chilton
Reviewers
Description

This pull request includes a lot of new testing which all passes. I have manually verified the behavior of manually installed galaxy dependencies. Additionally, I have ran the tool shed test suite and the only failing tests were the previously outlined failing tests as identified by Greg in comments on pull request #223.

This implements my interpretation of a variety of things discussed in this thread:

http://dev.list.galaxyproject.org/Test-Toolshed-Biopython-package-dependency-Atlas-fails-to-install-Was-Re-UnboundLocalError-local-var-tp4661765p4661879.html

Additionally, it provides a cleaner more expansive solution to the problem outlined in pull request #220.

Usage:

If one wishes to modify how Galaxy resolves requirements in tools one can create a tool resolvers configuration XML file (dependency_resolvers_conf.xml) partly modelled of Nate's work on job configurations.

The XML corresponding roughly to Galaxy's current default resolution would be

<dependency_resolvers>
  <tool_shed_packages />
  <galaxy_packages />
</dependency_resolvers>

New Feature: Galaxy can be modified to favor manually curated env.sh files as I advocated for in the above e-mail thread by changing dependency_resolvers_conf.xml as follows:

<dependency_resolvers>
  <galaxy_packages />
  <tool_shed_packages />
</dependency_resolvers>

New Feature: Galaxy can be configured to fallback on the default version of a dependency if the exact version specific in the tool XML is not present by configuring dependency_resolvers_conf.xml as follows:

<dependency_resolvers>
  <galaxy_packages />
  <tool_shed_packages />
  <galaxy_packages versionless="true" />
</dependency_resolvers>

New Feature: Galaxy can be configured to search multiple directories for packages as outlined below. Parts of this were there in the previous implementation, but there was no way to configure it:

<dependency_resolvers>
  <galaxy_packages />
  <tool_shed_packages />
  <galaxy_packages base_path="/opt/galaxy/legacy/" />
  <!-- base_path defaults to tool_dependency_dir if not specified -->
</dependency_resolvers>

New Feature (sort of): Galaxy can load environment modules (with whatever precedence using modules tag). This is module work is completely untested and is largely here for demonstration purposes. Someone (Simon?) with the intention of actually using these should step up and fix the bugs I am certain are present in this implementation.

<dependency_resolvers>
  <modules />
  <galaxy_packages />
  <tool_shed_packages />
</dependency_resolvers>

By default the modules tag will cause the output of module avail to be searched. To search a specific directory on the filesystem for actual modulefiles the XML (instead of delegating out to module), dependency_resolvers_conf.xml can be defined as follows:

<dependency_resolvers>
  <modules find_by="directory" directory="/opt/Modules/modulefiles" />
  <galaxy_packages />
  <tool_shed_packages />
</dependency_resolvers>

The modules tag also supports the versionless attribute. To first search for a specific version of a module, then fall back to the tool shed and finally fallback to any version of the module that is available, dependency_resolvers_conf.xml could be specified as follows:

<dependency_resolvers>
  <modules />
  <tool_shed_packages />
  <modules versionless="true" />
</dependency_resolvers>

New Feature: Arbitrary dependency resolution code can be added by external developers by just placing a valid implementation into the folder lib/galaxy/tools/deps/resolvers.

This class should conform to the DependencyResolver abstract base class defined in galaxy.tools.deps.resolvers and define a resolve method that returns an object conforming to the Dependency abstract base class defined in the same module.

See ModuleDependencyResolver in lib/galaxy/tools/deps/resolvers/modules.py for a concrete example.

Differences with respect to revision 1.

  • Added test cases to verify that this indeed preserves Galaxy default behavior with respect to tool shed tools with manually installed dependencies. (I had a small note previously that I thought this behavior had changed. It had not.)

  • Added test case to verify tool shed versus Galaxy precedence.

  • Bug fix 'tool_dependency_conf.xml' spellings (thanks to Bjoern).
  • Bug fix related to relative vs. absolute path.
  • Bug fix related to None versus [].
  • Refactored some common code to be shared between dynamic resolves and dynamic job destinations (galaxy.util.submodule).
  • Added prefetch option to module dependency loader.
  • Added test cases and bug fixes for environment module loader.

Comments (5)

  1. Greg Von Kuster

    Ok, sounds good. In case you are not yet aware, the next stable branch is tentatively scheduled to be made on Monday, October 21, so the sooner this is merged, the better. The next Galaxy release is tentatively scheduled for Monday, November 4.