View source
default (8a9f23db98a4)
  • Contributors
    1. Loading...
Author Commit Message Date Builds
250+ commits behind default.
John Chilton
Allow ComputeEnvironment to rewrite 'arbitrary' paths. Previous changes enabled targetted rewriting of specific kinds of paths - working directory, inputs, outputs, extra files, version path, etc.... This change allows rewriting remaining 'unstructured' paths - namely data indices. Right now tool evaluation framework uses this capability only for SelectParameter values and fields - which is where these paths will be for data indices. Changeset lays out the recipe for doing this and the functionality could easily be extended for arbitrary parameters or other specific kinds of inputs. The default ComputeEnvironment does not rewrite any paths obviously, but the abstract base c…
John Chilton
Refactor wrapping values in build_param_dict. Split wrap_values into two methods - a reusable method for walking the tool param tree that takes in the second method which actually wraps the values at a given "leaf".
John Chilton
Refactor build_param_dict into smaller methods.
John Chilton
Modify metadata command generation to allow different working directory path remotely. Utilize in LWR to fixup remote metadata generation when command rewriting is disabled.
John Chilton
Touch-up formatting of datatypes/metadata.py for clarity of subsequent changeset.
John Chilton
Rework version string handling for ComputeEnvironment abstraction. Add version_path method to ComputeEnviornment interface - provide default implementation delegating to existing JobWrapper method as well as LwrComputeEnviornment piggy backing on recently added version command support. Build full command to do this in JobWrapper.prepare so that ComputeEnviornment is available and potentially remote path can be used.
John Chilton
Extend LwrComputeEnvironmet to handle rewriting extra_files_path.
John Chilton
Allow 'false_path' style replacing of extra files paths. Updates DatasetFilenameWrapper and DatasetPath to allow this.
John Chilton
Move false_path logic out of tool evaluation code and into cheetah wrapper. This should ease replacing extra_files_path in subsequent commits.
John Chilton
Add rewrite_parameters (defaults to False) option to LWR runner. When false the current behavior of the LWR is maintained and cheetah templates are evaluated as if the command will be submitted locally and then the LWR will rewrite the command and config files and transfer files as needed. If this option is set to True, an instance of the new class LwrComputeEnvironment will be passed along to job_wrapper.prepare allowing the LWR runner to instruct with the tool evaluation process to use the remote paths for input, output, and config files during evaluation. This option feels less hacky - I am skeptical there are tools that will work this way that would not have worked the other way…
John Chilton
Introduce dataset_path_rewrites to slightly simplify tools/evaluation.py.
John Chilton
Rework interface between jobs and tools. Pull code out of job wrapper and tool for building and evaluating against template environments and move them into a new ToolEvaluator class (in galaxy/tools/evalution.py). Introduce an abstraction (ComputeEnvironment) for various paths that get evaluated that may be different on a remote server (inputs, outputs, working directory, tools and config directory) and evaluate the template against an instance of this class. Created a default instance of this class (SharedComputeEnvironment). The idea will be that the LWR should be able to an LwrComputeEnvironment and send this to the JobWrapper when building up job inputs - nothing in this commit is LWR specific though so other runners should be able to remotely stage jobs using other mechanisms as well.… This commit adds extensive unit tests of this tool evaluation - testing many different branches through the code, with and without path rewriting, testing job hooks, config files, testing the cheetah evaluation of simple parameters, conditionals, repeats, and non-job stuff like $__app__ and $__root_dir__. As well as a new test case class for JobWrapper and TaskWrapper - though this just tests the …
John Chilton
Move cheetah wrappers into own module... For clarity - more work toward reduces tools/__init__.py to a more managable size. This also includes an initial suite of test cases for these wrappers - testing simple select wrapper, select wrapper with file options, select wrapper with drilldown widget, raw object wrapper, input value wrapper, and the dataset file name wrapper with and without false paths.
John Chilton
Eliminate input (hence all) path rewriting in TaskWrapper. Extend concept of false paths to apply to inputs as well, use the rewrites implemented in TaskPathRewriter.
John Chilton
Refactor DatasetPath out into newer galaxy.jobs.datasets module. Add unit test for DatasetPath class.
John Chilton
Reorganize tool dataset wrapper instantiation for future extension. This changeset should change how anything functions, but will make subsequent changes less dramatic.
John Chilton
Eliminate output path rewriting in TaskWrapper. Rework task splitting to rewrite paths before cheetah evalulation not after. Utilizing DatasetPath.false_path mechanisms implemented for config.outputs_to_working_directory. I think this could be made less convoluted if the combination of outputs_to_working_directory and Task splitting were optimized in some way - but this changeset keeps the behavior of that the same (copying twice) just to isolate changes to how these paths are evaluated in Cheetah.
John Chilton
Rework logic in tools related to false paths. Basically, if a false path is set trust it, don't explicitly check app.config.outputs_to_working_directory. The hope here is that this logic can then be used in other scenarios where a false_path might be setup at the layers (e.g. job_runner) above tool.
John Chilton
Reduce code duplication between JobWrapper and TaskWrapper. Create shared method for building up param dict.
John Chilton
TaskWrapper - utilize user_template_environment like JobWrapper.
John Chilton
Reduce code duplication between JobWrapper and TaskWrapper.