Implement job files API.

#327 Merged at 2727f16
  1. John Chilton

Provides a mechanism for remote job execution mechanisms to read and write files on behalf of an "active" jobs.

For a variety of reasons simply providing access to datasets is insuccifient for this - there are working directory files needed for metadata calculation, inputs and outputs may be of intermediate form (task split files for instance), tool files, location files, etc.... Therefore this API endpoint provides access to a jobs view of these files.

Some attempt is made to verify the paths that are written to are valid for the supplied job (they either correspond to the output datasets of the job or the working directory of the job). Authorizing suchs paths for reading is much more difficult and left undone due to the unstructure and arbitrary nature of .loc files.

To implement this securely and with minimal configuration required - single-purpose "job_key"s are generated to authenticate these API calls. Each key allow usage of the API only for a single, active job.

Update LWR job runner to be able to leverage this API to stage files remotely via HTTP for "remote_transfer" actions (added recently to the LWR client). This addition will vastly increase the utility of the recently added option to drive the LWR via message queue (previously staging files could only occur via file system copy - requiring a shared file system between the remote head node the LWR sat on and Galaxy).

Comments (0)