simplify pipeline system

Issue #80 new
Thomas Gilgenast created an issue

currently, the lib5c pipeline ecosystem is complex

it is expensive to extend and maintain (e.g., parameter values and their defaults must be duplicated in the Task definitions)

it is also difficult to understand (basically nothing can be done unless you first invest in learning luigi, which generally speaking “isn’t worth it“)

we are currently testing ways to automatically convert a set of plain Python functions into both a CLI and a luigi pipeline (without requiring any knowledge of how luigi works); if these tests work out, the lib5c pipeline should eventually be converted to use this system for reduced complexity and increased maintainability/extendablility

this would also solve #66

it is likely that the signatures of top-level API functions might need to change to accomodate this

the priority for this is somewhate reduced because this particular pipeline is not heavily used at the current time; other projects like hiclite stand to gain much more from a pipeline redesign

Comments (0)

  1. Log in to comment