cb6ba59·Author: Scott Wiant·Closed by: Scott Wiant·2023-09-08
Description
This PR allows the workflow developer to specify different resource requirements for different tasks via the selection of a resource_queue when adding a node to a workflow. Presently we only have one resource profile so the defaults that trickle through the layers are fine. We are soon going to have an additional profile for high memory tasks. Workflow which have these tasks would need the following specification in order to run on the high memory resources.
from dkist_processing_core import ResourceQueue
from dkist_processing_core import Workflow
from dkist_processing_core import TaskBase
class HighMemTask(TaskBase):
def run(self):
pass # use a lot of memory
workflow = Workflow(...)
workflow.add_node(task=HighMemTask, resource_queue=ResourceQueue.HIGH_MEMORY)
This PR allows the workflow developer to specify different resource requirements for different tasks via the selection of a resource_queue when adding a node to a workflow. Presently we only have one resource profile so the defaults that trickle through the layers are fine. We are soon going to have an additional profile for high memory tasks. Workflow which have these tasks would need the following specification in order to run on the high memory resources.
from dkist_processing_core import ResourceQueue from dkist_processing_core import Workflow from dkist_processing_core import TaskBase class HighMemTask(TaskBase): def run(self): pass # use a lot of memory workflow = Workflow(...) workflow.add_node(task=HighMemTask, resource_queue=ResourceQueue.HIGH_MEMORY)Â