Tasks are simply function with the following properties:
- They are stateless, meaning that given the same input, they always return the same output.
- All their input types are specified using type hints.
- Their output type(s) are also specified using type hints.
In addition, the function must be decorated with one of the Task decorators provided by smttask.
Tasks in this directory broadly fall into two categories:
Tasks which simply bind parameters
: These are used to avoid storing the output of a function in a task description. For example, the ~sinnfull.tasks.base.CreateSyntheticDataset Task takes a succinct set of random initialization keys for the model parameter and simulator and returns a complete data set. Since there is often little point in recording these tasks, they are generally ~smttask.MemoizedTask. They are used as inputs for recorded tasks.
Tasks which do actual work
: These tasks may take substantial computational time, and so are generally ~smttask.RecordedTasks. Their inputs and outputs are recorded to disk every time they are run; on subsequent runs with the same input, the execution is skipped and the output retrieved from disk.
The proposed structure is to place data access, model creation and optimization tasks within the base module, and analysis tasks in the analysis. For convenience, all tasks are imported into __init__.py – this allows them to be imported elsewhere in the project as from sinnfull.tasks import <taskname>.
Creating a new task is quite straightforward, but it may still be helpful to start from one of those included in here.py.
Tasks can be used as inputs to other tasks: this makes it easy to break down complex analysis sequences into individual tasks, with optional caching of intermediate results if the corresponding tasks are recorded. We call a sequence of tasks a workflow. Scripts which construct workflows from individual tasks are placed in the workflows directory.
Since there are no special language or convention for combining tasks, and everything is done in Python, one can use arbitrary code to construct workflows. For example, the Optimize_WF_template uses a complex validation function to ensure that a model's stationary_stats method is compatible with assumptions of the workflow. Since this test is run when the workflow is created, rather than when it is run, errors are caught much sooner.