USE6 Workflow Management Systems

Workflow Management Systems are basic tools for reproducible data analysis, especially if many data analyis steps are involved.

Workflow Management Systems cover all data analysis from A to Z, e.g. data preprocessing, quality filtering, analysis and statistical evaluation of processed data. This may include, starting jobs, staging data to compute nodes, running the computations, deleting temporary data, generating publication ready reports, archiving and cleaning work environments.

There are some Workflow Management Systems, such as Nextflow or Snakemake, which are designed to automate this process on HPC Clusters.

• Explain the purpose and benefits of a Workflow Management System.
• Identify and select an appropriate workflow from a workflow catalogue.
• Recall the importance of reproducibility in research and describe how Workflow Management Systems ensure it in HPC environments.
• Configure workflows to run optimally on a particular system (e.g. selecting appropriate queues, parameterising resources, and choosing input files from various file systems).
• Diagnose and resolve errors by identifying failed jobs, viewing logs, and restarting workflows from failed steps.

• USE6.1 Selecting Workflows
• USE6.2 Workflow Configuration
• USE6.3 Workflow Parameterization for HPC Clusters
• USE6.4 Running workflows