RADICAL-Ensemble Toolkit

RADICAL Ensemble Toolkit (EnTK) is a Python library for developing and executing large-scale ensemble-based workflows. Different from a general-purpose workflow system, EnTK exposes a programming application interface (API) specifically designed to concisely describe applications in which compute tasks are grouped into concurrent or sequential pipelines. Each pipeline is a sequence of stages and each stage is a set of tasks. Tasks can be arbitrary executables or Python functions, requiring single/multi core/GPU/node or MPI/OpenMP. EnTK uses RADICAL-Pilot, enabling the execution of workflows are scale on diverse HPC platforms. Ensemble-based workflows are common in many scientific domains, including drug discovery, molecular modeling, material engineering and climate science.

• Repository: https://github.com/radical-cybertools/radical.entk

• Issues: https://github.com/radical-cybertools/radical.entk/issues

Contents:

• Introduction
  • Overview
  • Intended users
• Ensemble Toolkit
  • Design
  • Implementation
  • Performance
• Installation
  • Installing Ensemble Toolkit
  • Preparing the Environment
  • Troubleshooting
• User Guide
  • Getting Started
  • Adding Tasks
  • Adding Stages
  • Adding Pipelines
  • Adding Shared Data
  • Adding Data
  • Changing Target Machine
  • Profiling
• Examples
  • Pipeline of Ensembles
  • Ensemble of Pipelines
  • Sequence of Workflows
• Advanced Examples
  • Adaptive applications: Task-count
  • Adaptive applications: Task-order
  • Adaptive applications: Task-attribute
• API Reference for Users
  • Application Creation components API
  • Application Manager API
• Exceptions
  • EnTKError
  • EnTKMissingError
  • EnTKTypeError
  • EnTKValueError
• Developer Documentation
  • UML diagrams and other resources
  • API Reference for Developers
  • Unit tests & Integration tests

Note

Please use the following to reference Ensemble Toolkit:

Balasubramanian, Vivek, Matteo Turilli, Weiming Hu, Matthieu Lefebvre, Wenjie Lei, Ryan Modrak, Guido Cervone, Jeroen Tromp, and Shantenu Jha. “Harnessing the power of many: Extensible toolkit for scalable ensemble applications.” In 2018 IEEE international parallel and distributed processing symposium (IPDPS), pp. 536-545. IEEE, 2018.