Pipeline API

class radical.entk.Pipeline[source]

A pipeline represents a collection of objects that have a linear temporal execution order. In this case, a pipeline consists of multiple ‘Stage’ objects. Each `Stage_i` can execute only after all stages up to `Stage_(i-1)` have completed execution.

_decrement_stage()[source]: Purpose: Decrement stage pointer. Reset completed flag.

_increment_stage()[source]: Purpose: Increment stage pointer. Also check if Pipeline has completed.

_validate()[source]: Purpose: Validate that the state of the current Pipeline is ‘DESCRIBED’ (user has not meddled with it). Also validate that the current Pipeline contains Stages.

classmethod _validate_entities(stages)[source]

Purpose: Validate whether the argument ‘stages’ is of list of Stage objects

Argument: list of Stage objects

add_stages(value)[source]

Appends stages to the current Pipeline

Argument: List of Stage objects

as_dict()[source]

Convert current Pipeline (i.e. its attributes) into a dictionary

Returns: python dictionary

property completed

Returns whether the Pipeline has completed

Returns: Boolean

property current_stage

Returns the current stage being executed

Returns: Integer

from_dict(d)[source]

Create a Pipeline from a dictionary. The change is in inplace.

Argument: python dictionary
Returns: None

property lock

Returns the lock over the current Pipeline

Returns: Lock object

property luid

Unique ID of the current pipeline (fully qualified). For the pipeline class, his is an alias to uid.

Example

>>> pipeline.luid
pipe.0001

Getter: Returns the fully qualified uid of the current pipeline
Type: String

property name

Name of the pipeline useful for bookkeeping and to refer to this pipeline while data staging. Do not use a ‘,’ or ‘_’ in an object’s name.

Getter: Returns the name of the pipeline
Setter: Assigns the name of the pipeline
Type: String

resume()[source]

Continue execution of paused stages and tasks.

The resume() method can only be called on a suspended pipeline, an exception will be raised if that condition is not met.

The state of a resumed pipeline will be set to the state the pipeline had before suspension.

property stages

Stages of the list

Getter: Returns the stages in the current Pipeline
Setter: Assigns the stages to the current Pipeline
Type: List

property state

Current state of the pipeline

Getter: Returns the state of the current pipeline
Type: String

property state_history

Returns a list of the states obtained in temporal order

Returns: list

suspend()[source]

Pause execution of the pipeline: stages and tasks that are executing will continue to execute, but no new stages and tasks will be eligible for execution until resume() is called.

The suspend() method can not be called on a suspended or completed pipeline, doing so will result in an exeption.

The state of the pipeline will be set to SUSPENDED.

property uid

Unique ID of the current pipeline

Getter: Returns the unique id of the current pipeline
Type: String