9.2.2. Pipeline API

class radical.entk.Pipeline[source]

A pipeline represents a collection of objects that have a linear temporal execution order. In this case, a pipeline consists of multiple ‘Stage’ objects. Each `Stage_i` can execute only after all stages up to `Stage_(i-1)` have completed execution.

_decrement_stage()[source]

Purpose: Decrement stage pointer. Reset completed flag.

_increment_stage()[source]

Purpose: Increment stage pointer. Also check if Pipeline has completed.

_validate()[source]

Purpose: Validate that the state of the current Pipeline is ‘DESCRIBED’ (user has not meddled with it). Also validate that the current Pipeline contains Stages.

classmethod _validate_entities(stages)[source]

Purpose: Validate whether the argument ‘stages’ is of list of Stage objects

Argument:list of Stage objects
add_stages(value)[source]

Appends stages to the current Pipeline

Argument:List of Stage objects
completed

Returns whether the Pipeline has completed

Returns:Boolean
current_stage

Returns the current stage being executed

Returns:Integer
from_dict(d)[source]

Create a Pipeline from a dictionary. The change is in inplace.

Argument:python dictionary
Returns:None
lock

Returns the lock over the current Pipeline

Returns:Lock object
luid

Unique ID of the current pipeline (fully qualified). For the pipeline class, his is an alias to uid.

example:
>>> pipeline.luid
pipe.0001
Getter:Returns the fully qualified uid of the current pipeline
Type:String
name

Name of the pipeline useful for bookkeeping and to refer to this pipeline while data staging. Do not use a ‘,’ or ‘_’ in an object’s name.

Getter:Returns the name of the pipeline
Setter:Assigns the name of the pipeline
Type:String
resume()[source]

Continue execution of paused stages and tasks.

  • The resume() method can only be called on a suspended pipeline, an exception will be raised if that condition is not met.
  • The state of a resumed pipeline will be set to the state the pipeline had before suspension.
stages

Stages of the list

Getter:Returns the stages in the current Pipeline
Setter:Assigns the stages to the current Pipeline
Type:List
state

Current state of the pipeline

Getter:Returns the state of the current pipeline
Type:String
state_history

Returns a list of the states obtained in temporal order

Returns:list
suspend()[source]

Pause execution of the pipeline: stages and tasks that are executing will continue to execute, but no new stages and tasks will be eligible for execution until resume() is called.

  • The suspend() method can not be called on a suspended or completed pipeline, doing so will result in an exeption.
  • The state of the pipeline will be set to SUSPENDED.
to_dict()[source]

Convert current Pipeline (i.e. its attributes) into a dictionary

Returns:python dictionary
uid

Unique ID of the current pipeline

Getter:Returns the unique id of the current pipeline
Type:String