Base TaskManager API

class radical.entk.execman.base.task_manager.Base_TaskManager(sid, rmgr, rts, zmq_info)

A Task Manager takes the responsibility of dispatching tasks it receives from a ‘pending’ queue for execution on to the available resources using a runtime system. Once the tasks have completed execution, they are pushed on to the completed queue for other components of EnTK to thread.

Arguments

rmgr

(ResourceManager) Object to be used to access the Pilot where the tasks can be submitted

Currently, EnTK is configured to work with one pending queue and one completed queue. In the future, the number of queues can be varied for different throughput requirements at the cost of additional Memory and CPU consumption.

_advance(obj, obj_type, new_state, qname)

_setup_zmq(zmq_info)

_sync_with_master(obj, obj_type, qname)

_tmgr(uid, rmgr, zmq_info)

Purpose: Method to be run by the tmgr thread. This method receives

a Task from the ‘pending’ queue and submits it to the RTS. At all state transititons, they are synced (blocking) with the AppManager in the master thread.

Details: The AppManager can re-invoke the tmgr thread with this

function if the execution of the workflow is still incomplete. There is also population of a dictionary, placeholder_dict, which stores the path of each of the tasks on the remote machine.

check_manager()

Purpose: Check if the tmgr thread is alive and running

start_manager()

Purpose: Method to start the tmgr thread. The tmgr function is not to be accessed directly. The function is started in a separate thread using this method.

terminate_manager()

Purpose: Method to terminate the tmgr thread. This method is blocking as it waits for the tmgr thread to terminate (aka join).

RP TaskManager API

class radical.entk.execman.rp.task_manager.TaskManager(sid, rmgr, zmq_info)

A Task Manager takes the responsibility of dispatching tasks it receives from a queue for execution on to the available resources using a runtime system. In this case, the runtime system being used RADICAL Pilot. Once the tasks have completed execution, they are pushed on to another queue for other components of EnTK to access.

Arguments

rmgr

(ResourceManager) Object to be used to access the Pilot where the tasks can be submitted

Currently, EnTK is configured to work with one pending queue and one completed queue. In the future, the number of queues can be varied for different throughput requirements at the cost of additional Memory and CPU consumption.

_process_tasks(task_queue, tmgr)

Purpose: The new thread that gets spawned by the main tmgr thread

invokes this function. This function receives tasks from ‘task_queue’ and submits them to the RADICAL Pilot RTS.

_tmgr(uid, rmgr, zmq_info)

Purpose: This method has 2 purposes: receive tasks from the

‘pending’ queue, start a new thread that processes these tasks and submits to the RTS.

The new thread is responsible for pushing completed tasks (returned by the RTS) to the dequeueing queue. It also converts Tasks into TDs and CUs into (partially described) Tasks. This conversion is necessary since the current RTS is RADICAL Pilot. Once Tasks are recovered from a CU, they are then pushed to the completed queue. At all state transitions, they are synced (blocking) with the AppManager in the master process.

Details: The AppManager can re-invoke the tmgr thread with this

function if the execution of the workflow is still incomplete. There is also population of a dictionary, placeholders, which stores the path of each of the tasks on the remote machine.

_update_resource(pilot, tmgr)

Update used pilot.

start_manager()

Purpose: Method to start the tmgr thread. The tmgr function

is not to be accessed directly. The function is started in a separate thread using this method.

Dummy TaskManager API

class radical.entk.execman.mock.task_manager.TaskManager(sid, rmgr, zmq_info)

A Task Manager takes the responsibility of dispatching tasks it receives from a ‘pending’ queue for execution on to the available resources using a runtime system. Once the tasks have completed execution, they are pushed on to the completed queue for other components of EnTK to process.

Arguments

rmgr

(ResourceManager) Object to be used to access the Pilot where the tasks can be submitted

Currently, EnTK is configured to work with one pending queue and one completed queue. In the future, the number of queues can be varied for different throughput requirements at the cost of additional Memory and CPU consumption.

_process_tasks(task_queue, rmgr)

Purpose: The new thread that gets spawned by the main tmgr process

invokes this function. This function receives tasks from ‘task_queue’ and submits them to the RADICAL Pilot RTS.

_tmgr(uid, rmgr, zmq_info)

Purpose: Method to be run by the tmgr process. This method receives

a Task from the ‘pending’ and submits it to the RTS. Currently, it also converts Tasks into CUDs and CUs into (partially described) Tasks. This conversion is necessary since the current RTS is RADICAL Pilot. Once Tasks are recovered from a CU, they are then pushed to the completed queue. At all state transititons, they are synced (blocking) with the AppManager in the master process.

Details: The AppManager can re-invoke the tmgr process with this

function if the execution of the workflow is still incomplete. There is also population of a dictionary, placeholders, which stores the path of each of the tasks on the remote machine.

start_manager()

Purpose: Method to start the tmgr process. The tmgr function

is not to be accessed directly. The function is started in a separate thread using this method.