Definition

A Task in REEF is a unit of work to be executed on an Evaluator. In its simplest form, a Task is merely an object implementing the Task interface which prescribes a single method:

public byte[] call(byte[] input);

From REEF's perspective, a Task is therefore a single threaded method call. It starts when entering the call method. It is a RunningTask while it hasn't returned from it and is a CompletedTask when it has. Should there be an Exception thrown by call(), we call it a FailedTask.

Task identity is established by a user-defined string set inTaskConfiguration.IDENTIFIER. All subsequent task-related events in the Driver will carry that ID. Note that REEF doesn't take any particular precautions to ensure unique Task identifiers. It is up to the application to do so. While technically feasible to assign the same identifier to multiple Tasks, this isn't advised as it makes error handling, debugging and logging unnecessarily hard.

Inputs and outputs of a Task

The return value of the call method will be made available to the Driver as part of the CompletedTask event. Note that it isn't advised to return large values in this fashion, but merely small control flow or status information. Sending large data on this channel creates the risk of overloading the Driver at scale. The networking APIs provided by REEF IO are much better suited for data transmissions than this channel.

The parameter given to the call method is also to be used in a similar fashion: The Driver passes its value as part of the Task submission. It is meant e.g. to convey a restart point for the task. Note that the same functionality can now be better provided by Tang and a constructor parameter.

Communicating between a Task and a Driver

REEF provides some facilities to communicate between a Driver and a Task. These mostly stem from allowing application code to "free-ride" on REEF's control flow channels such as the heartbeat between the Evaluator and the Task.

Sending a message from the Driver to a Task

In order to send information from the Driver to a Task, the RunningTask event provides the method onNext(byte[]). The given byte array will be pushed as soon as possible to the Task, where it is presented to the handler registered viaTaskConfiguration.ON_MESSAGE. Note: If no such handler is bound, the Task will fail upon receiving the message.

Sending a message from a Task to the Driver

REEF maintains a heartbeat between any Evaluator and the Driver. There are two ways by which a heartbeat can be triggered.

  • Upon some schedule (which may also vary at runtime due to load conditions on the Driver), each Evaluator will report its current status to the Driver. This is used by the Driver to maintain health status and load statistics of the Evaluators.
  • Whenever the status of the Evaluator changes (e.g. when a Task completes), a Heartbeat is triggered immediately.

Whenever the Evaluator performs a heartbeat, it will ask the Task whether it has any message to share with the Driver by inquiring the class registered inTaskConfiguration.ON_SEND_MESSAGE. It is wise for that message to be small, as we otherwise run the risk of overwhelming the Driver with heartbeat traffic at scale.

Multithreaded Tasks

Just because REEF views a Task as a method call doesn't restrict the Task to be single threaded. A Task is free to spawn threads in the course of its execution. However, a Task that does so needs to take care of a few considerations:

  • All Threads spawned need to exit before the Task.call() method returns. Otherwise, you run the risk of resource leakage.
  • Exceptions on spawned Threads need to be caught and re-thrown by the Thread.call() method. Before that, all spawned threads need to be shut down, just like during a normal exit of Task.call(). If an exception from an another thread isn't caught, REEF's JVM level exception handler will catch it and declare a FailedEvaluator. This is inefficient, but not technically wrong: The Driver will then have to allocate another Evaluator and try again.
  • No labels