...
Discussion thread: <link to mailing list DISCUSS thread>
Released: TBD
Problem
Samza operates in a multi-tenant environment with cluster managers like Yarn and Mesos where a single host can run multiple Samza containers. Often due to soft limits configured for cluster managers like Yarn and no notion of dynamic workload balancing in Samza a host lands in a situation where it is underperforming and it is desired to move one or more containers from that host to other hosts. Today this is not possible without affecting other jobs on the hot host or restarting the affected job manually. In other use cases like resetting checkpoints of a single container or supporting canary or rolling bounces the ability to restart a single or a subset of containers without restarting the whole job is highly desirable.
...
Solution 1. Write locality to Coordinator Stream (Metastore) and restart job [Rejected]
Pros | Cons |
|
|
Solution 2. Container Placement Service [Accepted]
...
1. Container Placement Actions (Move / Restart)
API | registerContainerPlacementAction |
Description | Active Container: Stop container process on source-host and starts it for
StandBy Container: Stop container process on source-host and starts it on:
|
Parameters | uuid: unique identifier a request, populated by the client applicationId: unique identifier of the deployed app for which the action is taken processor-id: Samza resource id of container e.g 0, 1, 2 destination-host: valid hostname / “ANY_HOST” / “STANDBY” request-expiry-timeout: [optional]: timeout for any resource request to the cluster manager |
Status code | CREATED, BAD_REQUEST, ACCEPTED, IN_PROGRESS, SUCCEEDED, FAILED |
Returns | Since this is an ASYNC API, the status can be queried by processorId using |
Failure Scenarios | There are following cases under which a request to place container might fail:
|
Note: For supporting canary above parameter list can be easily extended to support the following parameters
Parameters | user-version: user application version [optional] samza-version: samza framework version [optional] jvm-args: arbitrary string to be used as jvm arguments [optional] |
2. Container Status
API | containerStatus |
Description | Gives the status & info of the container placement request, for ex is it running, stopped what control commands are issued on it |
Parameters | processor-id: Samza resource id of container e.g 0, 1, 2 applicationId: unique identifier of the deployed app for which the action is taken uuid: unique identifier a request |
Status code | ACCEPTED, UNAUTHORIZED |
Returns | Status of the Container placement action |
2. Enable & Disable StandBy [Stretch]
API | controlStandBy |
Description | Starts or Stops a standBy container for the active container |
Parameters | processor-id: Samza resource id of container e.g 0, 1, 2 |
Status code | ACCEPTED, UNAUTHORIZED |
Architecture
For implementing a scalable container placement control system, the proposed solution is divided into two parts:
...