...
1. Container Placement Actions (Move / Restart)
API | registerContainerPlacementActionplaceContainer |
Description | Active Container: Stop container process on source-host and starts it for
StandBy Container: Stop container process on source-host and starts it on:
|
Parameters | uuid: unique identifier a request, populated by the client applicationId: unique identifier of the deployed app for which the action is taken processor-id: Samza resource id of container e.g 0, 1, 2 destination-host: valid hostname / “ANY_HOST” / “STANDBY” request-expiry-timeout: [optional]: timeout for any resource request to the cluster manager |
Status code | CREATED, BAD_REQUEST, ACCEPTED, IN_PROGRESS, SUCCEEDED, FAILED |
Returns | Since this is an ASYNC API nothing is returned, the status of the request can be queried by processorId using |
Failure Scenarios | There are following cases under which a request to place container might fail:
|
...
Note: For supporting canary above parameter list can be easily extended to support the following parameters
Parameters | userapp-version: user application version [optional] samza-version: samza framework version [optional] jvm-args: arbitrary string to be used as jvm arguments [optional] |
...
API | containerStatus |
Description | Gives the status & info of the container placement request, for ex is it running, stopped what control commands are issued on it |
Parameters | processor-id: Samza resource id of container e.g 0, 1, 2 applicationId: unique identifier of the deployed app for which the action is taken uuid: unique identifier a request |
Status code | ACCEPTEDBAD_REQUEST, UNAUTHORIZED |
Returns | Status of the Container placement action |
...
- Control Plane as described plane above the job that allows taking control actions by multiple controllers like Samza Dashboard, Start points controller.
- ContainerPlacementHandler is a stateless handler registered to control plane that translates control dispatches placement actions to invoke Container Placement Service APIs
...
Samza Metastore will provide an API to write to the coordinator stream. One simple way to expose Container Placement API is, Container Placement handler can have a coordinator stream consumer polling control messages from coordinator stream & acting on them. CPH will take actions maintaining some in-memory state with Container Placement Service to not take an action twice. In addition, since control actions are associated with a deployment id, they are automatically invalidated across restarts
Pros | Cons |
|
|
...
Pros | Cons |
|
|
...
- ContainerPlacementHandler is a stateless handler dispatching ContainerPlacementRequestMessages from Metastore to Container Placement Service & ContainerPlacementResponseMessages from Container Placement Service to metastore for external controls to query the status of an action. (PR).
- Metastore used today by in Samza by default is Kafka (coordinator stream) which is used to store configs & container mappings & is log compacted
- ContainerPlacementRequestMessage & ContainerPlacementResponseMessage are maintained in individual namespaces using NamespaceAwareMetaStore
- Key for storing the ContainerPlacementRequestMessage & ContainerPlacementResponseMessage in Metastore is chosen to be processorId (logical container id 0,1,2, etc) because at the worst the messages in metastore will be 2n where n is the size of containers, this choice to ensure job boot up times are not affected due to such messages
- One way to delete stale ContainerPlacementMessages is to delete messages from the previous incarnation in the metastore on job restarts
...
- To expose this API
...
- , a metadata store writer tool can be used. The same tool is going to give Open source users access to Start points APIs.
KV for ContainerPlacementRequestMessages & ContainerPlacementResponseMessage
Key | Value |
processorId | uuid: unique identifier a request, populated by client applicationId: unique identifier of the deployed app for which the action is taken destination-host: valid hostname / “ANY_HOST” / “STANDBY” request-expiry-timeout: [optional]: timeout for any resource request to cluster manager |
Part 2. Container Placement Service
...