THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Key for storing the ContainerPlacementRequestMessage & ContainerPlacementResponseMessage in Metastore is chosen to be UUID + "." + messageType(ContainerPlacementResponseMessage or ContainerPlacementRequestMessage). Value will be payload container ContainerPlacementRequestMessage & ContainerPlacementResponseMessage. Messages are written and read to the Metastore through the MetadataStore abstraction. Since the metastore is eventually consistent, duplicate messages are required to be handled by ContainerPlacementService.
ContainerPlacementRequestMessage:
| Key | Value | Field Description | Field Type |
|---|---|---|---|
| "UUID.subType" | uuid | Unique identifier of a response message | Required |
processorId | Logical processor id 0,1,2 of the container | Required | |
| deploymentId | Unique identifier for a deployment | Required | |
| subType | Type of message here: ContainerPlacementResponseMessage | Required | |
| destinationHost | Destination host where the container is desired to be moved | Required | |
| statusCode | Status of the current action | Required | |
| responseMessage | Response message in conjunction to status | Required | |
| timestamp | The timestamp of the response message | Required | |
| requestExpiry | Eequest expiry which acts as a timeout for any resource request to cluster resource manager | Optional |
Sample KV
| Key | Value |
|---|---|
[1,"samza-place-container-v1","f068175b-c9b6-4f34-982b-ecb5619f21de.ContainerPlacementRequestMessage"] | {"processorId":"1","deploymentId":"app-atttempt-001","subType":"ContainerPlacementRequestMessage","uuid":"f068175b-c9b6-4f34-982b-ecb5619f21de","destinationHost":"ANY_HOST","statusCode":"CREATED","timestamp":1578693870484} |
ContainerPlacementResponseMessage:
| Key | Value | Field Description | Field Type |
|---|---|---|---|
| "UUID.subType" | uuid | Unique identifier of a response message | Required |
processorId | Logical processor id 0,1,2 of the container | Required | |
| deploymentId | Unique identifier for a deployment | Required | |
| subType | Type of message here: ContainerPlacementResponseMessage | Required | |
| destinationHost | Destination host where the container is desired to be moved | Required | |
| statusCode | Status of the current action | Required | |
| responseMessage | Response message in conjunction to status | Required | |
| timestamp | The timestamp of the response message | Required | |
| requestExpiry | Eequest expiry which acts as a timeout for any resource request to cluster resource manager | Optional |
Sample KV
| Key | Value |
|---|---|
[1,"samza-place-container-v1","88b0d30c-d518-4307-9e8e-c8529eb30f04.ContainerPlacementResponseMessage"] | {"processorId":"1","deploymentId":"app-atttempt-001","subType":"ContainerPlacementResponseMessage","responseMessage":"Request is accepted","uuid":"88b0d30c-d518-4307-9e8e-c8529eb30f04","destinationHost":"ANY_HOST","statusCode":"ACCEPTED","timestamp":1578694070875} |
GC policy for stale messages in metastore
- One way to delete stale ContainerPlacementMessages is to delete request / responses from the previous incarnation of the job in the metastore on job restarts
- Once the request is complete, ContainerPlacementService can issue an async delete to the metastore
- Request / response message can be externally cleaned by a tool
Part 2. Container Placement Service
...
- Remove the HostAwareContainerAllocator & ContainerAllocator, simplify Container Allocator as a simple lightweight entity allocating requests to available resources (PR1, PR2)
- Introduce ContainerManager which acts as a brain for validating and issuing any actions on containers in the Job Coordinator for both active & Standby containers. (PR)
- Transfer state & validation of container launch & expired request handling from ContainerAllocator to ContainerManager
- Transfer state & validation for callback handler (ContainerProcessManager) to ContainerManager
- Encapsulates logic and state related to container placement actions like move, restarts for active & standby container in ContainerManager (PR-1, TDB)
- It is ContainerManager’s duty to validate any ContainerPlacementRequestMessages & also invalidate messages from the previous deployment incarnation
- It is ContainerManager’s duty to write ContainerPlacementResponseMessages to Metastore for external control to query the status of the request
- ContainerPlacementMetadata is a metadata holder for container actions (ControlActionMetadata) for ex request_id, current status, requested resources etc
2.1 Container Move
2.1.1 Stateless Container Move & Stateful Container Move (without Standby)
...