...
Solution 1. Write locality to Coordinator Stream (Metastore) and restart job [Rejected]
This approach proposes to directly change the container locality messages (Container to host mapping messages) in the Metadata Store (currently Kafka) and issuing a restart
Pros | Cons |
- Simple to implement the current tool does that for host affinity enabled jobs (since they maintain locality mapping)
| - Needs a job restart and does a best effort to get preferred hosts for containers but has no guarantee on getting them
- If a job has standby containers enabled, this method involves changing standby mapping in addition to active container mappings
- Job faces downtime when the job has hundreds of containers and only one of them needs to be restarted, if it is stateful there is a likelihood that containers might not get the new asked resource on the restart and start bootstrapping
- This solution is not scalable to be used by Controllers who want to take multiple control actions on containers across several jobs, for example, auto-sizing controller
- This method will not be work for building Canary / Cluster Balancer
|
...
Samza Metastore provides an API to write to the coordinator stream. One simple way to expose Container Placement API is, Container Placement handler can have a coordinator stream consumer polling control messages from the coordinator stream
under a separate namespace (different than the one where locality messages of containers live) & acting on them.
Pros | Cons |
- No need to build Authentication & Authorization, already handled by the Metadata auth service
- No need to enable Rate limiting since requests are queued so the flow of requests can be regulated at the consumer side
| - If AM dies there can be still queued requests in Coordinator Stream, such requests have to be handled across AM restarts
- Coordinator stream is log compacted so control messages written to the coordinator stream need to be deleted to prevent it from growing to large sizes which can affect job start times
|
...
