...

Host Affinity & Kubernetes

This document describes a mechanism to allow Samza to request containers from YARN on a  specific machine. This locality-aware container assignment is particularly useful for containers to access their local state on the machine. This mechanism leverages a feature in YARN to be able to request container by hostname.

Similar primitive is provided in Kubernetes to allow users to request pods by hostname. This document describes the feature. Particularly, “preferredDuringSchedulingIgnoredDuringExecution” policy can be used to “run my pod on host X, if not satisfied, run it elsewhere.”

Alternatively, If a remote storage, instead of local, can be used for persisting Samza task state, the goal of container-state-rebinding can be achieved by dynamically attach the remote storage to the container even if the container is restarted on a different host, by leveraging the Kubernetes PersistentVolume primitive. This is usually useful in a cloud environment where remote storage is typically accessible.

Reference

• The java client for interacting with Kubernetes APIs: https://github.com/fabric8io/kubernetes-client

...