Versions Compared

Old Version 12

changes.mady.by.user Weiqing Yang

Saved on

compared with

New Version 13

changes.mady.by.user Weiqing Yang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Prepare a base container image for Samza application including all the Samaza framework jars etc.

  2. The run-app.sh and ApplicationRunner needs to be modified to support submitting apps to Kubernetes api-server.

  3. Develop a Samza Operator, similar to YARN AM, that creates task Pods in Kubernetes.

  4. Develop a module that can download the dependencies from remote artifact repositories. The module can be in the init-container or embedded in the main/regular container.

    Samza-Operator Details

    The Samza operator is very similar to YARN AM. When it starts, it first reads the JobModel from coordinator stream and then create pods from Kubernetes with the container information provided. The Kubelet will then start the containers.

    Image Added

    Note that there is a difference here between Kubernetes and YARN. In YARN,  there are usually two stages: AM first request the containers and then launch the container on NodeManager

    In Kubernetes, requesting containers and starting containers are done in a single call. The createPod API will request the pods and then the pods will be started by kubelet automatically.

    Due to this difference, the current samza on YARN implementations regarding storing the resource requests and then match the received containers with the stored pending requests are not needed.

Interfaces

WIP .. Coming soon

...