Page History

...

JobModel is the data model in samza that logically represents a samza job. The JobModel hierarchy is that samza jobs have one to many containers(ContainerModel), and each container has one to many tasks(TaskModel). Each data model contains relevant information, such as logical id, partition information, etc. In the standalone deployment model, the JobModel is stored in zookeeper. Coordinator stream(kafka topic) is used to store the JobModel in the yarn deployment model.

Existing host affinity implementation in yarn is accomplished through the following two phases:
A. ApplicationMaster(JobCoordinator) in yarn deployment model generates the Job model(optimal processor to task assignment) and persists the JobModel in coordinator stream(kafka topic) associated with the samza job.
B. ContainerAllocator phase(which happens after JobModel generation) schedules each stream processor to run on a physical host by coordinating with the underlying ClusterManager and orchestrates the execution of the stream processors.

Zookeeper is used in standalone for coordination between the processors of a stream application. Amongst all the available processors of a stream application, a single processor will be elected as a leader in standalone. The leader will generate the JobModel and propagate the JobModel to all the other processors in the group. Distributed barrier in zookeeper will be used to block the message processing until the latest JobModel is picked by all the processors in the group.

Here’re the list of important and notable differences in processor and JobModel generation semantics between yarn and standalone deployment model:

...

Here’re the list of important and notable differences in processor and JobModel generation semantics between yarn and standalone deployment model:

• Number of containers is a static configuration in yarn deployment model and a job restart is required to change the number of containers. However, an addition/deletion of a processor to a processors group in standalone is quite common and an expected behavior.
• A stream processor is assigned a physical host by ContainerAllocator after the JobModel generation phase in yarn. Physical host in which a stream processor is going to run is known before the JobModel generation phase in standalone(ContainerAllocator phase is not needed in standalone to associate a processor with the physical host).

Existing host affinity implementation in yarn is accomplished through the following two phases:
A. ApplicationMaster(JobCoordinator) in yarn deployment model generates the Job model(optimal processor to task assignment) and persists the JobModel in coordinator stream(kafka topic) associated with the samza job.
B. ContainerAllocator phase(which happens after JobModel generation) schedules each container to run on a physical host by coordinating with the underlying ClusterManager and orchestrates the execution of the container.

Zookeeper is used in standalone for coordination between the stream processors of a stream application. Amongst all the available processors of a stream application, a single processor will be elected as a leader in standalone. The leader will generate the JobModel and propagate the JobModel to all the other processors in the group. Distributed barrier in zookeeper will be used to block the message processing until the latest JobModel is picked by all the processors in the group

...

.

Overall high level changes:

...