Page History

...

Based on different execution environment, the runner JobRunner behavior is different:

...

• LocalJobRunner will launch a StreamProcessor corresponding to each JobNode within the local JVM. In this mode the LocalJobRunner will run on every host that the application is deployed to. It is also used to launch jobs in test environment as well.

• RemoteJobRunner will submit a Samza job for remote execution for each corresponding JobNode. It is for clusters like Yarn.

...

LocalApplicationRunner.run() {
  
  StreamGraph streamGraph = createStreamGraph();
  ExecutionPlan plan = executionPlanner.plan(streamGraph);

 If (plan has intermediate streams) {  
    startLeaderElection(() -> onBecomeLeader() {
     streamManager.createStreams();
     releaseLatch();
    })
  
    waitForLatch(); // blocked until the latch is reached
 }

  List<LocalJobRunner> jobs;
  plan.getJobConfigs().forEach() { jobConfig ->
     LocalJobRunner jobRunner = new LocalJobRunner();
     jobRunner.start(jobConfig);
     jobs.add(jobRunner);
  }


  jobs.forEach(job -> {
     wait for job complete;
  })
}

...

• Configure app.class = <user-app-name> and task.execute = run-local-app.sh

• Deploy the application to single deployment gateway (RM in YARN).

• Run run-app.sh on the deployment gateway to start the whole application. Run stop-app.sh on the same node to stop it.

  • RemoteApplicationRunner will automatically configure task.execute = run-local-app.sh for each job to be deployed

• When the cluster executor actually launches the Samza container processes, it runs run-local-app.sh on the target host, exactly like in the standalone deployment.

...