THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Implementation and Test Plan
1. RemoteApplicationRunner
When the RemoteRunner executes the ProcessorGraph, each ProcessorNode is executed as a physical Job (For now we only support single-node ProcessorGraph, so only one physical job will be created). The Jobs will be submitted through the JobRunner. JobRunner is the runner for a single remote Job. JobRunner is cluster agnostic, meaning it should be able to submit jobs to different types of clusters based on configs (right now we only support Yarn). The following pseudocode shows the start of RemoteRunner:
| Code Block |
|---|
void RemoteRunner::run(StreamApplication app) {
StreamGraph streamGraph = createStreamGraph(app);
ProcessorGraph processorGraph = executionPlanner.plan(streamGraph);
streamsManager.createStreams(processorGraph.getIntermediateStreams());
streamGraph.getProcessorNodes().forEach() { processor ->
(new JobRunner).run(processor.generateConfig());
}
} |
RemoteApplicationRunner.stop() is similar by replacing JobRunner.run() with JobRunner.stop()
Note: this depends on SAMZA-1089 that adds stop() and status() in JobRunner.
How this works in real deployment
User configs app.application.runner.class to be RemoteRunner.
Deploy the application to RM
Execute run-app.sh and stop-app.sh to start/stop the application. In the scripts, ApplicationRunnerMain.main() is invoked to run RemoteApplicationRunner (based on the config in 1).
2. LocalApplicationRunner
Compatibility, Deprecation, and Migration Plan
...