THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
In the current implementation of ApplicationRunner, there are a few issues:
- Instantiation of specific implementation of ApplicationRunner is exposed to the user, which requires user to choose a specific implementation of ApplicationRunner in source code, depending on the deployment environment (I.e. YARN vs standalone).
ApplicationRunner only supports high-level API and does not fully support low-level API:
In standalone environment, user's program written in StreamTask/AsyncStreamTask classes is only supported in LocalApplicationRunner w/ a run() method
In YARN, RemoteApplicationRunner only support high-level API applications and falls back to JobRunner for low-level API applications.
There is no unified API to allow user to specify high-level API and low-level API
...
in initialization either.
There is no defined standard lifecycle of a user
...
application process in both YARN and standalone deployment.
...
Hence,
...
no consistent pattern to insert user
...
code into
...
the application’s full lifecycle
There is no standard method to insert user-defined application initialization sequence
...
In YARN, all
...
application processes are
...
initialized (i.e. configure re-writer, stream processor initialization, etc.) by
...
the build-in
...
main functions in Samza framework (i.e. ApplicationRunnerMain on launch host and LocalContainerRunner on NodeManagers).
...
In standalone,
...
user
...
can put
...
arbitrary code in user
...
main
...
function to initialize the application process.
There
...
is no defined method to allow user to inject customized logic
...
before start/stop the user defined processors (I.e. StreamProcessors defined by user application) either, in addition to initialization
Motivation
Our goal is to allow users to write a high-level or low-level API applications once and deploy in both YARN and standalone environments without code change. The following requirements are necessary to achieve our goal:
- H ides
- ide the choice of specific implementation of ApplicationRunner via configuration, not in source code.
Define a
set of standard application lifecycle methods for applications written in both low- and high-level APIsunified API to allow user to specify processing logic in high- and low-level API in all environment (I.e. all ApplicationRunners) w/ all lifecycle methods
Define a set of standard application execution methods to deploy both low- and high-level APIs applications in YARN
deployed in YARN and standalone environments.
Define a
set of standard processor life-
cycle cycle aware
hooks API to allow
user’s
customized logic to be injected in all lifecycle stages in customized logic to be injected before and after start/stop the processors in both YARN and standalone environments
. Support applications written in low-level API in all environment (I.
e. all ApplicationRunners) w/ all lifecycle methods
Note that we need to define the following concepts clearly:
...
- Application’s execution methods in runtime are the functions to change the deployment status of an application runtime instance
...
- (I.e.
...
- run/status/
...
- kill/waitForFinish)
- ApplicationRunner’s execution methods are the deployment environment specific implementation
...
- to the above API methods
- Application’s processor lifecycle aware methods are the user-defined functions to inject customized logic to be executed before or
...
- after we start or stop the stream processing logic defined by the user application (I.e. beforeStart/afterStart/beforeStop/afterStop are called when we start/stop the StreamProcessors in local host)
Proposed Changes
The proposed changes are the followings:
...
- Define a
...
- unified API
...
- ApplicationBase as a single entry point for users to implement all user-
...
- customized logic, for both high-level API and
...
- low-level API
...
-
User implements a single describe() method to implement all user processing logic before creating the runtime application instance
Sub-classes StreamApplication and TaskApplication provide specific describe() methods for high-level API and low-level API, respectively
...
Implement customized logic before / after calling the application lifecycle methods (e.g. start/stop)
Define a unified API class ApplicationSpec to contain
High- and low-level processing logic defined via ApplicationBase.describe()
User implemented ProcessorLifecycleListener class that includes customized logic to be invoked before and after starting/stopping the StreamProcessor(s) in the user application
Methods
...
are beforeStart/afterStart/beforeStop/afterStop
Define the
...
class ApplicationRuntime to represent a runtime instance of a user application,
...
with a
...
set of standard execution methods to change the deployment status of the runtime application instance in different environments (I.e. YARN and standalone)
...
A runtime instance of user application is defined as a bundle of both
ApplicationSpec that
...
contains all user
...
customized logic.
This would be setup through the user application’s describe() method when constructing the ApplicationSpec object
ApplicationRunner that deploys and runs the user code. This would be instantiated from the configuration, not exposed to user at all.
The set of
...
standard execution methods for an ApplicationRuntime object includes
...
run/
...
kill/status/waitForFinish
A high-level overview of the proposed changes is illustrated below: 
Figure-1: high-level user programming model 
Figure-2: Interaction and lifecycle of runtime API objects (using StreamApplication as in LocalApplicationRunner as an example).
The above design achieves the following goals:
...