...

• Reduce unnecessary downtime due to task restoration
• Make rebalance performance better for stream applications, A.K.A alleviating Stop-The-World Effect.
• Auto scaling support for KStream applications, including
  • Scale up capacity
  • Scale down capacity

Proposed Changes

Terminology

...

We have set of workers, where each worker contains:

Workers = []

Worker {

  Set<ActiveTask>;

  Set<StandbyTask>; 

  instanceId;

}Set<StandbyTask> 

As we could see, there should be only exactly one learner task after each round of rebalance, and there should be exactly one corresponding active task at the same time. 

...

to help user define their customized strategy.

Compatibility, Deprecation, and Migration Plan

• Metadata size increase
• No downtime upgrade due to change of protocolType
• Stream instance replacement
  • The right approach for a global application instance replacement is
    • Increase the capacity of the current stream job to 2
    • Mark existing stream instances as leaving
    • Learner tasks finished on new hosts, shutting down old ones.

FAQ

Why do we call stream workers?

...