Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In KIP-447 we introduced a new thread-producer which is capable of exactly-once semantics across multiple tasks. The new mode of EOS, called eos-beta, is intended to eventually be the preferred processing mode for EOS as it improves the performance and scaling of partitions/tasks. The only downside is that it requires brokers to be on version 2.5+ in order to understand the latest APIs that are necessary for this thread-producer.

...

Additionally, the current name given to the new-and-improved processing guarantee may give users the impression that it's not stable or production-ready. We should also deprecate eos-beta, although in name only, to remove the "-beta" and all connotations.

Public Interfaces

public class StreamsConfig {

    @Deprecated // use EXACTLY_ONCE_V2
    public static final String EXACTLY_ONCE = "exactly_once"; 

    @Deprecated // use EXACTLY_ONCE_V2
    public static final String EXACTLY_ONCE_BETA = "exactly_once_beta"; 

    /**
     * Config value for parameter {@link #PROCESSING_GUARANTEE_CONFIG "processing.guarantee"} for exactly-once processing guarantees.
     * <p>
     * Enabling exactly-once-v2 requires broker version 2.5 or higher.
     * If you enable this feature Kafka Streams will use fewer resources (like broker connections)
     * compared to the {@link #EXACTLY_ONCE} case. This config corresponds
     * to the old EXACTLY_ONCE_BETA processing mode, which has been deprecated 
     * in favor of this config.
     */
    public static final String EXACTLY_ONCE_V2 = "exactly_once_v2";
}

Proposed Changes

We plan to deprecate both existing EOS configs, and add a new config which corresponds to the existing "eos-beta" option. What we now call "eos-beta" will become "eos-v2", in code, comments, configs, and docs. No changes to the actual eos implementation will be made at this time – when eos-alpha is finally removed (likely in 4.0) then we can significantly clean up the task and producer management code since it will be all thread-level producers

Compatibility, Deprecation, and Migration Plan

Existing users of eos-alpha are encouraged to upgrade their brokers in preparation for its removal in a later version. Existing eos-beta users will simply need to swap in the new config name.

Rejected Alternatives

A few options were discussed on the original ticket:

...