Status

Current state: Under Discussion ADOPTED (in 3.0)

Discussion thread: here TODO 

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In KIP-447 we introduced a new thread-producer which is capable of exactly-once semantics across multiple tasks. The new mode of EOS, called eos-beta, is intended to eventually be the preferred processing mode for EOS as it improves the performance and scaling of partitions/tasks. The only downside is that it requires brokers to be on version 2.5+ in order to understand the latest APIs that are necessary for this thread-producer.

...

Additionally, the current name given to the new-and-improved processing guarantee may give users the impression that it's not stable or production-ready. We should also deprecate eos-beta, although in name only, to remove the "-beta" and all connotations.

Public Interfaces

public class StreamsConfig {

    /**
    @Deprecated // use EXACTLY_ONCE_V2 * @deprecated Since 3.0.0, will be removed in 4.0. Use {@link #EXACTLY_ONCE_V2 "exactly_once_v2"} instead.
     */
    @Deprecated 
    public static final String EXACTLY_ONCE = "exactly_once"; 

    @Deprecated // use EXACTLY_ONCE_V2
    /**
     * @deprecated Since 3.0.0, will be removed in 4.0. Use {@link #EXACTLY_ONCE_V2 "exactly_once_v2"} instead.
     */
    @Deprecated
    public static final String EXACTLY_ONCE_BETA = "exactly_once_beta"; 

    /**
     * Config value for parameter {@link #PROCESSING_GUARANTEE_CONFIG "processing.guarantee"} for exactly-once processing guarantees.
     * <p>
     * Enabling exactly-once-v2 requires broker version 2.5 or higher.
     * If you enable this feature Kafka Streams will use fewer resources (like broker connections)
     * compared to the {@link #EXACTLY_ONCE} case. This config corresponds
     * to the old EXACTLY{@link #EXACTLY_ONCE_BETA} processing mode, which has been deprecated 
     * in favor of this config.
     */
    public static final String EXACTLY_ONCE_V2 = "exactly_once_v2";

}

class Producer {

    /**
     * @deprecated Since 3.0.0, will be removed in 4.0. Use {@link #sendOffsetsToTransaction(Map, ConsumerGroupMetadata)} instead.
     */
    @Deprecated
    public void sendOffsetsToTransaction(Map<TopicPartition, OffsetAndMetadata> offsets,
                                         String consumerGroupId);
}

Proposed Changes

We plan to deprecate both existing EOS configs in 3.0, and add a new config which corresponds to the existing "eos-beta" option. What we now call "eos-beta" will become "eos-v2", in code, comments, configs, and docs. No changes to the actual eos implementation will be made at this time – when time, and the configs will remain until 4.0. At that time, both the deprecated configs and the underlying eos-alpha is finally removed (likely in 4.0) then we can implementation will be removed. This will allow us to significantly clean up the task and producer management code, since it there will only be all thread-level producers remaining.

Since some users may skip the StreamsConfig and set the string value of "exactly_once" or "exactly_once_beta" directly, we will also log a warning if those configs are detected to raise the visibility for users who may have missed the deprecation.

In addition, we plan to deprecate the old Producer#sendOffsetsToTransaction overload which accepts a String consumerGroupId parameter, in favor of the new overload which accepts a ConsumerGroupMetadata. The deprecated method currently just invokes the new method anyway, so this should not have any impact on users (ie, both methods apply to the same cases)

Compatibility, Deprecation, and Migration Plan

Existing users of eos-alpha are encouraged to upgrade their brokers in preparation for its removal in a later version. Existing eos-beta users will simply need to swap in the new config name.

Rejected Alternatives

A few options were discussed on the original ticket:

...