Page History

...

1. User must be utilizing both consumer and producer as a complete EOS application,
2. User needs to store transactional offsets inside Kafka group coordinator, not in any other external system for the sake of fencing,
3. Producer needs to call sendOffsetsToTransaction(offsets, groupMetadata) to be able to fence properly.

For Kafka Streams that needs to be backward compatible to older broker versions, we add a new config value for config parameter processing.guarantees that we call "exaclty_once_beta". Hence, users with new broker versions can opt-in the new feature (ie, using a single producer per thread for EOS instead of a producer per task).

public class StreamsConfig {

    public static final String EXACTLY_ONCE_BETA = "exactly_once_beta";

}

Compatibility, Deprecation, and Migration Plan

...

Following the above principle, Kafka Streams uses version probing to perform online upgrade. Step by step guides aretwo rounds of rolling bounced for all Kafka Streams instances are required as follows:

1. Broker must be upgraded to 2.5 first. This means the `inter.broker.protocol.version` (IBP) has to be set to the latest. The new thread producer model shall not be used if broker IBP is low.
2. Upgrade the stream application binary and choose to set UPGRADE_FROM_CONFIG config to 2.3 4 or lower and set PROCESSING_GUARATNEE to "exaclty_once_beta". Do the first rolling bounce, and make sure the group is stable with every instance on 2.5 6 binary.
3. Remove/unset UPGRADE_FROM_CONFIG, in order to point the application to actual Kafka client version. Do a second rolling bounce and now the application officially starts using new thread producer for EOS.

...