Status

Current state: Under Discussion

Discussion thread: TBD

JIRA:   Unable to render Jira issues macro, execution error. Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

1. The main motivation of this KIP is stated in the related jira: "Today the downstream sub-topology's parallelism (aka the number of tasks) are purely dependent on the upstream sub-topology's parallelism, which ultimately depends on the source topic's num.partitions. However this does not work perfectly with dynamic scaling scenarios". 
2. Depending on use-case, we might not need to use repartitioning topic, as keys might not change.
3. The KStream application has no control over the sink topic creation, which means user has to create their own connecting Kafka topics with desired partitions which is cumbersome.

Proposed Changes

public class Produced<K, V> {
	protected String topicName;
	protected Boolean topicCreated = false;
	protected Boolean repartitionNeeded = true;
	protected Boolean useAsInternalTopicOnly = true;
 	protected Integer partitions; 
	protected Produced(String topicName, Boolean alreadyCreated) {
    	this.topicName = topicName;
   	    this.alreadyCreated = alreadyCreated;
	}
 
	protected Produced(String topicName, Integer partitions) {
    	this.topicName = topicName;
   	    this.partitions = partitions;
	}

 
	public static <K, V> Produced<K, V> withTopic(String topicName, Boolean alreadyCreated) {
    	return new Produced<>(topicName, alreadyCreated);
	}
 
	public static <K, V> Produced<K, V> withTopic(String topicName, Integer partitions) {
    	return new Produced<>(topicName, partitions);
	}

	public Produced<K, V> repartitionNeeded(Boolean repartitionNeeded) {
		this.repartitionNeeded = repartitionNeeded;
		return this;
	}
 
	public Produced<K, V> useAsInternalTopicOnly(Boolean useAsInternalTopicOnly) {
		this.useAsInternalTopicOnly = useAsInternalTopicOnly;
		return this;
	}
}

Public interfaces

KStream

 KStream<K, V> through(final Produced topicInfo);
 void to(final Produced topicInfo);

The change with Produced is not compatible with the definition of “The specified topic should be manually created before it is used”. However, since we reject

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.