THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The main motivation of this KIP is stated in the related jira: "Today the downstream sub-topology's parallelism (aka the number of tasks) are purely dependent on the upstream sub-topology's parallelism, which ultimately depends on the source topic's num.partitions. However this does not work perfectly with dynamic scaling scenarios".
...
By delegating the stream topology power to create repartition topic with customized number of partitions gives user more flexibility. Also, for API like #to or #through, KStream has no access to sink topic creation, which means user has to create their own connecting Kafka
...
topic for every single new application which is cumbersome. Thus we are proposing extending the capability of #Produced API to automatically create topics when the target topic has not created.
Proposed Changes
| Code Block | ||
|---|---|---|
| ||
public class Produced<K, V> {
protected String topicName;
protected Boolean topicCreated = false;
protected Boolean repartitionNeeded = true;
protected Boolean useAsInternalTopicOnly = true;
protected Integer partitions;
protected Produced(String topicName, Boolean alreadyCreated) {
this.topicName = topicName;
this.alreadyCreated = alreadyCreated;
}
protected Produced(String topicName, Integer partitions) {
this.topicName = topicName;
this.partitions = partitions;
}
public static <K, V> Produced<K, V> withTopic(String topicName, Boolean alreadyCreated) {
return new Produced<>(topicName, alreadyCreated);
}
public static <K, V> Produced<K, V> withTopic(String topicName, Integer partitions) {
return new Produced<>(topicName, partitions);
}
public Produced<K, V> repartitionNeeded(Boolean repartitionNeeded) {
this.repartitionNeeded = repartitionNeeded;
return this;
}
public Produced<K, V> useAsInternalTopicOnly(Boolean useAsInternalTopicOnly) {
this.useAsInternalTopicOnly = useAsInternalTopicOnly;
return this;
}
} |
...