DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: DraftUnder Discussion
Discussion thread: https://lists.apache.org/thread/7y7svyp3f560fzv1bgcr893vn258cn06
JIRA:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
public interface StreamPartitioner<K, V> {
Optional<Set<Integer>> partitions(String topic, K key, V value, int numPartitions);
/**
* Determine the number(s) of the partition(s) to which a record with the given key and value should be sent,
* for the given topic and current partition count.
*
* @param topic the topic name this record is sent to
* @param key the key of the record
* @param value the value of the record
* @param headers the record headers
* @param numPartitions the total number of partitions
* @return an Optional of Set of integers between 0 and {@code numPartitions-1}
*/
,
* Empty optional means using default partitioner
* Optional of an empty set means the record won't be sent to any partitions i.e drop it.
* Optional of Set of integers means the partitions to which the record should be sent to.
* */
default Optional<Set<Integer>> partitions(String topic, K key, V value, Headers headers, int numPartitions) {
return partitions(topic, key, value, numPartitions);
}
} |
| Code Block | ||||
|---|---|---|---|---|
| ||||
// already existing method public <K> KeyQueryMetadata queryMetadataForKey(final String storeName, final K key, final Serializer<K> keySerializer) { return queryMetadataForKey(storeName, key, new RecordHeaders(), keySerializer); } // already existing method public <K> KeyQueryMetadata queryMetadataForKey(final String storeName, final K key, final StreamPartitioner<? super K, ?> partitioner) { return queryMetadataForKey(storeName, key, new RecordHeaders(), partitioner); } /** * Determine the number(s) of the partition(s) to which a record with * Finds the metadata containing the active hosts and standby hosts where the key being queried would reside. * * @param storeName the {@code storeName} to find metadata for * @param key the key to find metadata for * @param headers the record headers * @param keySerializer serializer for the key * @param <K> key type * Returns {@link KeyQueryMetadata} containing all metadata about hosting the given key for the given store, value, and headers should be sent, * for the given topic and current partition count. * * @param topic the topic name this record is sent to * @param key the key of the record * @param value the value of the record * @param headers the headers of the record * @param numPartitions the total number of partitions * @return an Optional of Set of integers between 0 and {@code numPartitions-1} */ default Optional<Set<Integer>> partitions(String topic, K key, V value, Headers headers, int numPartitions) { return partitions(topic, key, value, numPartitions); } } |
Proposed Changes
Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Test Plan
Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
Rejected Alternatives
...
* or {@code null} if no matching metadata could be found.
*/
public <K> KeyQueryMetadata queryMetadataForKey(final String storeName,
final K key,
final Headers headers,
final Serializer<K> keySerializer) {
validateIsRunningOrRebalancing();
return streamsMetadataState.keyQueryMetadataForKey(storeName, key, headers, keySerializer);
}
/**
* Finds the metadata containing the active hosts and standby hosts where the key being queried would reside.
*
* @param storeName the {@code storeName} to find metadata for
* @param key the key to find metadata for
* @param headers the record headers
* @param partitioner the partitioner to be used to locate the host for the key
* @param <K> key type
* Returns {@link KeyQueryMetadata} containing all metadata about hosting the given key for the given store, using
* the supplied partitioner, or {@code null} if no matching metadata could be found.
*/
public <K> KeyQueryMetadata queryMetadataForKey(final String storeName,
final K key,
final Headers headers,
final StreamPartitioner<? super K, ?> partitioner) {
validateIsRunningOrRebalancing();
return streamsMetadataState.keyQueryMetadataForKey(storeName, key, headers, partitioner);
} |
Proposed Changes
- Interface Enhancement: Add the headers-aware
partitionsmethod toStreamPartitioner. This is adefaultmethod to ensure binary compatibility. - Public API for Interactive Queries: Add new overloads to
KafkaStreams#queryMetadataForKeythat accept `Headers`. This allows users to perform metadata lookups for header-dependent partitioning strategies - No changes are planed for
KafkaStreamsNamedTopologyWrapper. This class and entire feature are deprecated, so no changes are planed there - Internal Callers and Propagation:
RecordCollectorImpl#sendto pass record headers when calling the partitioner.StreamsMetadataStateto support the new header-aware lookups.
- Built-in Implementations:
WindowedStreamPartitioner: Override the new method to propagate headers to the underlyingWindowedSerializer#serializeBaseKey.DefaultStreamPartitioner: Override the new method to propagate headers to thekeySerializer#serialize
Compatibility, Deprecation, and Migration Plan
- Backward Compatibility: change is fully backward compatible. Existing custom
StreamPartitionerimplementations will continue to work as they will use the default implementation of the new method. - Deprecation: No methods are being deprecated in this KIP.
- Migration: Users wishing to leverage headers in their custom partitioning logic should migrate their
StreamPartitionerimplementations to override the new `partitions` method.
Test Plan
- Unit Tests and Integration tests:
- Verify
Partitioners correctly propagate headers. - Verify propagated headers are used to determine target partition
- Verify
Rejected Alternatives
- Updated the existing method without default: Rejected as it would break all existing user implementations.