Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Kafka Streams enforces a strict non-null-key policy in the DSL across all key-dependent operations (like aggregations and joins).
For left-joins, it makes sense to still accept a `null`, and add the left-hand record with an empty right-hand-side to the result.
Similarly, for outer-joins it makes sense to keep the record.
The defined semantics for left/outer join are: keep the stream record if no matching join record was found.
Thus, Kstreams today's behavior is inconsistent with the defined semantics.

Public Interfaces Proposed Changes

This KIP won't change any public interfaces. However, It will change the behavior of the following

Operators

  • left join Kstream-Kstream
  • outer join Kstream-Kstream
  • left-foreign-key join Ktable-Ktable
  • left  join KStream-Ktable
  • left  join KStream-GlobalTable

Repartition of null-key records

...

Proposed Changes

  • left join Kstream-Kstream: no longer drop left records with null-key and call ValueJoiner with 'null' for right value.
  • outer join Kstream-Kstream: no longer drop left/right records with null-key and call ValueJoiner with 'null' for right/left value.
  • left-foreign-key join Ktable-Ktable: no longer drop left records with null-foreign-key returned by the ForeignKeyExtractor and call ValueJoiner with 'null' for right value.
  • left join KStream-Ktable: no longer drop left records with null-key and call ValueJoiner with 'null' for right value.
  • left join KStream-GlobalTable: no longer drop left records with null-key and call KeyValueMapper with 'null' for left  key. The case where KeyValueMapper returns null is already handled in the current implementation.

Repartition of null-key records

Currently, repartitioning causes records with 'null'-keys to be dropped.
This behavior needs to be changed.
The above described change in behavior would otherwise not work for topologies with repartitioning.
Going forward, they will be sent to an arbitrary partition.

Compatibility, Deprecation, and Migration Plan

...