Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The first one has been present for the majority of Kafka Stream's inception, when results are forwarded regardless . The second option has been added by implementing the so-called "suppression" operator which collects results throughout the window, and extending into the grace period. However, one mode of processing should've been implemented long ago, and that is emit on change. As mentioned in the JIRA, in many cases, the user would process a prior result to yield a new one, yet, this new result is forwarded to a downstream processor even though the operation performed had been effectively a noopno-op. This is problematic in certain circumstances where there is a high number of such operations, leading to a number of useless results being forwarded down the topology.

...

However, the main constraint for this approach, if we were to follow through, is that the hash code must reflect any changes in the individual component fields of the result. The key building block for this entire optimization is that the user provides a good hash code override of the given value type. Of course, this isn't always the case.  After all, it isn't safe to assume that the user has provided a safe hash code function for us to take advantage of. ThereforeIn conclusion, we can probably come up with the following choices for the user:

...

  1. Let's say we receive record 1 with timestamp 0 which leads to a change in the result processed
  2. Afterwards, we receive another record with timestamp 100 which would lead to a no-op

It is required that the timestamp be 0. 

To resolve this situation, the current best bet is probably to load the timestamp along with the hash code / full prior result. 

Added methods

TBD. Discussion required first.

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

...