...

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-3821

  has been open for a long time and discusses a lot of ideas. Some of the ideas are:
  • a SourceRecord object with a null (or special) topic, or a null key and value
    • Didn't want to rely on null semantics to avoid confusion.
    • Another topic adds to the operational burden.
  • a SourceRecord object with a null key and value
    • Same, didn't want to rely on null semantics.
  • a subclass of SourceRecord that denotes this particular kind of request. 
    • This could have been passed as part of output of poll() method  but as such SourceRecord don't have sub-classes and yet is specialised. Extending it doesn't seem the right thing to do in this case. This would have also needed to throw a ClassNotFoundException  for older runtimes which is already being done but it won't be needed if the approach from the KIP is followed.
  • Changing the signature of poll() to accept a SourceRecordReceiver parameter
    • This breaks backward compatibility
  • Creating a dedicated topic where records are being send over periodic time intervals.
    • This was the original approach prescribed in the KIP but it was shelved because:
      • Another topic adds to the operational burden.
      • Since the mechanism was akin to heartbeat records, the general consensus was to have heartbeat mechanism by not needing to produce records explictly.
  • That ticket has a lot of comments/suggestions and some open PRs as well. This KIP chose to use the option which seemed the most non-invasive.

Future Work

• Allow writing tombstone offsets to offsets topic when a task sends null offset for a partition. Currently, even though a task might choose to send a null offset for a partition, that won't have any effect on the offsets being flushed.