Status

Current state: Under  Discarded

Discussion thread: here

Discussion Vote thread: here

JIRA: KAFKA-6725

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

After some code analysis, we determined that there wasn't an existing API that a Connector could use to determine that the task was about to be shut down for a rebalance. After some thought we decided that this was a problem more generally about giving connectors the information about the state of the world outside the connector such that they could make intelligent decisions about how to behave in preCommit. Giving this information to a connector gives developers writing connectors options to change their behavior if a rebalance or shutdown is pending.

To achieve this we We would like to add such an API so that the preCommit hook of a Connector can decide if it should behave differently because the task is actively trying to shut down.

...

Proposed Changes

1. We'll add the isClosing method to the SinkTaskContext as mentioned above.
2. We'll add the implementation for that to WorkerSinkTaskContext
3. We'll add a setter named setClosingInProgress to WorkerSinkTaskContext that'll change the internal state
4. In WorkerSinkTask.commitOffsets:
   1. If the closing parameter is set to true, we'll invoke setClosingInProgress on the context to true. This will occur before the invocation of preCommit.
   2. Before commitOffsets returns, it will setClosingInProgress will setClosingInProgress back to false.
5. Add a note to the preCommit javadoc to indicate the importance of checking SinkTaskContext.isClosed when running,

This change will give preCommit the opportunity to provide some additional commits that it would like Kafka Connect to make before the task is fully shut down.

...

• We considered adding a closing parameter to preCommit, but that would be a breaking change for all connectors and would require them to change their code when upgrading to the next version of Connect. It would also break binary compability.

...

• unless it was done very carefully. The additional complexity would buy us marginal extra value, if any at all, so we decided to go with the addition to the context object.