Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

DELETE requests for connectors running in distributed mode (DistributedHerder.java) are handled asynchronously. When deleted, the connector configuration is removed from the backing store, and these changes are processed in on the Herder's main poll thread. Here

Compatibility, Deprecation, and Migration Plan


Worker class

In both cases, herders invoke methods from the Worker class to start/stop the connector instance. What this KIP proposes is to overload the  Worker#stopAndAwaitConnector(...) method, passing a flag indicating that the connector is being stopped because it has been deleted, and have the worker call the new  deleted() method on the connector.

Compatibility, Deprecation, and Migration Plan

The proposed change is fully backward-compatible with existing Kafka Connect releases. The new method The proposed change is fully backward-compatible with existing Kafka Connect releases. The new method added to the public interface includes an empty (noop) implementation, so existing connectors don't need to implement it if not needed.

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

Integration tests will be added to make sure that the new method is invoked when a connector is deleted. Mainly:

  • Adding new tests to the ConnectWorkerIntegrationTest integration test cases and auxiliary classes (EmbeddedConnectClusterAssertions,SampleSourceConnector,SampleSinkConnector etc.)

Rejected Alternatives

In theory, users can monitor Kakfa Connect configuration topics to determine if/when a connector has been deleted. Reacting to this event outside of the connector's context is probably not very useful, as there might not be enough contextual information to perform any meaningful action. It is also better to keep these concerns encapsulated within the connector framework itselfIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.