Status

Current state: "Accepted"

Discussion thread: here

Vote thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

A new endpoint (GET /connectors/{connector}/tasks-config) was added to Kafka Connect's REST API to expose task configurations in KIP-661. However, the original patch for Kafka Connect's REST API had already added an endpoint (GET /connectors/{connector}/tasks) to retrieve the list of a connector's tasks and their configurations (ref - https://github.com/apache/kafka/pull/378 , https://issues.apache.org/jira/browse/KAFKA-2369) and this was missed in KIP-661. We can deprecate the endpoint added by KIP-661 in 3.7 (the next minor AK release) and remove it in 4.0 (the next major AK release) since it's redundant to have two separate endpoints to expose task configurations. Related discussions in https://github.com/apache/kafka/pull/13424#discussion_r1144727886 and https://issues.apache.org/jira/browse/KAFKA-15377.

Public Interfaces

• GET /connectors/{connector}/tasks-config -  this endpoint will be deprecated in 3.7 and removed in 4.0. A note will be added to the public documentation indicating that the endpoint is deprecated, Connect's OpenAPI spec will be updated to mark the endpoint as deprecated, and a warning log will also be emitted every time the endpoint is invoked.

Proposed Changes

The GET /connectors/{connector}/tasks-config endpoint will be deprecated in 3.7 and removed in 4.0. A note will be added to the public documentation indicating that the endpoint is deprecated, Connect's OpenAPI spec will be updated to mark the endpoint as deprecated, and a warning log will also be emitted every time the endpoint is invoked.

Compatibility, Deprecation, and Migration Plan

• The GET /connectors/{connector}/tasks-config endpoint will be deprecated in 3.7 and removed in 4.0
• Connect's embedded cluster for integration testing will be updated to use the GET /connectors/{connector}/tasks endpoint here.
• Any users of the endpoint will need to migrate their usage to the GET /connectors/{connector}/tasks endpoint starting from 4.0. The response structures are slightly different (although the content is functionally identical) and that will be called out in the deprecation documentation and warning. Example responses for each of the two endpoints -

{
  "file-sink-test-0": {
    "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
    "file": "test.txt",
    "task.class": "org.apache.kafka.connect.file.FileStreamSinkTask",
    "topics": "test-topic",
    "tasks.max": "2",
    "name": "file-sink-test"
  },
  "file-sink-test-1": {
    "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
    "file": "test.txt",
    "task.class": "org.apache.kafka.connect.file.FileStreamSinkTask",
    "topics": "test-topic",
    "tasks.max": "2",
    "name": "file-sink-test"
  }
}

[
  {
    "id": {
      "connector": "file-sink-test",
      "task": 0
    },
    "config": {
      "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
      "file": "test.txt",
      "task.class": "org.apache.kafka.connect.file.FileStreamSinkTask",
      "topics": "test-topic",
      "tasks.max": "2",
      "name": "file-sink-test"
    }
  },
  {
    "id": {
      "connector": "file-sink-test",
      "task": 1
    },
    "config": {
      "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
      "file": "test.txt",
      "task.class": "org.apache.kafka.connect.file.FileStreamSinkTask",
      "topics": "test-topic",
      "tasks.max": "2",
      "name": "file-sink-test"
    }
  }
]

Rejected Alternatives

1. Deprecate and remove the GET /connectors/{connector}/tasks endpoint instead. Rejected because - 
   • This endpoint is much older and has existed ever since Kafka Connect (or Copycat 🐱 as it was known back then) was released (in 0.9.0.0) which means it likely has much wider adoption.
2. Continue to support both the endpoints. Rejected because - 
   • As evidenced in , it becomes a bit of a maintenance burden to have two different endpoints for the same purpose. All contributors and maintainers may not even be aware about the existence of both the endpoints and it'd be difficult to diligently ensure that any fixes or changes are applied across both the endpoints.