Status

Current state: Under Discussion

Discussion thread:https://www.mail-archive.com/dev@kafka.apache.org/msg97405.html

JIRA:  Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Connect's API provides a list method which returns all running connector names (/connectors) as well as state info methods parameterized by name (/connectors/{connectorName}/status). This leads to a situation where users of this API need to make N api calls to get information about all their running connectors.

Public Interfaces

The current `/connectors` endpoint will gain a new multi query param `?expand=(status|config|info)` which will return a map {connectorName->expansion->(ConnectorStateInfo|ConnectorInfo|Map<String, String>)}.

the following expansions are allowed

w:kafka norwood$ curl -s http://localhost:8083/connectors | jq
[
  "blah"
]

w:kafka norwood$ curl -s 'http://localhost:8083/connectors?expand=status&expand=config' | jq
{
  "blah": {
    "config": {
      "name": "blah",
      "config": {
        "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
        "file": "/tmp/lol",
        "tasks.max": "10",
        "name": "blah",
        "topic": "test-topic"
      },
      "tasks": [
        {
          "connector": "blah",
          "task": 0
        }
      ],
      "type": "source"
    },
    "status": {
      "name": "blah",
      "connector": {
        "state": "RUNNING",
        "worker_id": "10.200.25.241:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "10.200.25.241:8083"
        }
      ],
      "type": "source"
    }
  }
}

Proposed Changes

The changes are ~entirely in ConnectorsResource and the Herders. PR: https://github.com/apache/kafka/pull/6658

Compatibility, Deprecation, and Migration Plan

This change is backwards compatible.

Rejected Alternatives

• Adding an entirely new endpoint.
  • The way that ConnectorsResource is setup right now is not conducive to adding another endpoint that is not namespaced under /connectors/{connectorName}
  • We could conceivably rework more code in ConnectorsResource to allow us to have a /connectorsExpanded endpoint and then retain the other /connectors/{connectorName}/status endpoints, but this introduces even more questions, e.g. what would be under /connectorsExpanded/{connector}/status? 
• Add a connect rest extension.
  • This seems like overkill for basic functionality. We have all the data available in the herder now, so requiring additional install steps to get basic info seems like a bit much.