Table of Contents |
---|
Status
Current state: Under discussionAccepted
Discussion thread: here
JIRA: KAFKA-5657
Released: 1.0.0
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
We will modify the following REST API endpoint endpoints for Connect to include the type of the connector:
Code Block |
---|
GET /connectors/(string:name)
GET /connectors/(string:name)/status |
Proposed Changes
The aforementioned endpoint endpoint GET /connectors/(string:name)
currently returns the following structure:
...
Code Block |
---|
HTTP/1.1 200 OK Content-Type: application/json { "name": "hdfs-sink-connector", "config": { "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector", "tasks.max": "10", "topics": "test-topic", "hdfs.url": "hdfs://fakehost:9000", "hadoop.conf.dir": "/opt/hadoop/conf", "hadoop.home": "/opt/hadoop", "flush.size": "100", "rotate.interval.ms": "1000" }, "type": "Sinksink", "tasks": [ { "connector": "hdfs-sink-connector", "task": 1 }, { "connector": "hdfs-sink-connector", "task": 2 }, { "connector": "hdfs-sink-connector", "task": 3 } ] } |
For the following REST API endpoint:
Similarly, the endpoint GET /connectors/(string:name)/
Proposed Changes
The aforementioned endpoint currently status
currently returns the following structure:
Code Block |
---|
HTTP/1.1 200 OK Content-Type: application/json { "connector.classname": "io.confluent.connect.hdfs.HdfsSinkConnectorhdfs-sink-connector", "tasks.maxconnector": "10", { "topicsstate": "test-topicRUNNING", "hdfs.url "worker_id": "hdfs://fakehost:9000"8083" }, "hadoop.conf.dirtasks": "/opt/hadoop/conf", [ { "hadoop.home "id": "/opt/hadoop"0, "flush.size "state": "100RUNNING", "rotate.interval.msworker_id": "1000fakehost:8083" } |
We will add a `type` field to the array that indicates whether the given Connector is a Source or SInk. e.g.:
Code Block |
---|
HTTP/1.1 200 OK Content-Type: application/json { "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector", { "id": 1, "type": "Sink", "tasks.maxstate": "10FAILED", "topics": "test-topic", "hdfs.urlworker_id": "hdfs://fakehost:90008083", "hadoop.conf.dir": "/opt/hadoop/conf", "hadoop.hometrace": "/opt/hadoop",org.apache.kafka.common.errors.RecordTooLargeException\n" "flush.size": "100", } "rotate.interval.ms": "1000"] } |
For the following REST API endpoint:
Code Block |
---|
GET /connectors/(string:name)/status |
Proposed Changes
The aforementioned endpoint currently returns the following structureWe will add a `type` field to the response to indicate whether the given Connector is a Source or Sink. e.g.:
Code Block |
---|
HTTP/1.1 200 OK { "name": "hdfs-sink-connector", "connector": { "type": "sink", "state": "RUNNING", "worker_id": "fakehost:8083" }, "tasks": [ { "id": 0, "state": "RUNNING", "worker_id": "fakehost:8083" }, { "id": 1, "state": "FAILED", "worker_id": "fakehost:8083", "trace": "org.apache.kafka.common.errors.RecordTooLargeException\n" } ] } |
Compatibility, Deprecation, and Migration Plan
This
...
propsal only adds a field to existing JSON response messages from existing REST API endpoints, and does not otherwise change the structure of the responses. Many applications will be able to handle the response documents and ignore fields they don't know about. Therefore, this should be backwards compatible.
KIP-151 brought in the connector type enum which generates lowercase. This is kept in this KIP.
Though consumers of the API should not make assumptions about the capitalization of connector type.
Rejected Alternatives