...
Public Interfaces
A new optional field "initial_state"
will be added to the request body format for the POST /connectors
endpoint (the existing format can be seen in Kafka Connect's current OpenAPI specification). The new optional field "initial_state"
can have a value of "RUNNING", "PAUSED"
or "STOPPED"
(case-insensitive). If the value of the "initial_state"
field is invalid, a 400 Bad Request
response will be returned. If the field is omitted in the request body, the connector will be created in the RUNNING
state by default (preserving the existing behavior). An example request body would look like:
Code Block | ||||
---|---|---|---|---|
| ||||
{
"name": "file-sink-test",
"config": {
"connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
"file": "test.txt",
"topics": "test-topic",
"tasks.max": "1"
},
"initial_state": "STOPPED"
} |
Note that when a connector is created in a PAUSED
or STOPPED
state, no tasks will be spawned for the connector until it is resumed via the PUT /connectors/{connector}/resume
endpoint.
...
When the leader Connect worker (distributed mode only) receives a create connector request (such requests made to non-leader workers are first forwarded to the leader internally) with a valid non-null "initial_state"
field, it will write the appropriate target state to the config topic along with the connector configuration. If the Connect cluster is configured to use a transactional producer to write to the config topic (see KIP-618), the target state and the connector configuration will be written atomically in a single transaction. If the Connect cluster is not configured to use a transactional producer to write to the config topic, the target state will be written to the config topic before the connector configuration. This would ensure that in the unlikely but not impossible case that the leader worker dies between writing the two messages to the config topic, the connector won't be brought up in the wrong state. Target states for unknown connectors are essentially ignored so this order of writes is safer overall.
...
The changes in standalone mode will be similar - the connector's target state will be updated prior to the connector being started if a valid "initial_state"
is specified in the connector creation request.
...
Summary: The current target states that can be written to the config topic are STARTED
(when a connector is resumed), PAUSED
and STOPPED
and we could use these as valid values for the new "initial_state"
field in the connector creation request body.
Rejected because: These target states aren't currently directly exposed to the users and the states that are actually exposed to the users via the GET /connectors/{connector}/status
endpoints are: UNASSIGNED
, RUNNING
, PAUSED
, FAILED
, RESTARTING
and STOPPED
. Using RUNNING
over STARTED
as a valid value for the new "initial_state"
field in the connector creation request body makes more sense since that's the state that users are already familiar with (the PAUSED
and STOPPED
states are common across and aren't an issue).
...
Rejected because: It isn't particularly useful to allow modifying the state of a connector along with updating its configuration since both those operations can already be done separately (whereas with the connector creation case, it isn't currently possible to modify the state of a non-existent connector) and the POST /connectors
endpoint already covers the connector creation case. Furthermore, the PUT /connectors/{connector}/config
endpoint currently accepts a map of connector key-value configurations as its request body (OpenAPI specification for reference), and adding a new "initial_state"
field (or equivalent) field would require significantly modifying the request body format and supporting two different request body formats for the endpoint. We could potentially use a request parameter or request header to specify the state, but it would be fairly unusual to have two completely different ways of specifying the connector state across two endpoints (and it seems more appropriate to include the state in the request body anyway since it's a part of the request itself rather than metadata or a modifier).
...
Rejected because: The PAUSED
state is still a valid connector state and has been present for a lot longer than the STOPPED
state ( PAUSED
was introduced in AK 0.10.0.0 and STOPPED
was introduced in AK 3.5.0). It would likely be more confusing for users if we treated PAUSED
as an invalid "initial_state
in "
in the connector creation request body.