THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
This page documents all of the KIPs related to Connect since AK 0.10.0.0 (the release after Connect was first introduced in 0.9.0.0), and identifies the default behaviors that may make sense to change in AK 3.0.0. See the list of release plans for the list of all KIPs.
Most (if not all) Connect changes to date have been backward compatible with respect to:
- worker configurations
- log configuration
- connector configurations
- connector implementations
This means that users that are operating Connect can easily do a rolling upgrade of their Connect clusters without having to change their worker configurations, Log4J configurations, or any persisted connector configurations and without having to upgrade any connector implementations. Likewise, users operating Connect can choose when to upgrade individual connector implementations.
The changes proposed below continue to avoid requiring changes to persisted connector configurations or requiring upgrades of connector implementations. Many of the proposed changes have been recommended for some time, but making them in the AK 3.0.0 major release may require some users to make small changes to their worker configuration and Log4J worker configurations during the upgrade process.
This page is not a KIP.
Discussion thread: here
| AK Release | KIP | Default Behavior | Desired Change for AK 3.0.0 | Deprecation KIP (if different) | Jira Issue |
|---|---|---|---|---|---|
| 0.10.0.0 | KIP-51 - List Connectors REST API | Adds new REST resources | None | n/a | n/a |
| KIP-52: Connector Control APIs | Adds new REST resources | None | n/a | n/a | |
| KIP-56: Allow cross origin HTTP requests on all HTTP methods | Adds to the worker config the `access.control.allow.methods` property with a default that allows cross origin requests for GET, POST and HEAD | None | n/a | n/a | |
| 0.10.1.0 | KIP-65: Expose timestamps to Connect | Modifies the record interfaces to support timestamps | None | n/a | n/a |
| KIP-75 - Add per-connector Converters | Adds option of defining converters in connector configurations | None | n/a | n/a | |
| 0.10.2.0 | KIP-66: Single Message Transforms for Kafka Connect | Added SMTs to connector configurations | None | n/a | n/a |
| KIP-89: Allow sink connectors to decouple flush and offset commit | Changed sink task interfaces | None | n/a | n/a | |
| 0.11.0.0 | KIP-66: Single Message Transforms for Kafka Connect | optionally used in connector configs | None | n/a | n/a |
| KIP-128: Add ByteArrayConverter for Kafka Connect | optionally used in worker and/or connector configs | None | n/a | n/a | |
| KIP-146 - Classloading Isolation in Connect | behavior enabled by default | None | n/a | n/a | |
| KIP-151 Expose Connector type in REST API | change in selected REST responses | None | n/a | n/a | |
| KIP-154 Add Kafka Connect configuration properties for creating internal topics | defaults remain sensible | None | n/a | n/a | |
| 1.0.0 | KIP-196: Add metrics to Kafka Connect framework | added new metrics | None | n/a | n/a |
| KIP-197 Connect REST API should include the connector type when describing a connector | change in selected REST responses | None | n/a | n/a | |
| 1.1.0 | KIP-145 - Expose Record Headers in Kafka Connect | added new interfaces and APIs | None | n/a | n/a |
| KIP-208: Add SSL support to Kafka Connect REST interface | SSL is not enabled by default. KIP-573 changed the Deprecates ` | Remove the ` We will not enable SSL by default since that requires the user to set up a keystore and truststore, which Connect can't do by default. | Removal already approved in KIP-208 |
| |
| KIP-212: Enforce set of legal characters for connector names | new connector names must only use legal characters, and may not have leading and trailing whitespace. | None | n/a | n/a | |
| KIP-215: Add topic regex support for Connect sinks | added `topics.regex` sink connector configuration property. | None. Removing the `topics` sink connector configuration property would be invasive, and it has not been deprecated. | n/a | n/a | |
| KIP-238: Expose Kafka cluster ID in Connect REST API | change in selected REST responses | None | n/a | n/a | |
| 2.0.0 | KIP-174 - Deprecate and remove internal converter configs in WorkerConfig | deprecated these two worker configs, stating that they will be removed in a subsequent major release. | Remove the deprecated `internal.key.converter` and `internal.value.converter` worker configuration properties. Connect has been logging usage of these as warnings for some time, and it is likely using something other than the defaults is fragile and may not work due to the assumption in Connect. | Already approved in KIP-174, though removal explicitly approved in KIP-738: Removal of Connect's internal converter properties |
|
| KIP-285: Connect Rest Extension Plugin | optionally used in worker config | None | n/a | n/a | |
| KIP-297: Externalizing Secrets for Connect Configurations | optionally used in worker config | None | n/a | n/a | |
| KIP-298: Error Handling in Connect | The default behavior in Connect remains to kill the task on the first error in any stage. To change this behavior, several worker configurations must be enabled; the defaults are to not enable error logging, not include message details, and to not use a DLQ topic. | None. Error handling and logging should remain opt-in. | n/a | n/a | |
| KIP-305: Add Connect primitive number converters | Added new converter implementations, which are optionally specified in worker and connector configurations. | None | n/a | n/a | |
| 2.1.0 | None | None | None | n/a | n/a |
| 2.2.0 | None | None | None | n/a | n/a |
| 2.3.0 | KIP-411: Make default Kafka Connect worker task client IDs distinct | Changed distributed worker configuration's default for `client.id`. | None | n/a | n/a |
| KIP-415: Incremental Cooperative Rebalancing in Kafka Connect | Changed worker configuration's default for `connect.protocol`, and added several other worker configurations. | None | n/a | n/a | |
| KIP-449: Add connector contexts to Connect worker logs | Defined ability to enable connector contexts in Log4J logs, but did not enable this by default. | Change the `connect-log4j.properties` file to enable connector contexts in Log4J logs by default. | KIP-721: Enable connector log contexts in Connect Log4j configuration |
| |
| KIP-454: Expansion of the ConnectClusterState interface | Expanded ConnectClusterState interface, added new interface, and changed selected REST responses. | None | n/a | n/a | |
| KIP-458: Connector Client Config Override Policy | Worker config sets connector.client.config.override.policy=None by default to maintain backward compatibility. | Change the default for `connector.client.config.override.policy` to `All`, still allowing connector configurations to override client behavior. | KIP-722: Enable connector client overrides by default |
| |
| KIP-465: Add Consolidated Connector Endpoint to Connect REST API | Adds optional query parameter to /connectors endpoint | None | n/a | n/a | |
| 2.4.0 | KIP-481: SerDe Improvements for Connect Decimal type in JSON | Added to JSON converter configuration a `decimal.format` property that defaults to `BASE64`, and that can optionally be set to `NUMERIC` | None. Even though NUMERIC is probably a better default, changing this might require users to change potentially many existing persisted connector configurations before upgrading. This is not worth changing. | n/a | n/a |
| KIP-495: Dynamically Adjust Log Levels in Connect | Adds new `/admin` REST endpoints. | None | n/a | n/a | |
| KIP-507: Securing Internal Connect REST Endpoints | Changed worker configuration's default for `connect.protocol`, and added several other worker configurations. | None | n/a | n/a | |
| KIP-521: Enable redirection of Connect's log4j messages to a file by default | Modified default Log4J configuration to also write worker logs to a file, in addition to the console. | None | n/a | n/a | |
| 2.5.0 | KIP-558: Track the set of actively used topics by connectors in Kafka Connect | Adds new worker configuration properties, where behavior is enabled by default | None | n/a | n/a |
| 2.6.0 | KIP-131 - Add access to OffsetStorageReader from SourceConnector | Allowed source tasks to get OffsetStorageReader via task context. | None | n/a | n/a |
| KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics | Added topic.creation.enable worker configuration option, defaulting to `true`. Adds multiple optional connector configuration properties with sensible defaults. | None | n/a | n/a | |
| KIP-437: Custom replacement for MaskField SMT | Expanded behavior of MaskField SMT to allow replacement literals. | None | n/a | n/a | |
| KIP 577: Allow HTTP Response Headers to be Configured for Kafka Connect | Added `response.http.headers.config` worker configuration, defaults to empty string. | ???? | |||
| KIP-585: Filter and Conditional SMTs | Added new optional properties that can be used in connector configuration | None | n/a | n/a | |
| KIP-586: Deprecate commit records without record metadata | Deprecates the SourceTask.commitRecord(SourceRecord record) method. | None. We could remove the method, but that would prevent installing and running any connector implementation that still uses that method. Keeping the deprecated method is little effort yet still maximizes compatibility for connector implementations. | n/a | n/a | |
| KIP-605: Expand Connect Worker Internal Topic Settings | Modifies the allowed values of several existing worker configuration properties, and adds optional worker configuration properties for topic settings. | None | n/a | n/a | |
| KIP-606: Add Metadata Context to MetricsReporter | Modifies the MetricsReporter extension interface for Connect, Streams, and Broker. | None | n/a | n/a | |
| KIP-610: Error Reporting in Sink Connectors | Adds the ability for a sink task to get an error reporter | None | n/a | n/a | |
| 2.7.0 | None | None | None | n/a | n/a |
| 2.8.0 | KIP-661: Expose task configurations in Connect REST API | Adds `/{connector}/tasks-config` endpoint that returns configuration of connector's tasks. Does not add any worker or connector configurations. | None | n/a | n/a |
defaults may make sense to change. for which defaults were chosen to maintain backward compatibility, but which may be with a major upgrade