This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka connect currently supports running a single version of a connector. As the externals system that a connector supports evolves over time, the connectors may need to make incompatible changes between versions to keep up with new capabilities. Without the ability to run multiple versions of a connector, we would need to run the connectors in two different connect clusters if we want to support both version simultaneously. In addition, support for multiple connectors makes it possible to do two phase upgrades to connectors with the benefit of easier rollbacks. A connector runtime restart will still be required to install a newer version, however existing connectors can still continue to use the older version. The migration to the newer version is decoupled from the installation of the connector and will not require a cluster restart. The decoupling also makes rollbacks easier and will no longer require a cluster rollback. Finally, multiple version eases development toil while making changes to a connector. 

Public Interfaces

Connector version will be provided using a new connector configuration parameter, named connector.version . The naming scheme can be debated but it should be indicative of the context in which it applies and not have potential for existing connector conflicts. The connect runtime will interpret this the new config in the following ways. 

...

The connector-plugins endpoint that retrieves the config of a connector, /{plugin-name}/config  will be updated to include a version query parameter which will default to latest. 

Proposed Changes

Running multiple versions requires two primary capabilities to be present in the runtime.

...

The work here is augmentative, primarily to the plugin loading mechanism. The set of changes is more or less laid out in this draft PR I did as a POC. 

The existing connector-plugins  endpoint list lists all version the versions of a connector. A user can easily identify which version of a connector is currently running through this, when not directly present in the configs. 

Compatibility, Deprecation, and Migration Plan

Any existing connector without version information in the configuration will continue to default to using the latest version available in the plugins path. Any cluster updates with a later version of the connector installed will result in the connector also getting updated to the latest version. This preserves the existing behaviour. 

A connector can be updated to assign it a particular version, and beyond this point the connector will always run with the assigned version (provided the version is present) even across cluster upgrades. However cluster downgrades to older connector runtimes will end up using the older version of the connector. Since the assignment of version to a connector needs to be explicitly opted in by the user through a connector update, this limitation is easily evident. 

Test Plan

Integration and Unit tests will be added. 

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.