Status

Current state: Under discussion.

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka Connect is a highly extensible framework, designed to run software modules that integrate a wide variety of data sources and destinations with Kafka and, by definition, are developed independently of the framework. Examples of such modules are connectors, transformations and data converters. Because such independent development, dependency conflicts might arise naturally, between the framework and the modules, as well as among the modules themselves. This happens when, for instance, one connector is using the latest version of a package that added a new feature, while another connector depends on a previous version of the same package where this feature is missing. 

...

In this case, the framework will do the heavy lifting in terms of development, offering a transparent resolution of dependency conflicts and keeping the changes in Kafka Connect's public interfaces minimal. 

Public Interfaces

The only publicly visible changes that are proposed in order to implement class loading isolation in Kafka Connect is the addition of two new Connect worker config properties:

...

While the introduction of these two new configuration properties are the only changes proposed to the public interfaces of Kafka Connect, next are also described the main implementation steps that will be carried out within the framework to support class loading isolation.

Proposed Changes

Add module.path and module.isolation.enabled configuration properties for workers running both in standalone and distributed mode. By default module.path is empty and module.isolation.enabled is set to false.

...

• the Connect framework controls the threads that run module code (e.g. connector tasks).
• module classes and dependencies are required to be supplied explicitly through module.path.

Compatibility, Deprecation, and Migration Plan

• Existing users will not be impacted since isolation will be disabled by default. 
• Users enabling class loading isolation might experience higher demands in memory usage due to additional loading of otherwise common classes. However this increase is not expected to be prohibitive in most cases.

Test Plan

• Targeted Unit tests will be developed to test the components that will implement class isolation in the framework. Additionally, all other tests will be set to run with class loading isolation turned on. 
• System tests will be written to test class loading isolation when explicitly conflicting dependencies are introduced by connectors. 
• Microbenchmarks will be designed to make sure the effect of classloader context switching is negligible. 

Rejected Alternatives

• Use OSGi. OSGi. Along with its much broader scope, an OSGi implementation would bring significant implementation complexity, both in the framework and the connector development. 

• Design for project Jigsaw and wait. With this KIP we describe an implementation path whose execution is expected to be focused and efficient. Major upgrades, such as an upgrade to the next Java version are not expected to move faster than the proposed implementation.

...