DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Kafka Connect provides a rich ecosystem of pluggable components, including Connectors, Converters, Transformations, and Predicates.
Each of these component types exposes configuration metadata through a config() method that returns a ConfigDef object describing the configuration properties the component accepts.
However, this config() method is currently defined independently across multiple interfaces and classes:
| Code Block | ||
|---|---|---|
| ||
public abstract class Connector implements Versioned {
public abstract ConfigDef config();
}
public interface Converter {
default ConfigDef config() {
return new ConfigDef();
}
}
public interface HeaderConverter extends Configurable, Closeable {
ConfigDef config();
}
public interface Transformation<R extends ConnectRecord<R>> extends Configurable, Closeable {
ConfigDef config();
}
public interface Predicate<R extends ConnectRecord<R>> extends Configurable, AutoCloseable {
ConfigDef config();
} |
This fragmentation creates several challenges:
Component Discovery: Tools that need to discover and introspect component configurations must search for multiple different interfaces/classes. There is no single, unified mechanism to identify all classes that provide configuration specifications.
Code Duplication: The same method signature is duplicated across multiple interfaces.
Extensibility: Future component types that require configuration specification must remember to implement the config() method independently, without compile-time enforcement of a common contract.
Tooling Complexity: External tools (such as schema generators, UI builders, validation frameworks, and documentation generators) must maintain separate logic to handle each component type, increasing maintenance burden and the risk of inconsistencies.
This improvement benefits several use cases:
- Schema Generation: Tools like Debezium's schema generator need to automatically extract configuration metadata from components to generate descriptors for UI applications.
- Configuration Validation: Build-time and runtime validation tools can uniformly discover and validate configurations across all component types.
- Documentation Generation: Automated documentation tools can consistently extract and present configuration options.
- Configuration Management UIs: Web interfaces for managing Kafka Connect deployments (like Debezium Platform) need to dynamically discover available components and their configuration requirements.
Public Interfaces
The idea is to introduce a new interface in the org.apache.kafka.connect.components package:
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
public interface Converter extends Closeable, ConfigSpecifier { /** * Configuration specification for this converter. * @return the configuration specification; may not be null */ @Override default ConfigDef config() { return new ConfigDef();} } |
public interface Converter extends Closeable, ConfigSpecifier { /** * Configuration specification for this converter. * @return the configuration specification; may not be null */ @Override default ConfigDef config() { return new ConfigDef();} }
| Code Block | ||||
|---|---|---|---|---|
| ||||
public interface Transformation<R extends ConnectRecord<R>> extends Configurable, Closeable, ConfigSpecifier {
/** Configuration specification for this transformation.
*/
@Override
ConfigDef config();
} |
...