...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In order to start When starting a connector, users must provide the connector configuration. The configuration often also includes configurations for other plugins such as SMTs or converters. Today, Connect does not provide a way to see what configurations a connector requires. Instead users have to plugins are available apart from connectors (via the /connector-plugins endpoint). This make it difficult for users building data pipeline to know which plugins are available and what is possible. Basically they have to know how the Connect runtime is set up. Even once they know the plugins that are available, they then have to go look at the connector plugins documentation or, in the worst case, look directly at the connector source code . The proposal is to add an endpoint to the Connect REST API that exposes the configuration definitions of connectors.
This new endpoint will allow 2 things:

• users will be able to discover the required configurations for connectors installed in a Connect cluster
• tools will be able to generate wizards for configuring and starting connectors

Public Interfaces

A new Connect REST API endpoint: /connector-plugins/{connector}/configdef available via GET that returns the configuration definitions of the specified connector.

...

to find their configuration definitions.

All plugins that are directly usable should be discoverable via the REST API. Their configuration definitions should also be easily retrieved. This would significantly ease the process of building pipelines and enable building tools and UIs that can manage Connect data pipelines.

Public Interfaces

The proposal is to add 2 new endpoints to address these limitations:

• /plugins: This will return all plugins that are Connectors, Transformations, Converters, HeaderConverters and Predicates. Plugins will be grouped by plugin.path. This will make it clear to users what's available to use as it's not possible to use a Connector from one path with Transformations from another.

Example output with the runtime plugin.path set to plugin.path=/Users/mickael/tmp/path1/,/Users/mickael/tmp/path2/:

{
  "classpath": [
    {
      "class": "org.apache.kafka.connect.transforms.DropHeaders",
      "type": "transformation",
      "location": "classpath"
    },
    {
      "class": "org.apache.kafka.connect.transforms.predicates.HasHeaderKey",
      "type": "predicate",
      "location": "classpath"
    },
    {
      "class": "org.apache.kafka.connect.converters.ByteArrayConverter",
      "type": "converter",
      "location": "classpath"
    },
    {
      "class": "org.apache.kafka.connect.storage.SimpleHeaderConverter",
      "type": "header_converter",
      "location": "classpath"
    }
  ],
  "/Users/mickael/tmp/path1/": [
    {
      "class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
      "type": "sink",
      "location": "file:/Users/mickael/tmp/path1/connect-file-3.0.0.jar"
    }
  ],
  "/Users/mickael/tmp/path2/": [
    {
      "class": "com.github.jcustenborder.kafka.connect.transform.common.BytesToString$Key",
      "type": "transformation",
      "location": "file:/Users/mickael/tmp/path2/kafka-connect-transform-common-0.1.0.14.jar"
    }
  ]
}

• /plugins/<type>/<name>/configdef: This will return the configdef for the specified plugin. Type can be either sink, source, transformation, predicate, converter or header_converter. Name must be the fully qualified class name of the plugin.

For example, accessing http://localhost:8083/plugins/transformation/org.apache.kafka.connect.transforms.Cast$Value/configdef will return:

[
  {
	    "name": "filespec",
	    "type": "STRINGLIST",
	    "required": falsetrue,
	    "default_value": null,
	    "importance": "HIGH",
	    "documentation": "Destination filename. If not specified, the standard output will be used",
	List of fields and the type to cast them to of the form field1:type,field2:type to cast fields of Maps or Structs. A single type to cast the entire value. Valid types are int8, int16, int32, int64, float32, float64, boolean, and string. Note that binary fields can only be cast to string.",
    "group": null,
	    "width": "NONE",
	    "display_name": "filespec",
	    "dependents": [],
	    "order": -1
  }
]

This will reuse the ConfigKeyInfo entity which is already exposed via PUT /connector-plugins/{connector-type}/config/validate.

Proposed Changes

A new endpoint REST resource named PluginsResource will be defined in ConnectorPluginsResource.javaand it will expose the following 2 endpoints under /plugins:

@GET
@Path("/")
public Map<String, List<PluginInfo>> listPlugins() {
    return doListPlugins();
}

@GET
@Path("/{type}/{connectorTypeplugin}/configdef")
public List<ConfigKeyInfo> getConnectorPluginConfigDef(getConnectorConfigDef(final @PathParam("type") String type, final @PathParam("connectorTypeplugin") String connTypeplugin) {
    
return herder.connectorPluginConfigDef(connTypedoGetConfigDef(type, plugin);
}

A new method will be added to Herder.java to provide the functionnality.

List<ConfigKeyInfo> connectorPluginConfigDef(String connType);

Compatibility, Deprecation, and Migration Plan

This KIP is a proposing new endpointendpoints, existing endpoints are not changed.

Rejected Alternatives

None