View Source

Target release	Release name or number
Epic	Link to related Jira epic or feature
Document status
Document owner	Unknown User (pvillard)
Designer	Lead designer
Developers	Lead developer
QA	Lead tester

Goals

Improve the framework to provide support for discoverability when it comes to configure processors to connect to external systems.
Provide a high level abstraction that can retrieve endpoints through a discoverability endpoint and have the ability to reference those endpoints in the configuration of components

Example:

One would provide the URL to Confluent Cloud control plane with proper authentication details
This would retrieve the available Kafka clusters, with the available authentication methods, as well as the topics
In a Kafka processor, one could select the discoverability service, so that the user can select the cluster to connect to, the topic to consume/publish from/to, etc.

Background and strategic fit

This concept is quite similar to parameter providers except that:

it could create and provide controller services to be used / referenced in components
it would require the ability to define the list of Allowable Values from another component (instead of something hard coded in the component's code)

Requirements

#

Title

User Story

Importance

Notes

1

Discoverability Endpoints

The user must be able to define a discoverability endpoint that will be in charge of retrieving the relevant elements

Must Have

Additional considerations or noteworthy references (links, issues)

2

Dynamic Listing of Allowable Values

The discoverability endpoint should be refreshable (similar to parameter providers) and this would update the list of allowable values where the components/values created by the discoverability endpoint can be referenced (example list of Kafka topics)

Must Have

3

CI/CD consideration

Reference to the components should accept custom ID to support CI/CD pipelines. Imagine the following:

NiFi in DEV - Confluent Cloud discoverability offers "Kafka Dev" cluster and "my_topic_dev" topic that is used in ConsumeKafka processor

NiFi in PROD - Confluent Cloud discoverability offers "Kafka Prod" cluster and "my_topic_prod" topic that is used in ConsumeKafka processor

When deploying the flow definition from DEV to PROD, it should be required to only change the discoverability endpoint, nothing else.

In other words, when defining the discoverability endpoint and "refreshing" it, it should be possible to associate a custom name to the elements, so that this name can be used in the flow definition across multiple deployments. It means that it would be required from a user (or automated deployment process) to first define the discoverability endpoint and the mapping of the discovered elements before deploying the flow definition (similar to parameter providers and derived parameter contexts).

User interaction and design

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Question	Outcome

Not Doing

List the features discussed which are out of scope or might be revisited in a later release.