THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Goals
- Improve the framework to provide support for discoverability when it comes to configure processors to connect to external systems.
- Provide a high level abstraction that can retrieve endpoints through a discoverability endpoint and have the ability to reference those endpoints in the configuration of components
Example:
- One would provide the URL to Confluent Cloud control plane with proper authentication details
- This would retrieve the available Kafka clusters, with the available authentication methods, as well as the topics
- In a Kafka processor, one could select the discoverability service, so that the user can select the cluster to connect to, the topic to consume/publish from/to, etc.
Background and strategic fit
This concept is quite similar to parameter providers except that:
- it could create and provide controller services to be used / referenced in components
- it would require the ability to define the list of Allowable Values from another component (instead of something hard coded in the component's code)
Requirements
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| 1 | Discoverability Endpoints | The user must be able to define a discoverability endpoint that will be in charge of retrieving the relevant elements | Must Have | |
| 2 | Dynamic Listing of Allowable Values | The discoverability endpoint should be refreshable (similar to parameter providers) and this would update the list of allowable values where the components/values created by the discoverability endpoint can be referenced (example list of Kafka topics) | Must Have | |
| 3 | CI/CD consideration | Reference to the components should accept custom ID to support CI/CD pipelines. Imagine the following: NiFi in DEV - Confluent Cloud discoverability offers "Kafka Dev" cluster and "my_topic_dev" topic that is used in ConsumeKafka processor NiFi in PROD - Confluent Cloud discoverability offers "Kafka Prod" cluster and "my_topic_prod" topic that is used in ConsumeKafka processor When deploying the flow definition from DEV to PROD, it should be required to only change the discoverability endpoint, nothing else. In other words, when defining the discoverability endpoint and "refreshing" it, it should be possible to associate a custom name to the elements, so that this name can be used in the flow definition across multiple deployments. It means that it would be required from a user (or automated deployment process) to first define the discoverability endpoint and the mapping of the discovered elements before deploying the flow definition (similar to parameter providers and derived parameter contexts). |
User interaction and design
Questions
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|