Status
Current state: Under Discussion
Discussion thread:
JIRA: FLINK-25756 - Dedicated Opensearch connectors
Released: <Flink Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Since the time Opensearch got forked from Elasticsearch a few things got changed. The projects evolve in different directions, the Elasticsearch clients up to 7.13.x were able to connect to Opensearch clusters, but since 7.14 - not anymore (Elastic continues to harden their clients to connect to Elasticsearch clusters only, https://github.com/elastic/elasticsearch/pull/73910). Arguably, the best way to proceed in this situation is to provide dedicated Opensearch connectors, on par with Elasticsearch ones.
Proposal
We propose to introduce dedicated Opensearch connectors (for 1.x and 2.x release lines) in Flink, based on existing Elasticsearch 7 connectors (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/elasticsearch/).
Scope
The Opensearch connectors will support:
- Integrate with Flink Sink API (FLIP-143).
- Build upon Flink New DynamicTableSink and DynamicTableSinkFactory interfaces (FLIP-95)