Status
Current state: Under Discussion
Discussion thread:
JIRA: FLINK-25756 - Dedicated Opensearch connectors
Released: <Flink Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Since the time Opensearch got forked from Elasticsearch a few things got changed. The projects evolve in different directions, the Elasticsearch clients up to 7.13.x were able to connect to Opensearch clusters, but since 7.14 - not anymore (Elastic continues to harden their clients to connect to Elasticsearch clusters only, https://github.com/elastic/elasticsearch/pull/73910). Arguably, the best way to proceed in this situation is to provide dedicated Opensearch connectors, on par with Elasticsearch ones.
Public Interfaces
The Opensearch connectors will support:
- Integrate with Flink Sink API (FLIP-143)
- Integrate with Flink new Source API (FLIP-27)
- Build upon Flink New DynamicTableSink and DynamicTableSinkFactory interfaces (FLIP-95)
Proposed Changes
We propose to introduce dedicated Opensearch connectors (for 1.x and 2.x release lines) in Flink, based on existing Elasticsearch 7 connectors (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/elasticsearch/).
Compatibility, Deprecation, and Migration Plan
The connectors are compatible with Opensearch 1.x and 2.x release lines.
Test Plan
Rejected Alternatives
Keep using existing Elasticsearch to connect to Opensearch cluster would not work.