...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
POC: https://github.com/apache/flink/pull/18541
Motivation
Since the time Opensearch got forked from Elasticsearch a few things got changed. The projects evolve in different directions, the Elasticsearch clients up to 7.13.x were able to connect to Opensearch clusters, but since 7.14 - not anymore (Elastic continues to harden their clients to connect to Elasticsearch clusters only, https://github.com/elastic/elasticsearch/pull/73910). Arguably, the best way to proceed in this situation is to provide dedicated Opensearch connectors, on par with Elasticsearch ones.
...
- Integrate with Flink Sink API (FLIP-143)
- Integrate with Flink new Source API (FLIP-27)
- Build upon Flink New DynamicTableSink and DynamicTableSinkFactory interfaces (FLIP-95)
- Sink streaming results with checkpointing enabled (at-least-once delivery semantics)
- Integrate with Table API
Proposed Changes
We propose to introduce dedicated Opensearch connectors (for 1.x and 2.x release lines) in Flink, based on existing Elasticsearch 7 connectors (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/elasticsearch/).
...
The connectors are compatible with Opensearch 1.x and 2.x release lines. With respect to Flink, this is new feature, no compatibility, deprecation, and migration plan is expected.
Test Plan
The integration test cases along with end-2-end test suites have been implemented, on par with Elasticsearch connectors test coverage.
...