Status

Current state: Under Discussion

Discussion thread:

JIRA:  FLINK-25756 - Dedicated Opensearch connectors

Released: <Flink Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Since the time Opensearch got forked from Elasticsearch a few things got changed. The projects evolve in different directions, the Elasticsearch clients up to 7.13.x were able to connect to Opensearch clusters, but since 7.14 - not anymore (Elastic continues to harden their clients to connect to Elasticsearch clusters only, https://github.com/elastic/elasticsearch/pull/73910). Arguably, the best way to proceed in this situation is to provide dedicated Opensearch connectors, on par with Elasticsearch ones.

Public Interfaces

The  Opensearch connectors will support:

• Integrate with Flink Sink API (FLIP-143)
• Integrate with Flink new Source API (FLIP-27)
• Build upon Flink New DynamicTableSink and DynamicTableSinkFactory interfaces (FLIP-95)

Proposed Changes

We propose to introduce dedicated Opensearch connectors (for 1.x and 2.x release lines) in Flink, based on existing Elasticsearch 7 connectors (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/elasticsearch/).

Compatibility, Deprecation, and Migration Plan

The connectors are compatible with Opensearch 1.x and 2.x release lines.

Test Plan

Rejected Alternatives

Keep using existing Elasticsearch to connect to Opensearch cluster would not work.