Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Status

...

Page properties


Discussion thread

...

...

Vote threadhttps://

...

...

9jq2fbo3r9lrm9y7cvgyw4h64vqxvjy7
JIRA

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyFLINK-25555

Release1.15


Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

We reduced the implementation overhead in Sink V2 and split the Sink V1 into three mixin interfaces providing different functionalities. Sink developers can not decide now decide based on their requirements which sink interfaces they need to combine to implement their sinks.

...

We envision the implementation of certain connectors can be done with the following interfaces.

ConnectorSinkStatefulSinkTwoPhaseCommittingSinkWithPreWriteTopologyWithPreCommitTopologyWithPostCommitTopology
Kafka(tick)(tick)(tick)


File(tick)(tick)(tick)


Elasticsearch(tick)




Async(tick)(tick)



Hive(tick)(tick)(tick)((tick))(tick)
Iceberg(tick)(tick)(tick)(tick)
(tick)

Hive and Iceberg will leverage small file compaction. Though both connectors implement it differently. Hive will try to do a best-effort compaction before actually making the files visible in the metastore because it does not support updates. On the other hand, the Iceberg connector writes the files immediately and the post topology will take of compacting the already written files and updating the file log after the compaction.

...