Versions Compared

Old Version 16

changes.mady.by.user Till Rohrmann

Saved on

compared with

New Version 17

changes.mady.by.user Fabian Paul

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

We reduced the implementation overhead in Sink V2 and split the Sink V1 into three mixin interfaces providing different functionalities. Sink developers can not decide now decide based on their requirements which sink interfaces they need to combine to implement their sinks.

...

We envision the implementation of certain connectors can be done with the following interfaces.

ConnectorSinkStatefulSinkTwoPhaseCommittingSinkWithPreWriteTopologyWithPreCommitTopologyWithPostCommitTopology
Kafka(tick)(tick)(tick)


File(tick)(tick)(tick)


Elasticsearch(tick)




Async(tick)(tick)



Hive(tick)(tick)(tick)((tick))(tick)
Iceberg(tick)(tick)(tick)(tick)
(tick)

Hive and Iceberg will leverage small file compaction. Though both connectors implement it differently. Hive will try to do a best-effort compaction before actually making the files visible in the metastore because it does not support updates. On the other hand, the Iceberg connector writes the files immediately and the post topology will take of compacting the already written files and updating the file log after the compaction.

...