...
We reduced the implementation overhead in Sink V2 and split the Sink V1 into three mixin interfaces providing different functionalities. Sink developers can not decide now decide based on their requirements which sink interfaces they need to combine to implement their sinks.
...
We envision the implementation of certain connectors can be done with the following interfaces.
Connector | Sink | StatefulSink | TwoPhaseCommittingSink | WithPreWriteTopology | WithPreCommitTopology | WithPostCommitTopology |
---|---|---|---|---|---|---|
Kafka | ||||||
File | ||||||
Elasticsearch | ||||||
Async | ||||||
Hive | ( | |||||
Iceberg |
Hive and Iceberg will leverage small file compaction. Though both connectors implement it differently. Hive will try to do a best-effort compaction before actually making the files visible in the metastore because it does not support updates. On the other hand, the Iceberg connector writes the files immediately and the post topology will take of compacting the already written files and updating the file log after the compaction.
...