...

• org.apache.flink.streaming.api.TimerService

  • registerProcessingTimeTimer vs registerEventTimeTimer

Rejected Alternatives

1) Use custom post and pre-commit topologies from the SinkV2 API proposed in FLIP-191 to avoid generating multiple small files.

FLIP-191 and FLIP-325 complement each other in addressing the small-file-compaction. They can not replace each other.

FLIP-191 provides the SinkV2 API so that sink developers can plugin custom logic to merge small files before making the files visible. However, without FLIP-325, sink operators have to trigger compaction and make the compacted files visible to downstream jobs everytime a checkpoint is triggered. When the frequency of making files visible to Hudi increases, the number of files in the Hudi will also likely increase (for the same amount of input data).

FLIP-325 can help reduce the number of files (and also increase sink throughput) by allowing Hudi Sink to optionally reduce the frequency of making data visible to Hudi when there is backlog.