Page History

...

This is much needed for SourceConnector that talk to external-system that maintain some watermark (for cleanup). These connector maintain offsets in OffsetTopic + signal low-watermark to external systems. Generally, low-watermark cannot be sent out for every record, and publishing offsets at record level is more suitable. When task failsis restarted, it must start from last committed offset. So, watermark must always be lower than last-committed-offset. Thus watermark must only be updated based on latest committed offset (and not based on latest SourceRecords published SourceRecord to KafkaTopic, which can be tracked via commitRecord).

Example: Debezium PostgresSourceConnector

...