Status
Current state: Under Discussion
Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-72-Introduce-Pulsar-Connector-tc33283.html
...
Page properties | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
- Pulsar as a streaming source with exactly-once guarantee.
- Sink streaming results to Pulsar with at-leastwith exactly-once semantics.
- Build upon Flink new Table API Type systemNew TableSource and TableSink interfaces (FLIP-3795), and can automatically (de)serialize messages with the help of Pulsar schemametadata (FLIP-107).
- Integrate with Flink new Catalog API (FLIP-30), which enables the use of Pulsar topics as tables in Table API as well as SQL client.
- Integrate with Flink new Source API (FLIP-27).
- Integrate with Flink new Sink API (FLIP-143).
- Integrate with Flink upsert-pulsar (FLIP-149).
Overall Design
Source: The exactly-once source should implement CheckpointedFunction to persist reading status (MessageId in Pulsar which uniquely identifies a message in topic) to state store, and could notify Pulsar of messages finished consumption by implementing CheckpointListener.
...