...

• Transactional guarantees: the streamer should not bypass any transaction mechanisms. Transaction per batch or transaction per entry could be used.
• Asynchronous API with backpressure control. Avoid blocking threads and/or excessive resource usage on clients and servers.
• Ordering guarantees: the streamer must guarantee that two objects added with the same key will be processed in the same order they were added by the client. In other words, the streamer can’t reorder items within a given partition. Receiver should not be invoked for item B if preceding item A is not yet processed.
• At Least Once is a must. Exactly Once is nice to have if technically feasible (probably not).delivery guarantee:
  • Automatic retry of failed batches.
  • If a streamer completes without error, all items are guaranteed to be processed.
• Partition awareness (best effort): receiver should run on a node where the given row is located. However, Ignite 3 does not provide partition pinning, so during topology changes this behavior is not guaranteedPartition pinning. Because the receiver usually expects the data to be colocated, we should prevent partitions from changing ownership while the receiver is running.

Description

Receiver Deployment

Receiver class should be deployed on all server nodes the cluster prior to data streamer usage - see IEP-103: Code Deployment.

Reactive Java API

Java 9 Reactive Streams is a standard API for asynchronous stream processing (initially inspired by Akka Streams). Our API will accept Flow.Publisherfrom the user and pull the data from there as needed, at the speed of server-side ingestion and processing.

...