DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
We propose making the broker-side deduplication buffer size configurable (or dynamic). This will allow operators to tune their clusters for high-latency environments by increasing the allowed max.in.flight.requests.per.connection for idempotent producers. This change will enable users to maintain exactly-once semantics and strict ordering without accepting a massive degradation in throughput over long-distance links.
Following is throughput difference among 1, 5, 10 max.in.flight.requests.per.connection. The server is in AWS us-east-1 and the producer is in AWS ap-northeast-1.
| max.in.flight.requests.per.connection | throughput | avg latency | max latency | 50th latency | 95th latency | 99th latency | 99.9th latency |
|---|---|---|---|---|---|---|---|
| 1 | 19.623234 records/sec | 25093.64 ms | 49788.00 ms | 25193 ms | 47286 ms | 49341 ms | 49788 ms |
| 5 | 95.129376 records/sec | 5111.89 ms | 10096.00 ms | 5135 ms | 9644 ms | 10095 ms | 10096 ms |
| 10 (change hard code value in code) | 183.083120 records/sec | 2558.52 ms | 5036.00 ms | 2528 ms | 4749 ms | 5034 ms | 5036 ms |
Public Interfaces
Broker Configuration
...