THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Performance Test Results
We have tested the performance of the initial implementation of this proposal.
...
Workload-1 (at-least-once, acks=all) | Workload-2 (acks=1) | |
|---|---|---|
| Producers | 10 producers 30MB / sec / broker (leader) | 10 producers 55MB / sec / broker (leader) |
| In-sync Consumers | 10 consumers 120MB / sec / broker (leader) | 10 consumers 220MB / sec / broker (leader) |
Test case 1 (Normal case):
No remote storage read, and no broker rebuldNormal traffic as described above.
| with tiered storage | without tiered storage | ||
|---|---|---|---|
Workload-1 (acks=all, low traffic) | Avg P99 produce latency | 25ms | 21ms |
| Avg P95 produce latency | 14ms | 13ms | |
Workload-2 (acks=1, high traffic) | Avg P99 produce latency | 9ms | 9ms |
| Avg P95 produce latency | 4ms | 4ms |
...
| with tiered storage | without tiered storage | ||
|---|---|---|---|
Workload-1 (acks=all, 12TB data per broker) | Max avg P99 produce latency | 56ms | 490ms |
| Max avg P95 produce latency | 23ms | 290ms | |
| Duration | 2min | 230ms | |
Workload-2 (acks=1, 34TB data per broker) | Max avg P99 produce latency | 12ms | 10ms |
| Max avg P95 produce latency | 6ms | 5ms | |
| Duration | 4min | 520min |
With tiered storage, the rebuilt rebuilding broker only need needs to fetch the latest data that has not been shipped to remote storage from the other brokers. Without tiered storage, the rebuilt broker has to fetch all the data that has not expired from the other brokers. With the same log retention time, tiered storage reduced the rebuild rebuilding time by more than 100 times.
Without tiered storage, the rebuilt rebuilding broker has to read a large amount of data from the local hard disks of the leaders. This competes page cache and local disk bandwidth with the normal traffic, and dramatically increases the acks=all produce latency.
...