DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
The details of the calculation can be provided per request.
Appendix: Performance Data
Configuration
Cluster: 3 brokers
Topic perf_cloud_native: 2-way replicated, one leader (AZ: use1-az4) writes to cloud storage, one reader (AZ: use1-az6) reads from cloud storage (uses KIP-1176 code)
Topic perf_local_disk: 2-way replicated (for fair comparison), standard replication path, leader (AZ: use1-az4) writes to local disk, follower (AZ: use1-az6) reads via cross AZ-transfer
Each topic has 1 partition.
Results
End to End Latency
A custom script was run to determine end-to-end-latency, which is defined here as the difference between system time when a record is consumed and system time when a record is created in memory on the producer side. The other producer/consumer metrics are all standard.
Workload
100 records, 1.6 KB avg record size, max batch size 32kb, acks=1, consumer client.rack=use1-az6 (read from follower)
Metrics
| Metric | perf_local_disk | perf_cloud_native |
| Producer Metrics | ||
| batch-size-avg | 31019 | 31479 |
| batch-size-max | 32569 | 31481 |
| record-queue-time-avg | 8.4 | 8.2 |
| record-queue-time-max | 10 | 10 |
| record-send-total | 100 | 100 |
| record-size-avg | 1625 | 1648 |
| record-size-max | 1625 | 1648 |
| records-per-request-avg | 20 | 20 |
| request-latency-avg | 7 | 6.6 |
| request-latency-max | 9 | 9 |
| byte-total | 155095 | 157395 |
| Consumer Metrics | ||
| bytes-consumed-total | 154790 | 157090 |
| fetch-latency-avg | 470.5294118 | 470.3125 |
| fetch-latency-max | 507 | 505 |
| fetch-size-avg | 38697.5 | 52363.33333 |
| fetch-size-max | 97514 | 157090 |
| fetch-total | 17 | 16 |
| records-consumed-total | 100 | 100 |
| records-per-request-avg | 25 | 33.33333333 |
End-to-End Latency Statistics (ms) | ||
| Mean (Average) | 708.9731 | 919.7159 |
| Min Latency | 523.6274 | 915.0023 |
| Max Latency | 1020.9279 | 942.0901 |
| p50 (Median) | 528.1985 | 919.2508 |
| p75 | 1019.5718 | 921.7726 |
| p90 | 1020.3517 | 924.0798 |
| p99 | 1020.9279 | 942.0901 |
| p99.9 | 1020.9279 | 942.0901 |
Kafka Producer/Consumer Perf Test Scripts
Workload
Used kafka/bin/kafka-producer-perf-test.sh and kafka/bin/kafka-consumer-perf-test.sh.
Producer: --num-records 100000 --record-size 2000 --throughput -1 --producer-props acks=1 retries=2 linger.ms=20 delivery.timeout.ms=300000 request.timeout.ms=30000
Consumer: --messages 100000 --timeout 240000 --consumer.config consumer.properties
The consumer.properties file contained client.rack=use1-az6 (the rack of the follower broker, so we consumed from the follower).
Metrics
| Metric | perf_local_disk | perf_cloud_native |
Producer Metrics | ||
| Records sent | 100,000 | 100,000 |
| Records/sec | 20,542.32 | 20,772.75 |
| Throughput (MB/sec) | 39.18 | 39.62 |
| Avg Latency (ms) | 638.39 | 635.4 |
| Max Latency (ms) | 817 | 760 |
| 50th Percentile | 667 | 666 |
| 95th Percentile | 745 | 710 |
| 99th Percentile | 802 | 746 |
| 99.9th Percentile | 815 | 758 |
Consumer Metrics | ||
| Data Consumed (MB) | 190.73 | 190.73 |
| Throughput (MB/sec) | 44.56 | 40.83 |
| Records Consumed | 100,000 | 100,000 |
| Records/sec | 23,364.49 | 21,404.11 |
| Rebalance Time (ms) | 3,635 | 3,961 |
| Fetch Time (ms) | 645 | 711 |
| Fetch MB/sec | 295.71 | 268.26 |
| Fetch Records/sec | 155,038.76 | 140,646.98 |
Analysis
Overall we see comparable performance between a standard topic and KIP-1176 for both producer & consumer metrics, in particular throughput and latency. The producer path is near identical performance, i.e., an ack to the producer is returned in the same time. Consumer performance is slightly slower as data must travel an additional hop to/from S3E1Z. There are additional tunings we are aware of that could further boost performance (e.g., tuning the interval that the leader/follower remote WAL tasks run).
For latency, there are two latency metrics. E2E latency measures when the consumer sees the data. Producer's request-latency measures when the producer can move on after a message is sent. We can see the request-latency number is very small (<10ms) on both cloud-native and local-disk cases, this is where acks=1 setting excels at and this is also the niche of our KIP.