Page History

...

The goal of this would be to do basic performance analysis and correctness testing in a distributed environment.

Required Metrics

Client Side Measurements

1. Throughput
2. Response Time/Latency
3. Timeouts
4. Consumer lag

Common Stats

1. Vmstat - Context Switch, User CPU utilization %, System CPU utilization %, Total CPU utilization %
2. Iostat - Reads/sec, Writes/sec, KiloBytes read/sec, KiloBytes write/sec, Average number of transactions waiting, Average number of active transactions, Average response time of transactions, Percent of time waiting for service, Percent of time disk is busy
3. Prstat - Virtual memory size of each java process, RSS size of each process, Total CPU utilization of each process

GC Log Analysis

1. Footprint (Maximal amount of memory allocated)
2. Freed Memory (Total amount of memory that has been freed)
3. Freed Memory/min (Amount of memory that has been freed per minute)
4. Total Time (Time data was collected for)
5. Acc Pauses (Sum of all pauses due to GC)
6. Throughput (Time percentage the application was NOT busy with GC)
7. Full GC Performance (Performance of full GC collections. Full GC collections are marked so in the gc logs.)
8. GC Performance (Performance of minor collections. These are collections that are not full according to the definition above.)
9. CMS counts and frequency (Number of CMS collections and their frequency)
10. CMS failure count and frequency (CMS failure metrics)

Server side metrics

1. Throughput and response time breakdown for each request at the LogManager, RequestPurgatory level
2. ISR membership churn aggregate and per partition
3. Number of expirations in the request purgatory
4. Leader election rate aggregate and per partition
5. Leader election latency aggregate and per partition
6. High watermark change aggregate and per partition
7. Replica lag time and bytes aggregate and per partition
8. Replica fetch throughput and response time aggregate and breakdown at the LogManager, RequestPurgatory level

Log analysis

1. Exceptions in logs, their frequency and types of exception
2. Warnings in logs, their frequency and types of warnings

Miscellaneous

1. Capture all the server machine profiles before tests are being executed (Such as disk space, number of CPUS etc)
2. Capture all configurations for each run

Phase I: Perf Tools

The goal of this phase is just to create tools to help run perf tests. We already have some of these so this will primarily just be about expanding and augmenting these.

...