...
Example of similar tools in other products: AWR [1] [2] [3] (Oracle) ; pgbadger [4], pgmetrics [5], powa [6] (PostgresSQL).
Description
...
- Enable profiling mode.
- Executes some arbitrary workload.
- Collects profiling info.
- Run the tool that will create the Report contains statistics of workload.
Proposed Changes
The Ignite will provide public facade to manage profiling mode:
ignite.profiling().enable(); // Turns on profiling mode.
ignite.profiling().disable(); // Turns off profiling mode.
ignite.profiling().isEnabled(); // Is profiling mode turn on?
Profiling mode can be managed from CLI (And JMX):
control.sh --profiling // Prints current profiling mode status.
control.sh --profiling enable // Turns on profiling mode.
control.sh --profiling disable // Turns off profiling mode.
The Ignite will provide the public SPI interface (ProfilingSpi
) to log statistics. It can be configured via IgniteConfiguration
. It describes follow methods:
- startProfiling(); // Starts profiling.
- stopProfiling(); // Stops profiling.
- log(String info); // Logs operation statistics.
...
The new ignite-profiling
module will contain:
- Default Implementation (
LogProfilingSpiImpl
) based on async logging to the configured file. - The script to collect logs from nodes and build the report:
report.sh(bat)
Performance report
The performance report will be in a human-readable text format format (html page) and should contain:
- Ignite and plugins versions, topology changes, profiling start/end time
- Queries (SQL, scan, ..) timings, resources:
- Queries that took up the most time
- Slowest queries
- Most frequent queries
- Failing queries
- Queries count by type
- Queries that took up the most CPU/IO/Disk
- User tasks timings (similar to queries timings)tasks timings, resources
- Caches and cache operations statisticsCache operations timings on affinity nodes:
- Get/Put/Remove
- RemoveAndGet
- PutAndGetTransactions
- Invoke
- Lock
- create/destroy caches
Transactions commit/rollback timings - Workload by nodes
- Checkpoints statistics
- PME WAL statistics
...
Additional investigation required to gather following statistics:
...
- Disk, I/O wait/read/write per query/task/cache operations
- Query parse time
- Time for Lock waiting blockstime
- User time per transaction
- Messages process timings
- I/O per data regions
This statistics will provide:
- Top query/operations by CPU time
- Top query/operations by IO time
- What operations use most resources?
Public API changes
The new interface will be added: ProfilingSpi
.
The new ignite facade will be added: ignite.profiling()
.
The new module will be created: ignite-profiling
.
Corner cases
Node left during profiling
Node left will not affect to the cluster profiling mode.
Node join during profiling
...
Proposed Changes
The Ignite will log some additional internal performance statistics to profiling files. The format is like WAL logging.
One disk-writer thread and off-heap memory buffer will be used to minimize affect on performance. Maximum file size and buffer size can be configured on start.
The new extension performance-statistics-ext
module will be introduced. It will contain the tool to build the report: build-report.sh(bat).
The JSON format is used to store aggregated statistics and next draw in the report.
The report is based on the bootstrap library and can be viewed in a browser offline.
Management
1) JMX:
PerformanceStatisticsMBean
void start()
// Start collecting performance statistics in the cluster.void stop()
// Stop collecting performance statistics in the cluster.boolean enabled()
// True if collecting performance statistics enabled.
2) Control.sh utility. Functionality is like JMX.
3) System properties:
- IGNITE_PERF_STAT_FILE_MAX_SIZE - Performance statistics maximum file size in bytes. Performance statistics will be stopped when the size exceeded.
- IGNITE_PERF_STAT_BUFFER_SIZE - Performance statistics offheap buffer size in bytes.
- IGNITE_PERF_STAT_FLUSH_SIZE - Performance statistics minimal batch size to flush in bytes.
- IGNITE_PERF_STAT_CACHED_STRINGS_THRESHOLD - Performance statistics maximum cached strings threshold. String caching will stop on threshold excess.
Risks and Assumptions
Enabled profiling mode will cause performance degradation.
Discussion Links
Dev-list discussion.
// Links to discussions on the devlist, if applicable.
Tickets
Report example
Image Added
Image Added
Image Added
Image Added
Image Added
Reference Links
- https://docs.oracle.com/cd/E11882_01/server.112/e41573/autostat.htm#PFGRF94176
- http://www.dba-oracle.com/t_sample_awr_report.htm
- http://expertoracle.com/2018/02/06/performance-tuning-basics-15-awr-report-analysis/
- https://github.com/darold/pgbadger
- https://pgmetrics.io/docs/index.html#example
- https://powa.readthedocs.io/en/latest/
Tickets
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | IGNITE-12666 |
---|
|
// Links or report with relevant JIRA tickets.