You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Current »

To be Reviewed By: September 24th, 2021

Authors: Alberto Gomez (alberto.gomez@est.tech)

Status: Draft | Discussion | Active | Dropped | Superseded

Superseded by: N/A

Related: N/A

Problem

Apache Geode is designed to provide linear-scaling low latency for transactions, reads, writes and query processing of indexed or unindexed data.

Even though the execution of a query in OQL is restricted to a maximum of one thread per cluster member, the number of objects generated to be garbage collected by an OQL query may be very high, which could lead to CPU spikes in all the CPUs of the cluster members and even JVM stops.
As a result, OQL queries executed on a Geode cluster could highly penalize the latency of transactions, reads and writes and make it very unpredictable.

Anti-Goals

This RFC does not intend to resolve the problem of OQL queries generating a lot of objects to be garbage collected.

Solution

A possible way to reduce long GC pauses when running an OQL query could be to slow it down (throttle) by pausing the execution at different points inside it. That way, the garbage collector will not get a too high amount of objects to be garbage collected at a too short period of time (will be able to garbage collect objects at a higher rate than they are created) and thus long GC stops will be avoided.

More specifically, queries could be throttled by adding sleeps at intervals of entries, while they are being processed as they are retrieved via an index or a region iterator.

This RFC proposes that query throttling is controlled by a throttling ratio parameter that specifies the percentage of time that the query must be paused with respect to the total time taken by the query without pauses.
For example, if a query without throttling would take normally 10 seconds and the throttling ratio is set to 1, then the time while the query will be paused would be 10 seconds and the time to get results would be 20 seconds. If the throttling ratio is set to 2, then the total time to get results would be 30 seconds.

The throttling will aim at doing cycles of execution + pause of 1 second. In order to achieve it, the amount of time to pause the query and the moments at which the execution will be paused would be calculated as follows:

  • Initially, the first pause will be done after 1000 entries have been processed (1000 / number of active buckets in case of a partitioned region).
  • Once the entries have been processed, the execution will be paused by sleeping for execution_time_to_filter_the_ entries * throttling ratio.
  • Additionally, the number of entries to be processed before pausing in the next cycle, so that it is of around 1 second (execution + pause), will be adjusted:
    • If the cycle time was lower than a second, the number of entries to process before pausing in the next cycle must be increased. Otherwise, it should be decreased.
  • After the pause, the processing of entries will continue until the adjusted value is reached. Next, the execution will be paused again and the entries before pausing in the next cycle readjusted. This will be done until there are no more entries to be processed.

Changes and Additions to Public Interfaces

Adding throttling to an OQL query will be done by adding some meta information like "<throttle <throttleValue>>", in a similar way as it is done to get trace information when executing queries. The <throttle ...> part will be optional.

Example

Executing an OQL query with a throttling ratio of 2.5:

"<throttle 2.5f> select * from portfolios where portfolios.type = '3'"

Performance Impact

The performance of queries making use of the throttling mechanism will decrease but it is expected that the latency of read, writes and transactions will increase (lower and more predictable latency) when they coexist with expensive queries.

Backwards Compatibility and Upgrade Path

No impacts as the throttling feature will be optional.

Prior Art

An alternative proposal to the one described in this RFC could be to lower the priority of the threads executing OQL queries. Some tests have been done in this respect but the observation has been that queries are slowed down only when the CPU available is close to 0. Therefore, this approach is not valid.

FAQ

Answers to questions you’ve commonly been asked after requesting comments for this proposal.

Errata

What are minor adjustments that had to be made to the proposal since it was approved?

  • No labels