You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

Introduction

Rebalancing relocates data from heavily loaded members to lightly loaded members. Currently Geode only supports manual rebalancing by issuing a gfsh command or a java function call. In most cases, the decision to rebalance is based on the data distribution in the cluster and max memory configuration of the members. As Geode monitors the data size, it can also automatically trigger rebalancing. Auto-balancing will redistribute data-load periodically and prevent conditions leading to failures.

Requirements

  1. Configurable size threshold to qualify system as off-balanced
  2. Configurable data-distribution skew
  3. Reuse existing manual-rebalancing flow
  4. Minimize the impact on concurrent operations caused by continuous rebalancing
    1. Configurable schedule
    2. Ability to disable auto-balancing
  5. Ability to plug a custom AR manager

Alternatives

The user can schedule a cron job to invoke the gfsh rebalance command on a periodic basis.

Description

Unhealthy-Member: A member is unhealthy if its heap is critical (included in ResourceAdvisor().adviseCriticalMembers())

Acceptor-member: A member can accept a new bucket if it has enough memory available to host the bucket, i.e. totalBytes + newBucket.getBytes() << localMaxMemory

Transfer-size: The total number of bytes that will be transferred during a rebalance operation 

How is load defined?

Load on a member is a function of

  1. Total number of buckets hosted on the member
  2. Number of primary buckets on the member
  3. Number of secondary buckets on the member
  4. Size of the buckets
  5. Maximum memory

When is a cluster off-balance?

  1. if transfer-size is more than X% of the total data size, rebalance can result in a consistent data distribution and create comparable free space on all nodes

  2. if some nodes in the cluster are heavily loaded, percentage heap utilization is much higher than other members in the cluster
  3. if the cluster is not running at configured redundancy levels
  4. Or any unhealthy node exists in the cluster

Use Cases

  1. Adding a node to a existing cluster after loading data. In this case the new node will be lightly loaded and may not host any buckets. In this scenario the transfer-size will be small percentage of total data size. Auto-rebalance will detect a skew and will take action

  2. After node failure and recovery, gfsh command "rebalance -simulate" reports a high transfer-size. In this case, the nodes may have comparable utilization, but a rebalance would result in a uniform region data distribution. So action would be taken

  3. Over time, some buckets may grow much larger than other buckets in the region. Or some regions may grow more than others. Rebalance would get triggered, resulting in a uniform distribution

Design

We would like to implement this as an independent module without modifying existing code, so that it can be easily applied to any version of the system. To enable auto-balancing, the user will place the auto-balance jar on their classpath and add an initializer to their cache.xml. The initializer will provide the following configuration

  1. Schedule - cron string: In order to minimize the impact on concurrent operations, we feel it’s important to provide the user with the ability to configure the frequency and timing of automatic rebalancing. Bucket movement does add load to the system and in our performance tests we can see that the throughput of concurrent operations drops during bucket movement. A user is expected to configure off-peak hours for rebalancing. So a schedule based on cron like configuration is useful.

  2. Size-threshold-percent - int between 1 and 99: Rebalancing will be triggered if the transfer-size is more than this threshold. This threshold is the percentage of the total data size. Rebalance operation computes transfer size based on relationship between regions, primary ownership and redundancy.

  3. Rebalancing could be harmful when the cache is initially being populated, because bucket sizes may vary wildly when there is very little data. Because of that, we will also provide a threshold before automatic rebalancing will kick in.

E.g.

<cache>
...
 <initializer>
  <!-- Optional auto-rebalance manager -->
  <class-name> com.gemstone.gemfire.cache.util.AutoBalancer </class-name>
 
  <!-- Optional. Default: Once a week on Saturday. E.g. check at 3 am every night -->
  <parameter name=”schedule”> 0 0 3 * * ? </parameter>
 
  <!-- Optional. Default: 20%. E.g. Don’t rebalance until the transfer size is more than 10% of the total data size -->
  <parameter name=”size-threshold-percent”> 10 </parameter>
 
  <!-- Optional. Default: 50%. E.g. Don’t rebalance a region until at least one member is 50% full -->
  <parameter name=”size-threshold-percentage”> 50 </parameter>
 </initializer>
... 
</cache>


We only want one member to be automatically rebalancing a given region. So each member that starts auto rebalancing will try to get a distributed lock. If the member obtains the lock it will do the auto rebalancing until rebalance completes. Otherwise it continue to wait for the next cycle and repeat.

At the scheduled interval the auto-balancer will check the balance of the system. It will do that by calling PartitionRegionHelper.getPartitionRegionInfo and fetching the size of all of the regions in bytes from all members. It will sum the colocated regions together (like rebalancing does). 

Note that this means there is a limitation that members configured with the auto rebalancer have all of the regions defined, because otherwise some regions may not be rebalanced. 


Init Cache Init Auto-Balancer Wait for next slot Start execution Invoke Rebalance Wait for completion DLock acquired? No Yescluster balanced? No Yes

Testing

We will need to add auto rebalancing to some existing tests and give it a schedule that will cause it to run during the test. We will also need to write unit tests for the rebalancing triggering and scheduling logic.

Limitations

  • Initializer: Geode has provision for a single initializer instance. Spring integration also depends on Initializer. So initializer based approach could block user from using some features. Initializer initializer based approach seems ok for POC. Also some parts of the code will be reusable, scheduler, locking and trigger logic.
  • For now start with a separate module (like gemfire-web) for rebalancer. We will consolidate smaller modules into a bigger one later if it gets too cluttered.
  • Quartz seems to be an overkill for just cron string parsing. Since rebalance is an expensive operation, we expect uses to schedule it off-peak hours. This is where cron based schedule is very useful. We are not exposing cron api externally and may replace it with a lighter implementation for cron parsing.
  • Only regions that are defined on the auto rebalancer node will be rebalanced. Users can add accessors if there is a region they want to make sure gets rebalanced but is not available everywhere.
  • Rebalancing always recovers redundancy, moves buckets, and moves primaries. This means that when the rebalancer kicks in, redundancy will be recovered, regardless of the settings for recovery-delay.
  • There is no way to disable or modify the automatic rebalancing without restarting members, since the configuration is part of the member configuration.
  • No labels