...

Rebalancing relocates data from heavily loaded members to lightly loaded members. Currently Geode only supports manual rebalancing by issuing a gfsh command or a java function call. In most cases, the decision to rebalance is based on the size data distribution in the cluster and max memory configuration of the membermembers. As Geode monitors the data size, it can also automatically trigger rebalancing. Auto-Rebalancing (AR) balancing is expected to periodically redistribute data-load in the cluster and prevent failures cased by unbalanced cluster more than manual rebalanceconditions leading to failures.

Requirements

1. Configurable size threshold to qualify system as unbalanced
2. Configurable distribution skew to trigger rebalance
3. Reuse existing manual-rebalancing
4. Minimize the impact on concurrent operations caused by continuous rebalancing
   
   1. Configurable schedule
   2. Ability to disable AR
   3. Minimum size qualification
   4. auto-balancing
5. Ability to plug a custom AR manager

...

1. Total number of buckets
2. Total number of members
3. Number of primary buckets on the member
4. Number of secondary buckets on the member
5. Size of the buckets
6. Maximum memory

When is a member unhealthy?

...

A member is unhealthy or heavily loaded if

1. its Its heap is critical (included in ResourceAdvisor().adviseCriticalMembers())
   
2. The the node is misconfigured, for e.g. max memory is not sufficient to host even one bucket

When is a member lightly loaded?

1. It if the member has enough memory:, i.e. totalBytes + newBucket.getBytes() << localMaxMemory

When

...

can a cluster be considered out of balance

1. if some nodes in the cluster are heavily loaded while most other nodes are free. 
2. if the cluster is not running at configured redundancy levels
3. if distributing 10% of the data can result in a consistent data distribution and create comparable free space on all nodes
4. Or any unhealthy
5. If (maxLoadedMember - minLoadedMember) > 20%
6. Or any heavily-loaded node exists in the cluster

...

1. For a bucket B, if there is a lightly loaded member which is not hosting B

Use Cases

1.  Adding a node to a existing cluster after loading data. In this case the new node will be lightly loaded and may not participate in data serving. In this scenario the total number of bytes rebalanced may not be a lot.
2. Node recovery after a few node failures. In this case some buckets may not have enough redundancy or primary ownership may be limited to a few nodes only.
   
3. Some buckets may grow up much larger than others

Design

We would like to implement this as an independent module without modifying existing code, so that it can be easily applied to any version of the system. To enable ARauto-balancing, the user will place the auto-rebalance balance jar on their classpath and add an initializer to their cache.xml. The initializer will provide the following configuration

1. Schedule - cron string: In order to minimize the impact on concurrent operations, we feel it’s important to provide the user with the ability to configure the frequency and timing of automatic rebalancing. Bucket movement does add load to the system and in our performance tests we can see that the throughput of concurrent operations drops during bucket movement. Schedule can be provided as a cron string. Default: Once a week on SaturdayA user is expected to configure off-peak hours for rebalancing. So a schedule based on cron like configuration is useful.
2. Size-threshold-percent - int between 1 and 99: Rebalancing will be triggered if the total number of bytes rebalance operation may move is more than this threshold, percentage of the total data size. Rebalance operation computes transfer size based on relationship between regions, primary ownership and redundancy.

3.  

   AR manager: the class to check the system state and trigger rebalance. Default: AutoRebalanceThreshold: Rebalancing will be triggered if the load difference between the most-loaded and least-loaded member is more than this threshold. Default: 20%

4. Rebalancing could be harmful when the cache is initially being populated, because bucket sizes may vary wildly when there is very little data. Because of that, we will also provide a threshold before automatic rebalancing will kick in.

E.g.

<cache>

...

 <initializer>

  <!-- Optional. Default: Once a week on Saturday. E.g. check at 3 am every night -->

  <parameter name=”schedule”> 0 0 3 * * * </parameter>

  <!-- Optional auto-rebalance manager -->

  <class-name> org.apache.geode.rebalance.AutoBalance </class-name>

  <!-- Optional. Default: 20%. E.g. Don’t rebalance until the variation in size between members is more than 10% -->

  <parameter name=”unbalance-percentage”> 10 </parameter>

  <!-- Optional. Default: 50%. E.g. Don’t rebalance a region until at least one member is 50% full -->

  <parameter name=”size-threshold-percentage”> 50 </parameter>

 </initializer>

... 

</cache>

...