Versions Compared

Old Version 7

changes.mady.by.user Ashvin Agrawal

Saved on

compared with

New Version Current

changes.mady.by.user Ashvin Agrawal

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Rebalancing relocates data from heavily loaded members to lightly loaded members. Currently Geode only supports manual rebalancing by issuing a gfsh command or a java function call. In most cases, the decision to rebalance is based on the data distribution in the cluster and max memory configuration of the members. As Geode monitors the data size, it can also automatically trigger rebalancing. Auto-balancing is expected to periodically will redistribute data-load in the cluster and load periodically and prevent conditions leading to failures.

...

  1. Configurable size threshold to qualify system as unbalanced
  2. Configurable distribution skew to trigger rebalance
  3. off-balancedReuse existing manual-rebalancing
  4. Minimize the impact on concurrent operations caused by continuous rebalancing
    1. Configurable schedule
    2. Ability to disable auto-balancing
    Ability to plug a custom AR manager
  5. Reuse existing manual-rebalancing flow for a consistent rebalancing experience

Alternatives

The user can schedule a cron job to invoke the gfsh rebalance command on a periodic basis.

Background

  1. A member is unhealthy, if its heap is critical. Ideally a user would want to redistribute load on a unhealthy member to other members iff the members have sufficient capacity (i.e. totalBytes + newBucketSize << localMaxMemory). In some cases this can cause entire cluster to fail. Redistribution of load may cause healthy members to become unhealthy. Rebalancing can also increases IO activity significantly. So it may be safer to manually rebalance the cluster if any node is unhealthy.

...

  1.  
  2. Current implementation of rebalance operation can be used to estimate transfer-size, before actually executing transfer size. Transfer size is the total number of bytes that may be moved during a rebalance operation. It is mainly based on the total number of buckets below redundancy level and load on individual nodes. It will be inefficient if rebalance is executed if transfer-size is too small. Moreover rebalancing when transfer size is high may overload the system.
  3. New capacity of a cluster can be increased by adding new nodes. A user can specify rebalance flag after the last node is added. This way frequent rebalance can be avoided. 

Based on the points discussed above, we plan to use transfer-size metric as the primary decision factor for triggering rebalance. Presence of empty node will be ignored assuming user may be adding more capacity. Similarly critical nodes will be ignored assuming such nodes need specific region based rebalance actions.

...

How is load defined?

Load on a member is a function of

  1. Total number of buckets
  2. Total number of members
  3. hosted on the member
  4. Number of primary buckets on the member
  5. Number of secondary buckets on the member
  6. Size of the buckets
  7. Maximum memory

When is a

...

cluster off-balance?

  1. [Auto-balance candidate] if transfer-size is more than X% of the total data size, rebalance

...

A member is unhealthy or heavily loaded if

  1. its heap is critical (included in ResourceAdvisor().adviseCriticalMembers())
  2. the node is misconfigured, for e.g. max memory is not sufficient to host even one bucket

When is a member lightly loaded?

  1. if the member has enough memory, i.e. totalBytes + newBucket.getBytes() << localMaxMemory

When can a cluster be considered out of balance

  1. if some nodes in the cluster are heavily loaded while most other nodes are free. 
  2. if the cluster is not running at configured redundancy levels
  3. if distributing 10% of the data

    can result in a consistent data distribution and create comparable free space on all nodes

  4. [Auto-balance candidate] if the cluster is not running at configured redundancy levels
  5. [prefer manual rebalance] or Or any unhealthy node exists in the cluster

Where can a bucket be moved?

  1. For a bucket B, if there is a lightly loaded member which is not hosting B.

Use Cases

  1.  Adding a node to a existing cluster after loading dataAfter node failure and recovery, gfsh command "rebalance -simulate" reports a high transfer-size. In this case the new node will be lightly loaded and may not participate in data serving. In this scenario the total number of bytes rebalanced may not be a lot.
  2. Node recovery after a few node failures. In this case some buckets may not have enough redundancy or primary ownership may be limited to a few nodes only.
  3. , the nodes may have comparable utilization, but a rebalance would result in a uniform region data distribution. So action would be taken

  4. Over time, some buckets may grow much larger than other buckets in the region. Or some regions may grow more than others. Rebalance would get triggered, resulting in a uniform distribution

    Some buckets may grow up much larger than others

Design

We would like to implement this as an independent module without modifying existing code, so that it can be easily applied to any version of the system. To enable auto-balancing, the user will place the auto-balance jar on their classpath and add an initializer to their cache.xml. The initializer will provide the following configuration

  1. Schedule - cron string: In order to minimize the impact on concurrent operations, we feel it’s important to provide the user with the ability to configure the frequency and timing of automatic rebalancing. Bucket movement does add load to the system and in our performance tests we can see that the throughput of concurrent operations drops during bucket movement. A user is expected to configure off-peak hours for rebalancing. So a schedule based on cron like configuration is useful.
  2. rebalance operation may move 

    Size-threshold-percent - int between 1 and 99: Rebalancing will be triggered if the

    total number of bytes

    transfer-size is more than this threshold. This threshold

    ,

    is the percentage of the total data size. Rebalance operation computes transfer size based on relationship between regions, primary ownership and redundancy.

  3. Minimum cluster: Rebalancing could be harmful when the cache is initially being populated, because bucket sizes may vary wildly when there is very little data. Because of that, we will also provide a threshold before automatic rebalancing will kick in.

E.g.

<cache>
...
 <initializer>
  <!-- Optional auto-rebalance manager -->
  <class-name> com.gemstone.gemfire.cache.util.AutoBalancer </class-name>
 
  <!-- Optional. Default: Once a week on Saturday. E.g. check at 3 am every night -->
  <parameter name=”schedule”> 0 0 3 * * *? </parameter>
 
  <!-- Optional auto-rebalance manager -->
  <class-name> org.apache.geode.rebalance.AutoBalance </class-name>
 
  <!-- Optional. Default: 20%10%. E.g. Don’t rebalance until the variation intransfer size between members is more than 10% of the total data size -->
  <parameter name=”unbalance”size-threshold-percentage”>percent”> 10 </parameter>
 
  <!-- Optional. Default: 100 50%MB. E.g. Don’t rebalance a region until atthe leasttransfer onesize memberis isatleast 50%100 fullMB -->
  <parameter name=”size”minimum-threshold-percentage”>size”> 50100000000 </parameter>
 </initializer>
... 
</cache>


We only want one member to be automatically rebalancing a given region. So each member that starts auto rebalancing will try to get a distributed lock. If the member obtains the lock it will do the auto rebalancing until the member diesrebalance completes. Otherwise it continue to wait for the lock to become availablenext cycle and repeat.

At the scheduled interval the AR auto-balancer will check the balance of the system. It will do that by calling PartitionRegionHelper.getPartitionRegionInfo and fetching the size of all of the regions in bytes from all members. It will sum the colocated regions together (like rebalancing does). 

Note that this means there is a limitation that members configured with the auto rebalancer have all of the regions defined, because otherwise some regions may not be rebalanced. 


PlantUML
aligncenter
(*) --> "Init Cache"
--> "Init ARAuto-Balancer"
--> "GrabWait ARfor distributednext lockslot"
--> "Wait for next slot" as schedule
-right-> "Execute AR"
 
if "is out of balance" then
 as schedule
if "DLock acquired?" then
  -left->[No] schedule
else
  --> [Yes] "Start execution"
  --> if "balanced?" then
    --> [No] schedule
  else
    --> [Yes]  "Invoke Rebalance"
else
-> [No] schedule    --> "Wait for completion"
    --> schedule
  endif
endif

Testing

We will need to add auto rebalancing to some existing tests and give it a schedule that will cause it to run during the test. We will also need to write unit tests for the rebalancing triggering and scheduling logic.

...