Rebalance API and Internal Implementation

HTTP API

• POST:
  example:
• DELETE:
  example

Internal Implementation:

• Metadata entity
  
  • Dataset: 
    for each dataset record, there is one field called "rebalanceCount".  If the dataset has never been rebalanced, it is Missing.
  • Nodegroup:
    when a dataset foo is created, we internally create a nodegroup with name foo_i (or just foo if i=0) where i=foo.rebalanceCount based on the current available nodes. Then, we let the nodegroup of foo be foo_i.
    
    
• For each shadow dataset foo, repeat the following process:
  

  1. create a new node group foo_i (where i= foo.rebalance_count + 1) that contains the current available nodes, if the node group has already been occupied, we let the new node group have name foo_<uuid>;

  2. create an uncommitted dataset foo (on node group foo_<i>) using node group foo_<i> with the same rebalance_count;

  3. drop any leftover files for the uncommitted dataset foo;

  4. upsert all documents from foo to foo (on node group foo_<i>) on all partitions

  5. update the metadata entity for dataset foo,  make the uncommitted foo become the committed foo in metadata

  6. drop files foo  (on node group foo_i-1) and drop node group foo_<i-1>

     
     

  • There are three metadata transactions for step 1 to 6:

  1. 1. step 1-4

     2. step 5:

     3. step 6: