HTTP API
- POST:
example: - DELETE:
example
Internal Implementation:
- Metadata entity
- Dataset:
for each dataset record, there is one field called "rebalanceCount". If the dataset has never been rebalanced, it is Missing. - Nodegroup:
when a dataset foo is created, we internally create a nodegroup with name foo_i (or just foo if i=0) where i=foo.rebalanceCount based on the current available nodes. Then, we let the nodegroup of foo be foo_i.
- Dataset:
- For each shadow dataset foo, repeat the following process:
create a new node group foo_i (where i= foo.rebalance_count + 1) that contains the current available nodes, if the node group has already been occupied, we let the new node group have name foo_<uuid>;
create an uncommitted dataset foo (on node group foo_<i>) using node group foo_<i> with the same rebalance_count;
drop any leftover files for the uncommitted dataset foo;
upsert all documents from foo to foo (on node group foo_<i>) on all partitions
update the metadata entity for dataset foo, make the uncommitted foo become the committed foo in metadata
drop files foo (on node group foo_i-1) and drop node group foo_<i-1>
There are three metadata transactions for step 1 to 6:
step 1-4
step 5:
step 6: