THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Identify files that are eligible for clustering
- Filter specific partitions (based on config to prioritize latest vs older partitions)
- Any files that have size > targetFileSize are not eligible for clustering
- Any files that have pending compaction/clustering scheduled are not eligible for clustering
- Any filegroups that have log files are not eligible for clustering (We could remove this restriction at a later stage.)
- Group files that are eligible for clustering based on specific criteria. Each group is expected to have data size in multiples of ‘targetFileSize’. Grouping is done as part of ‘strategy’.
- If sort columns are specified,
- Among the files that are eligible for clustering, it is better to group files that have overlapping data for the columns specified together.
- Example: say the target of clustering is to produce 1GB files. Partition initially has 8 * 512MB files. (After clustering, we expect data to be present in 4 * 1GB files.)
- Assume among 8 files, say only 2 files have overlapping data for the ‘sort column’, then these 2 files will be part of one group. Output of the group after clustering is one 1GB file.
- Assume among 8 files, say 4 files have overlapping data for the ‘sort column’, then these 4 files will be part of one group. Output of the group after clustering is two 1GB files.
- We could put a cap on group size to improve parallelism and avoid shuffling large amounts of data
- If sort columns are not specified, we could consider grouping files based on other criteria: (All of these can be exposed as different strategies).
- Group files based on record key ranges. This is useful because key range is stored in a parquet footer and can be used for certain queries/updates.
- Groups files based on commit time.
- Random grouping of files.
- Filter groups based on specific criteria (akin to orderAndFilter in CompactionStrategy)
- Finally, the clustering plan is saved to the timeline. Structure of metadata
...