Page History

...

1. Identify files that are eligible for clustering
1. Filter specific partitions (based on config to prioritize latest vs older partitions)
2. Any files that have size > targetFileSize are not eligible for clustering
3. Any files that have pending compaction/clustering scheduled are not eligible for clustering
4. Any filegroups that have log files are not eligible for clustering  (We could remove this restriction at a later stage.)
2. Group files that are eligible for clustering based on specific criteria. Each group is expected to have data size in multiples of ‘targetFileSize’.  Grouping is done as part of ‘strategy’.
1. If sort columns are specified,
1. Among the files that are eligible for clustering, it is better to group files that have overlapping data for the columns specified together.
2. Example: say the target of clustering is to produce 1GB files. Partition initially has 8 * 512MB files. (After clustering, we expect data to be present in 4 * 1GB files.)
1. Assume among 8 files, say only 2 files have overlapping data for the ‘sort column’, then these 2 files will be part of one group. Output of the group after clustering is one 1GB file. 
2. Assume among 8 files, say 4 files have overlapping data for the ‘sort column’, then these 4 files will be part of one group. Output of the group after clustering is two 1GB files.
3. We could put a cap on group size to improve parallelism and avoid shuffling large amounts of data 
2. If sort columns are not specified, we could consider grouping files based on other criteria: (All of these can be exposed as different strategies).
1. Group files based on record key ranges. This is useful because key range is stored in a parquet footer and can be used for certain queries/updates.
2. Groups files based on commit time. 
3. Random grouping of files.
3. Filter groups based on specific criteria (akin to orderAndFilter in CompactionStrategy)
4. Finally, the clustering plan is saved to the timeline. Structure of metadata

...