...
Partition pruning is already implemented in Apache Ignite in very simplified form [1]. Only WHERE condition with equality is considered and only for SQL queries without joins. We should expand it further.
[1] https://issues.apache.org/jira/browse/IGNITE-4509
...
In the following sections we first explain how partitions could be extracted from SQL parts, and how certain query rewrite techniques could help us with it. Then we will describe how extracted partition info is assembled in a form of tree. Then we discuss that partition extraction should be performed two times - before split for the whole query, and after split for query parts. Finally, we explain how partition info will be passed to thin clients, and how users will be able to control and monitor partition pruning.
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
(P1) OR (P2) => (P1, P2)
(P1) OR (ALL) => (ALL)
(P1) OR () => (P1)
(P1, P2) OR (P2, P3) => (P1, P2, P3)
(:1) OR (:2) => (:1, :2)
(P1, :1) OR (P2, :2) => (P1, P2, :1, :2) |
...
Joins are very common, so it is crucial to support partition extraction for them as well. General solution might be extremely complex, so we need to define reasonable bounds where could operateoptimization is applicable, and improve them iteratively in future. We start with query AST obtained from parser. Proposed flow to extract partitions is explained below. Some of explained these steps could be merged to improve performance.
REPLICATED
and and do not have node filter, then mark it as "ANY" and remove from the tree, as it doesn't affect JOIN outcome. Otherwise - exit, no need to bother with custom filters.RendezvousAffintiyFunction
is is deterministic, while FairAffinityFunction
is is not)PARTITIONED
caches caches, only equi-joins, where every table is assigned to a single co-location group.REPLICATED
caches caches with "ANY" policy should be eliminated as follows:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
(P1, :2) AND (ANY) => (P1, :2) (P1, :2) OR (ANY) => (P1, :2) |
If partition tree contain rules from different co-location groups, then exit.
...
Code Block | ||
---|---|---|
| ||
interface PartitionNode { Collection<Integer> apply(Object[] args); } class PartitionGroup implements PartitionNode { Collection<Object> parts; // Concrete partitions, arguments or both. } class PartitionExpression implements PartitionNode { PartitionNode left; PartitioNodePartitionNode right; } |
Partition tree is enriched with {{AffinityTopologyVersion
}} it it was built on, and affinity function descriptor. Descriptor can only be defined for well-known affinity functions, such as {{RendezvousAffinityFunction
}}, and defines the logic on how to convert an object to partition.
Code Block | ||
---|---|---|
| ||
class PartitionInfo { PartitionNode tree; AffintiyTopologyVersion affTopVer; AffinityFunctionDecriptor affFunc; } |
...
...