Intro to Spark SQL Catalyst

Background

Since that fact that Kylin4 is highly depend on Spark SQL, it better we have a deeper understanding of Spark SQL.

Catalyst is an execution-agnostic framework to represent and manipulate a dataflow graph, i.e. trees of relational operators and expressions.

The main abstraction in Catalyst is TreeNode that is then used to build trees of Expressions or QueryPlans.

Name	Role	Comment
SparkSession
Dataset	Structured Query with Data Encoder	Dataset is a strongly-typed data structure in Spark SQL that represents a structured query.
Catalyst	Tree Manipulation Framework	Catalyst is an execution-agnostic framework to represent and manipulate a dataflow graph, i.e. trees of relational operators and expressions.
TreeNode	Node in Catalyst Tree	`TreeNode` is a recursive data structure that can have one or many children that are again `TreeNodes`.
Expression	Executable Node in Catalyst Tree	`Expression` is a executable node (in a Catalyst tree) that can evaluate a result value given input values, i.e. can produce a JVM object per `InternalRow`.
QueryPlan	Structured Query Plan	`QueryPlan` is part of Catalyst to build a tree of relational operators of a structured query. Scala-specific, `QueryPlan` is an abstract class that is the base class of LogicalPlan and SparkPlan (for logical and physical plans, respectively).