Versions Compared

Old Version 13

changes.mady.by.user luoyuxia

Saved on

compared with

New Version 14

changes.mady.by.user luoyuxia

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejava
/**
 * Factory that creates {@link Parser}.
 *
 * <p>The {@link #factoryIdentifier()} is identified by matching it against {@link
 * TableConfigOptions#TABLE_SQL_DIALECT}.
 */
@PublicEvolving
public interface ParserFactory extends Factory {

    /** Creates a new parser. */
    Parser create(Context context);

    /** Context provided when a parser is created. */
    @PublicEvolving
    interface Context {
        CatalogRegistry getCatalogRegistry(); // interfaces provided dealing with get catalog, qulify identifier, etc.

        OperationContextOperationTreeBuilder getOperationContextgetOperationTreeBuilder(); // interfaces provided to build Operation.
    }
}

...

Code Block
languagejava
/**
 * A contextbuilder for building {@link org.apache.flink.table.operations.Operation}
 */
@PublicEvolving
public interface OperationContextOperationTreeBuilder {
     
    public QueryOperation scan(String path);

    public QueryOperation project(List<Expression> projectList, QueryOperation child);

    public QueryOperation project(
            List<Expression> projectList, QueryOperation child, boolean explicitAlias);

    public QueryOperation project(
            List<Expression> projectList, QueryOperation child, List<OverWindow> overWindows)

    // omit other public interaces has implemented in OperationTreeBuilder
}

Implement detail

We will introuce a slim module may called These public interfaces refered above will be added to module flink-table-planner-spi, which only expose these public interfaces refered above.  api-java. And then , when we want to support other dialectdialects,  we need to include this slim module moduel and implement ParserFactory to create a Parser for the specific dialect.

...

Code Block
languagejava
public class MySQLParser implements Parser {
    private OperationContextOperationTreeBuilder operationContextoperationTreeBuilder;

    public MySQLParser(OperationContextContext operationcontext) {
        this.operationoperationTreeBuilder = operationcontext.getOperationTreeBuilder;
    }
    
    @Override
    List<Operation> parse(String statement) {
      // parse it to AST(Abstract Semantic Tree)
      MySQLAST mysqlAST = parseStament(statement);
      // convert the AST to Flink OperationTree
      List<Operation> operations = convertToOperation(mysqlAST);
   }
   
   private List<Operation> convertToOperation(MySQLAST mysqlAST) {
      // may look like
      Operation operation = operationContextOperationTreeBuilder.project(Arrays.asList(Expressions.$("f0"), operationContext.scan("t1"));
      return Collections.singletonList(operation)
   }
   
   @Override
   UnresolvedIdentifier parseIdentifier(String identifier) {
    // may need to identifier `db.t` to array of [db, t]
    string[] names = parseMySQLIdentifier(identifier);
    return UnresolvedIdentifier.of(names);
  }

  @Override
  ResolvedExpression parseSqlExpression(
            String sqlExpression, RowType inputRowType, @Nullable LogicalType outputType) {
   // parse sqlExpression to ResolvedExpression
  }

  @Override
  String[] getCompletionHints(String statement, int position) {
   // just for example, return empty string array directly
   return new String[0];
  }
}

...

The support for Hive dialect, we should also do like this way: parsing to AST and convert it to Operation via OperationContextOperationTreeBuilder.  But we has implemeted Hive dialect, and the current implementation  is converting the sql to Calcite’s RelNode which is consistent to Hive’s implementation when using CBO in Hive. It'll take much efforts for we need to rewrite the codebase about Hive dialect totally for it's totally different from the current implementation.  It's hard to migrate to Operation tree at one shot.

So the tempory way is to provide the calcite dependency in introduce a slim  module may called flink-table-planner-spi module,  calcite , which provides the calcite dependency along with the ability to create RelNode, which invoves involves accessing the RelOptCluster, RelBuilder, etc, provided by PlannerContext.  But it's internal and only used by Hive connector.  At the end,  the calcite dependency should be removed and the Hive dialect should be migrate to Operation tree and the module can be dropped. To do this,  we propose to introduce the following interal interfaces/class:

CalciteContext

Code Block
languagejava
// Context for creating RelNode
@Internal
public interface CalciteContext extends ParserFactory.Context {

    CalciteCatalogReader createCatalogReader(
            boolean lenientCaseSensitivity, String currentCatalog, String currentDatabase);

    RelOptCluster getCluster();

    FrameworkConfig createFrameworkConfig();

    RelDataTypeFactory getTypeFactory();

    RelBuilder createRelBuilder(String currentCatalog, String currentDatabase);
}

...

Code Block
languagejava
/** Wrapper for Calcite RelNode tree. */
@Internal
public class CalciteQueryOperation implements QueryOperatio {
    private final RelNode calciteTree;
    private final ResolvedSchema resolvedSchema;
	
    public CalciteQueryOperation(RelNode calciteTree, ResolvedSchema resolvedSchema) {
       this.calciteTree = calciteTree;
       this.resolvedSchema = resolvedSchema;
    }
 }

Proposed Changes

  1. Introduce some public interfaces for supporting other dialects in flink-table-api-jave module
  2. Introduce a slim module called flink-table-planner-spl exposing interfaces for supporting other dialects.Add calcite  providing calcite dependencys for Hive dialect in moudle flink-table-planner-spl. But's it's just a tempory way to migrating Hive dialect, the specific dependencys or logic should be dropped at the end.

...