THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This proposal aims to support function DDL with the consideration of SQL syntax, language compliance, and advanced external UDF lib registration.
The Flink DDL is initialized and discussed in the design [1] by Shuyi Chen and Timo. As the initial discussion mainly focused on the table, type and view.
FLIP-69 [2] extend it with a more detailed discussion of DDL for catalog, database, and function. Original the function DDL was under the scope of FLIP-69.
After some discussion with the community, we found that there are several ongoing efforts, such as FLIP-64 [3], FLIP-65 [4], and FLIP-78 [5]. As they will
directly impact the SQL syntax of function DDL, the proposal wants to describe the problem clearly with the consideration of existing works and make sure
the design aligns with efforts of API change of temporary objects and type inference for UDF defined by different languages.
...
Before deep into the DDL SQL, we want to discuss the major requirements for defining a function within Flink runtime by related FLIPs:
External lib registration. The requirements come from the hive integration that enhances the adoption of Flink batch. HQL supports syntax like:
CREATE FUNCTION addfunc AS 'com.example.hiveserver2.udf.add' USING JAR 'hdfs:///path/to/jar' Language distinction. Due to bytecode language specifics in Scala, there are some limitations to extract type information from scala function. At the same time, support python UDF in table runtime is another ongoing effort. Thus, the SQL syntax needs to consider supporting multiple languages. Mysql create function syntax support language in this way
...
...
Temporary Function Support. FLIP-57
...
proposes to distinguish temporary and non-temporary functions for both catalog and system. As Temporary function will be registered only for the current session. It requires a flag from DDL to distinguish the function resolution order.
CREATE TEMPORARY FUNCTION addfunc AS 'com.example.hiveserver2.udf.add' USING JAR 'hdfs:///path/to/jar' Function Qualifier. Functions identifiers resolution consider object scopes whether in particular catalog, database or just current catalog and database. Thus, all of the function DDL needs to support 3-part path.
CREATE FUNCTION catalog1.addfunc AS 'com.example.hiveserver2.udf.add' LANGUAGE JVM
Function DDL SQL Syntax
Create Function Statement
...
The first change needed is to add more functions in CatalogFunction interface.
public interface CatalogFunction { String getClassName(); Enum getLanguage(); // TODO Map<String, String> getProperties(); CatalogFunction copy(); Optional<List<String>> getResourcePaths(); // TODO Optional<String> getDescription(); Optional<String> getDetailedDescription(); } |
|---|
In order to support loading external libraries and create UDFs from external libraries, we need to add a function in ExecutionEnvironment to register external libraries.
...