Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejava
public interface CatalogFunction {
    /**
     * Get a detailed resource description of the function.
     *
     * @return an {@link ResourceUri} list of the function
     */
    Optional<List<ResourceUri>>List<ResourceUri> getFunctionResources();

    /**
     * Distinguish if the function is a generic function.
     *
     * @deprecated This method is currently only used in hive to determine
     * if a function is a generic function. The behavior should be implemented
     * by hive itself, instead of providing a public api, so we deprecate it.
     *
     * @return whether the function is a generic function
     */
    @Deprecated
    boolean isGeneric();
}

...

Code Block
languagejava
public interface FunctionDefinitionFactory {
     /**
     * Creates a {@link FunctionDefinition} from given {@link CatalogFunction}. If the
     * {@link CatalogFunction} is created by user defined resource, the user of
     * {@link FunctionDefinitionFactory} needs to override this method explicitly.

     * The<p> implementationIt logicis needsrecommended to use the given user classclassloader loader to load custom classes
     * instead of the thread context class loader. the function class, 
	 * because if the function has a resource URL, the framework will help load 
	 * the resource into the given classloader.
     * 
     * @param name name of the {@link CatalogFunction}
     * @param catalogFunction the catalog function
     * @param userClassLoader the class loader is used to load user defined function's class
     * @return
     */
    default FunctionDefinition createFunctionDefinition(
            String name,
            CatalogFunction catalogFunction,
            ClassLoader userClassLoader) {
        if (catalogFunction.getFunctionResources().isPresent() 
                && !CollectionUtil.isNullOrEmpty(catalogFunction.getFunctionResources().get())) {
            throw new UnsupportedOperationException(
                    String.format("%s need to override default createFunctionDefinition for " 
                            + "loading user defined function class", this.getClass().getSimpleName()));
        } else {
            return createFunctionDefinition(name, catalogFunction);
        }
    }
}

...

For remote resource there are different storage scheme such as HTTP, HDFS, S3, OSS etc. During the compilation period of the query, it will first download the remote resource to a local temporary directory based on its path, and then add the local path to the user class loader. The next behavior is the same as local resource section.

We will use use Flink's FileSytem abstraction to download remote resource, which can support all types of file system, including HTTPOSS, S3, HDFS, etc, rather than binding to a specific implementation.

...