Versions Compared

Old Version 11

changes.mady.by.user Bowen Li

Saved on

compared with

New Version Current

changes.mady.by.user Chesnay Schepler

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


...

Page properties

...


Discussion thread

...

https://www.mail-archive.com/dev@flink.apache.org/msg29894.html
Vote thread
JIRA

...

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyFLINK-14132

Release1.10


Google Doc: https://docs.google.com/document/d/17CPMpMbPDjvM4selUVEfh_tqUK_oV0TODAUA9dfHakc/edit?usp=sharingReleased:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

// new APIs to TableEnvironment
interface TableEnvironment {
    // load a module instance to the end of the module list
    void loadModule(String name, Module m);

    // unload a module instance from module list and other modules remain the same relative positions
    void unloadModule(String name); // the type string defined in the module

    // list all the modules' names according to order in module list
    List<String> listModules();
}

// note the following modules will be of the order they are specified
tableEnv.loadModule("a", MymoduleMyModule.INSTANCE);
tableEnv.loadModule("b", new Xxx(properties));
tableEnv.loadModule("c", new Yyy());
tableEnv.unloadModule("a");

...

In case users forgot to specify core module, “modules” section will be commented out in yaml file as following 

#modules: # note the following modules will be of the order they are specified
#  - name: core
#     type: core

SQL:

  • SHOW MODULES: show module names in the existing module list in order
  • LOAD MODULE 'name' [WITH (‘type’=’xxx’, 'prop'='myProp', ...)] : load a module with given name and append to end of the module list
  • UNLOAD MODULE 'name’ : unload a module by name from module list and other modules remain the same relative positions


NOTE: the SQL syntax has been discussed again and received some modifications, see FLINK-21045 and the discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html


Resolution Order

Object will be resolved to modules in the order they are defined either in program or in yaml configs. When there are objects sharing the same name, resolution logic will go thru modules in order and return whatever the first one is found, the other ones sitting in the back in the order will be ignored. E.g. if modules are set as “xxx, yyy” where xxx and yyy modules both have a function named “f”, then “f” will always be resolved as that in xxx module.

...

interface ModuleFactory extends TableFactory {
   Module createModule(String name, Map<String, String> properties);
}

...

We currently only move built-in functions into CoremoduleCoreModule.

public class CoreModule implements Module {
  public static final Core INSTANCE = new Core();

  private CoreModule() { }

  @Override
  public Set<String> listFunctions() {
    return BuiltInFunctionDefinitions.getDefinitions().stream()
        .map(f -> f.getName())

        .collect(Collectors.toSet());
  }

  @Override
  public Optional<BuiltInFunctionDefinition> getFunctionDefinition(String name) {
    return BuiltInFunctionDefinitions.getDefinitions().stream()
          .filter(f -> f.getName().equalsIgnoreCaseequals(name))
          .findFirst();
  }
}

...

public abstract class HiveModuleFactory implements ModuleFactory {

  @Override
  public Module createModule(String name, Map<String, String> properties) {
    return new HiveModule(properties.get("hive-version"));
  }

  @Override
  public Map<String, String> requiredContext() {
    return new HashMap<String, String>() {{
      put("type", "hive");
    }};
  }

  @Override
  public List<String> supportedProperties() {
    return Arrays.asList("hive-version");
  }
}

...

Java/Scala:

tableEnv.loadModule(CoreModule.INSTANCE"hive", new HiveModule("2.2.1"));

...