Proposers
- lamber-ken
- ...
Approvers
Status
Current state:Under Discussion
Discussion thread: here
JIRA: here
Released: <Hudi Version>
Abstract
Currently, many configuration items and their default values are dispersed in the config file like HoodieWriteConfig. It’s very confused for developers, and it's easy for developers to use them in a wrong place especially when there are more and more configuration items. If we can solve this, developers will benefit from it and the code structure will be more concise.
Background
Here are some related concepts that might be useful to know.
- DefaultHoodieConfig - Default way to load hudi config through a properties
- HoodieWriteConfig - Storing many config items and their default values, also contains a builder which can add some configs by with* method, like withIndexConfig, withCompactionConfig and so on.
From these files, we can see that these configuration items are centralized in a fat config file. So it's need to introduce a class that binds the key and the defalut value together and
group these configuration items by component, also needs to provide a mechainsm which can return the default value when no value is mapped.
Implementation
Here, I will explain how to implement it, including the ConfigOption and the extract method.
1, A ConfigOption is a container which contains the config key and the defalut value.
public class ConfigOption<V> { private final String key; private final V defaultValue; ConfigOption(String key, V defaultValue) { this.key = key; this.defaultValue = defaultValue; } public String key() { return key; } public V defaultValue() { return defaultValue; } public static ConfigOption.OptionBuilder key(String key) { return new ConfigOption.OptionBuilder(key); } public static final class OptionBuilder { private final String key; OptionBuilder(String key) { this.key = key; } public <V> ConfigOption<V> defaultValue(V value) { return new ConfigOption<>(key, value); } } }
2, Provide a mechainsm which can return the default value when no value is mapped.
public class HoodieConfig { private HashMap<String, Object> confData; public Integer getInteger(ConfigOption<Integer> configOption) { Object rawValue = getRawValue(configOption.key()); return convertToInt(rawValue, configOption.defaultValue()); } private Object getRawValue(String key) { return this.confData.get(key); } private Integer convertToInt(Object obj, Integer defaultValue) { if (obj == null) { return defaultValue; } try { return Integer.parseInt(obj.toString()); } catch (NumberFormatException e) { return defaultValue; } } }
How to use
This part will explain each component how to work with ConfigOption. Take GraphiteReporter as an example.
1, We need define the options which related to GraphiteReporter in a independent class
public class GraphiteReporterOptions { public static final ConfigOption<String> GRAPHITE_SERVER_HOST = ConfigOption .key("hoodie.metrics.graphite.reporter.host") .defaultValue("GRAPHITE"); public static final ConfigOption<Integer> GRAPHITE_SERVER_PORT = ConfigOption .key("hoodie.metrics.graphite.reporter.port") .defaultValue(4756); }
2, Init the reporter
public class GraphiteReporter { public void initRepoter(HoodieConfig config) { String host = config.getString(GraphiteReporterOptions.GRAPHITE_SERVER_HOST); Integer port = config.getInteger(GraphiteReporterOptions.GRAPHITE_SERVER_PORT); LOG.info("GraphiteReporter started with host:{}, port:{}", host, port); } }
Rollout/Adoption Plan
In order to maintain backward compatibility, we need to rewrite methods prefixed with "with" in config class. By the way, these "Builders" built in config class are redundant if we apply this new configuration framework.
Test Plan
Unit tests to cover all operations
Testing using production workloads.