You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Proposers

Approvers

Status

Current state:Under Discussion

Discussion thread: here

JIRA: here

Released: <Hudi Version>

Abstract

Currently, many configuration items and their default values are dispersed in the config file like HoodieWriteConfig. It’s very confused for developers, and it's easy for developers to use them in a wrong place especially when there are more and more configuration items. If we can solve this, developers will benefit from it and the code structure will be more concise. 

Background

Here are some related concepts that might be useful to know. 

  • DefaultHoodieConfig - Default way to load hudi config through a properties
  • HoodieWriteConfig - Storing many config items and their default values, also contains a builder which can add some configs by with* method, like withIndexConfig, withCompactionConfig and so on.

From these files, we can see that these configuration items are centralized in a fat config file. So it's need to introduce a class that binds the key and the defalut value together and 

group these configuration items by component, also needs to provide a mechainsm which can return the default value when no value is mapped.

Implementation

Here, I will explain how to implement it, including the ConfigOption and the extract method.

1, A ConfigOption is a container which contains the config key and the defalut value. 

public class ConfigOption<V> {

    private final String key;
    private final V defaultValue;

    ConfigOption(String key, V defaultValue) {
        this.key = key;
        this.defaultValue = defaultValue;
    }

    public String key() { return key; }

    public V defaultValue() { return defaultValue; }

    public static ConfigOption.OptionBuilder key(String key) {
        return new ConfigOption.OptionBuilder(key);
    }

    public static final class OptionBuilder {

        private final String key;

        OptionBuilder(String key) { this.key = key; }

        public <V> ConfigOption<V> defaultValue(V value) {
            return new ConfigOption<>(key, value);
        }
    }
}


2, Provide a mechainsm which can return the default value when no value is mapped.

public class HoodieConfig {

    private HashMap<String, Object> confData;

    public Integer getInteger(ConfigOption<Integer> configOption) {
        Object rawValue = getRawValue(configOption.key());
        return convertToInt(rawValue, configOption.defaultValue());
    }

    private Object getRawValue(String key) {
        return this.confData.get(key);
    }

    private Integer convertToInt(Object obj, Integer defaultValue) {
        if (obj == null) {
            return defaultValue;
        }

        try {
            return Integer.parseInt(obj.toString());
        } catch (NumberFormatException e) {
            return defaultValue;
        }
    }
}

How to use

This part will explain each component how to work with ConfigOption. Take GraphiteReporter as an example.

1, We need define the options which related to GraphiteReporter in a independent class

public class GraphiteReporterOptions {

    public static final ConfigOption<String> GRAPHITE_SERVER_HOST = ConfigOption
            .key("hoodie.metrics.graphite.reporter.host")
            .defaultValue("GRAPHITE");

    public static final ConfigOption<Integer> GRAPHITE_SERVER_PORT = ConfigOption
            .key("hoodie.metrics.graphite.reporter.port")
            .defaultValue(4756);

}


2, Init the reporter

public class GraphiteReporter {

    public void initRepoter(HoodieConfig config) {
        
        String host = config.getString(GraphiteReporterOptions.GRAPHITE_SERVER_HOST);
        Integer port = config.getInteger(GraphiteReporterOptions.GRAPHITE_SERVER_PORT);

        LOG.info("GraphiteReporter started with host:{}, port:{}", host, port);
    }

}

Rollout/Adoption Plan

In order to maintain backward compatibility, we need to rewrite methods prefixed with "with" in config class. By the way, these "Builders" built in config class are redundant if we apply this new configuration framework.

Test Plan

Unit tests to cover all operations

Testing using production workloads.

  • No labels