Status

Current state: "Under DiscussionVoting"

Discussion thread: https://lists.apache.org/thread.html/a56c6b52e5f828d4a737602b031e36b5dd6eaa97557306696a8063a9@%3Cdev.flink.apache.org%3E

...

List of new interfaces:

• OptionBuilder#intType

Proposed Changes

Overview

Because config options are already used at a couple of places in the code base, we aimed to minimize the amount of changes necessary while enriching a config option with more declarative information.

...

• (...)/stringType(...)/...

• TypedConfigOptionBuilder, ListConfigOptionBuilder
• ConfigurationReader/ConfigurationWriter, ConfigurableFactory
• Configuration implements ConfigurationReader/ConfigurationWriter thus receives new get(...)/getOptional(...)/#set(...)
• ConfigOption#withValidator(...)/validate(...)/withExtendedDescription(...)
• OptionValidators, OptionValidator
• ConfigOptionSet, OptionSetValidator, OptionSetValidators

Proposed Changes

Overview

Because config options are already used at a couple of places in the code base, we aimed to minimize the amount of changes necessary while enriching a config option with more declarative information.

ConfigOptions.key("key")
.intType()
.defaultValue(12);

Proposed changes to ConfigOption:

In order for ConfigOption to contain information about the class it describes, we should add two additional fields to ConfigOption:

    private final Class atomicClass;

    private final boolean isList;

The atomicClass field describes the atomic type that this ConfigOption describes. There are 5 cases:

• atomicClass == e.g. Integer.class -> ConfigOption<Integer>
• atomicClass == Map.class -> ConfigOption<Map<String, String>>
• atomicClass == ? extends ConfigurableFactory<T> -> ConfigOption<T>
• atomicClass == Class.class -> for ConfigOption<Class<?>>
• atomicClass == e.g. Integer.class & isList = true for ConfigOption<List<Integer>>

This way we can describe all necessary types without backwards incompatible changes to the ConfigOption class

Proposed New Builder Pattern:

Example

The current builder pattern in ConfigOptions is not expressive enough to define a type, a list of types, or a type describing a class or enum. We suggest to introduce a new builder that can be accessed via:

ConfigOptions.key("key")
.intType()
.defaultValue(12);

Proposed changes to ConfigOption:

In order for ConfigOption to contain information about the class it describes, we should add two additional fields to ConfigOption:

    private final Class atomicClass;

    private final boolean isList;

The atomicClass field describes the atomic type that this ConfigOption describes. There are 5 cases:

• atomicClass == e.g. Integer.class -> ConfigOption<Integer>
• atomicClass == Map.class -> ConfigOption<Map<String, String>>
• atomicClass == ? extends ConfigurableFactory<T> -> ConfigOption<T>
• atomicClass == Class.class -> for ConfigOption<Class<?>>
• atomicClass == e.g. Integer.class & isList = true for ConfigOption<List<Integer>>

This way we can describe all necessary types without backwards incompatible changes to the ConfigOption class

Proposed New Builder Pattern:

The current builder pattern in ConfigOptions is not expressive enough to define a type, a list of types, or a type describing a class or enum. We suggest to introduce a new builder that can be accessed via:

ConfigOptions.key("key")

The entire The entire builder is defined as:

public static class OptionBuilder {
   	 private final String key;

   	 OptionBuilder(String key) {
   		 this.key = key;
   	 }

   	 TypedConfigOptionBuilder<Integer> intType() {
   		 return new TypedConfigOptionBuilder<>(key, Integer.class);
   	 }

   	 TypedConfigOptionBuilder<String> stringType() {
   		 return new TypedConfigOptionBuilder<>(key, String.class);
   	 }

   	 TypedConfigOptionBuilder<Duration> durationType() {
   		 return new TypedConfigOptionBuilder<>(key, Duration.class);
   	 }

   	 TypedConfigOptionBuilder<Map<String, String>> propertiesType() {
   		 return new TypedConfigOptionBuilder<>(key, Map.class);
   	 }

   	 <T> TypedConfigOptionBuilder<T> enumType(Class<T extends Enum<T>> clazz) {
   		 return new TypedConfigOptionBuilder<>(key, clazz);
   	 }
   	 
   	 // All supported atomic types: Boolean, Integer, Long, Double, Float, String, Duration, MemorySize, Enum, Properties(Map<String, String>)


   	 <T> TypedConfigOptionBuilder<T> configurableType(Class<? extends ConfigurableFactory<T>> clazz) {
   return new TypedConfigOptionBuilder<>(key, clazz);
}

   	 <T> TypedConfigOptionBuilder<Class<T>> classType(Class<T> clazz) {
   		 return new TypedConfigOptionBuilder<>(key, Class.class);
   	 }

    	/**
   	  * Creates a ConfigOption with the given default value.
   	  *
   	  * <p>This method does not accept "null". For options with no default value, choose
   	  * one of the {@code noDefaultValue} methods.
   	  *
   	  * @param value The default value for the config option
   	  * @param <T> The type of the default value.
   	  * @return The config option with the default value.
   	  */
   	 @Deprecated
   	 public <T> ConfigOption<T> defaultValue(T value) {
   		 checkNotNull(value);
   		 return new ConfigOption<>(key, value);
   	 }

   	 /**
   	  * Creates a string-valued option with no default value.
   	  * String-valued options are the only ones that can have no
   	  * default value.
   	  *
   	  * @return The created ConfigOption.
   	  */
   	 @Deprecated
   	 public ConfigOption<String> noDefaultValue() {
   		 return new ConfigOption<>(key, null);
   	 }

    }

    public static class TypedConfigOptionBuilder<T> {
   	 private final String key;
   	 private final Class clazz;

   	 TypedConfigOptionBuilder(String key, Class clazz) {
   		 this.key = key;
   		 this.clazz = clazz;
   	 }

   	 public ListConfigOptionBuilder<T> asList() {
   		 return new ListConfigOptionBuilder<>(key, clazz);
   	 }

   	 public ConfigOption<T> defaultValue(T value) {
   		 return new ConfigOption<>(
   			 key,
   			 clazz,
   			 false,
   			 Description.builder().text("").build(),
   			 value,
   			 EMPTY);
   	 }

   	 public ConfigOption<T> noDefaultValue() {
   		 return new ConfigOption<>(
   			 key,
   			 clazz,
   			 false,
   			 Description.builder().text("").build(),
   			 null,
   			 EMPTY);
   	 }
    }

    public static class ListConfigOptionBuilder<T> {
   	 private final String key;
   	 private final Class clazz;

   	 ListConfigOptionBuilder(String key, Class clazz) {
   		 this.key = key;
   		 this.clazz = clazz;
   	 }

   	 @SafeVarargs
   	 public final ConfigOption<List<T>> defaultValues(T... values) {
   		 return new ConfigOption<>(
   			 key,
   			 clazz,
   			 true,
   			 Description.builder().text("").build(),
   			 Arrays.asList(values),
   			 EMPTY);
   	 }

   	 public ConfigOption<List<T>> noDefaultValue() {
   		 return new ConfigOption<>(
   			 key,
   			 clazz,
   			 true,
   			 Description.builder().text("").build(),
   			 null,
   			 EMPTY);
   	 }
    }

...

 * <p>The validator is used when accessing (reading and writing) the value from the configuration.

 *

 * @see PredefinedValidatorsOptionValidators

 *

 * @param validator The validator for this option.

...

We suggest that the documentation generation would follow a Configurable a ConfigurableFactory class of an option and perform the same reflection-based lookup after public static options within the configurable class.

...

public static final ConfigOption<CachedFile> CACHED_FILE =
		key("cached-file")
			.configurableType(CachedFileFactory.class)
                        .defaultValue(new CashedFile());

class CachedFile implements Configurable {
	CachedFile("empty", "empty", false));

class CachedFileFactory implements ConfigurableFactory {
        public static final ConfigOption<String> PATH =
		key("path").defaultValue("path");

	public static final ConfigOption<String> FILE =
		key("file").defaultValue("file");

	public static final ConfigOption<Boolean> FLAG =
		key("flag").defaultValue(true);

	// call regular config options and assign fields:
	// fromConfiguration()/toConfiguration()
}

class CachedFile implements Configurable {
	

	private String path;
	private String file;
	private Boolean flag;

	public CachedFile()String { }

	// call regular config options and assign fields:
	// fromConfiguration()/toConfiguration()path, String file, String flag) { }
}

This would result in the following key-value Configuration:

...

/**
 * Validates a set of {@link ConfigOption}s in {@link ConfigOptionSet}.
 */
public interface SetValidatorOptionSetValidator {

	/**
	 * Validates and potentially correlates options stored in a {@link Configuration}.
	 *
	 * @param configuration Gives read-only access to the configuration.
	 * @return True if option is valid; false otherwise.
	 */
	boolean validate(ConfigurationReader configuration);

	/**
	 * Returns an explanatory string. It is used for printing exception as well as generating
	 * documentation.
	 *
	 * <p>For consistency, use camel-case syntax and square brackets for nesting.
	 *
	 * <p>For example, {@code minLongDelta[“min-retention”, ”max-retention”, 5]}.
	 */
	String explain();
}

Proposed utilities:

...

OptionSetValidators

We can offer a set of predefined validators such as `minLongDelta` that would verify the distance between two separate config options.

...

Group constraints:
- minSizeDelta[key1, key2, “512m”]

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the FLIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

, “512m”]

Compatibility, Deprecation, and Migration Plan

• All existing config options are still valid and have no changed behavior
• Deprecated ConfigOption#defaultValue(...)/noDefaultValue

Implementation Plan

Each feature section can be a separate commit or issue. Such as:

• New typed ConfigOption with builder pattern
• Option validation
• Option description extension
• Object options
• Lists
• Duration
• Memory Size
• Option set validation

Test Plan

The implementation can be tested with unit tests for every new feature section listed in Proposed Changes.

Rejected Alternatives

See corresponding feature sectionsIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.