Status

...

...

...

JIRA: here (<- link to https://issues.apache.org/jira/browse/FLINK-XXXX)

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Table of Contents

Motivation

`ConfigOption` and `Configuration` are crucial parts of the Flink project because every component in the stack needs possibilities of parameterization.

...

List of new interfaces:

• OptionBuilder#intType

Proposed Changes

Overview

• (...)/stringType(...)/...

• TypedConfigOptionBuilder, ListConfigOptionBuilder
• ConfigurationReader/ConfigurationWriter, ConfigurableFactory
• Configuration implements ConfigurationReader/ConfigurationWriter thus receives new get(...)/getOptional(...)/#set(...)
• ConfigOption#withValidator(...)/validate(...)/withExtendedDescription(...)
• OptionValidators, OptionValidator
• ConfigOptionSet, OptionSetValidator, OptionSetValidators

Proposed Changes

Overview

Because config options are already used at a couple of places in the code base, we aimed to minimize the amount of changes necessary while enriching a config option with more declarative information.

...

 * <p>The validator is used when accessing (reading and writing) the value from the configuration.

 *

 * @see PredefinedValidatorsOptionValidators

 *

 * @param validator The validator for this option.

...

We suggest that the documentation generation would follow a Configurable a ConfigurableFactory class of an option and perform the same reflection-based lookup after public static options within the configurable class.

...

public static final ConfigOption<CachedFile> CACHED_FILE =
		key("cached-file")
			.defaultValue(new CashedFile());

class CachedFile implements Configurable {
	configurableType(CachedFileFactory.class)
                        .defaultValue(new CachedFile("empty", "empty", false));

class CachedFileFactory implements ConfigurableFactory {
        public static final ConfigOption<String> PATH =
		key("path").defaultValue("path");

	public static final ConfigOption<String> FILE =
		key("file").defaultValue("file");

	public static final ConfigOption<Boolean> FLAG =
		key("flag").defaultValue(true);

	private String path;
	private String file;
	private Boolean flag;

	public CachedFile() { }

	// call regular config options and assign fields:
	// fromConfiguration()/toConfiguration()// call regular config options and assign fields:
	// fromConfiguration()/toConfiguration()
}

class CachedFile implements Configurable {
	

	private String path;
	private String file;
	private Boolean flag;

	public CachedFile(String path, String file, String flag) { }
}

This would result in the following key-value Configuration:

...

/**
 * Validates a set of {@link ConfigOption}s in {@link ConfigOptionSet}.
 */
public interface SetValidatorOptionSetValidator {

	/**
	 * Validates and potentially correlates options stored in a {@link Configuration}.
	 *
	 * @param configuration Gives read-only access to the configuration.
	 * @return True if option is valid; false otherwise.
	 */
	boolean validate(ConfigurationReader configuration);

	/**
	 * Returns an explanatory string. It is used for printing exception as well as generating
	 * documentation.
	 *
	 * <p>For consistency, use camel-case syntax and square brackets for nesting.
	 *
	 * <p>For example, {@code minLongDelta[“min-retention”, ”max-retention”, 5]}.
	 */
	String explain();
}

Proposed utilities:

...

OptionSetValidators

We can offer a set of predefined validators such as `minLongDelta` that would verify the distance between two separate config options.

...

We add group validation constraints to the end of the table:

Group constraints:
- minSizeDelta[key1, key2, “512m”]

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the FLIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

• All existing config options are still valid and have no changed behavior
• Deprecated ConfigOption#defaultValue(...)/noDefaultValue

Implementation Plan

Each feature section can be a separate commit or issue. Such as:

• New typed ConfigOption with builder pattern
• Option validation
• Option description extension
• Object options
• Lists
• Duration
• Memory Size
• Option set validation

Test Plan

The implementation can be tested with unit tests for every new feature section listed in Proposed Changes.

Rejected Alternatives

See corresponding feature sectionsIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.