...

Current state: Under Discussion

Discussion thread:  http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-configuration-from-environment-variables-td47946.html– (discussion prior to FLIP)

JIRA: –

Released: –

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Describe the problems you are trying to solve.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• DataStream and DataSet API, including classes related to that, such as StreamExecutionEnvironment

Proposed Changes

Flink currently requires configuration to be written to file. By allowing to override this configuration through environment variables, configuration can be made much more flexible. This becomes particularly useful in Kubernetes scenarios where some of the configuration can be defined through secrets exposed as environment variables, e.g. access keys. Furthermore, Flink can benefit from this internally as well as this mechanism provides an easy way to randomize end-to-end test configuration, see FLINK-19520.

The specific approach proposed here is inspired by, and follows in large parts, the design of the equivalent feature of the Spring framework. This provides confidence as the feature has been excessively used already, and familiarity with developers who have a knowledge overlap.

Public Interfaces

No public interfaces apart from configuration are affected. Flink configuration is also not affected directly, but indirectly by virtue of allowing environment variable to override entries in the Flink configuration.

Proposed Changes

After the Flink configuration has been parsed from the file, environment variables (following a clearly defined naming schema, see below) are taken into consideration and are allowed to amend or even replace the parsed configuration. Allowing environment variables to take precedence over the configured values is a deliberate and important choiceDescribe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

...