Status
Current state: [One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: here (<- link to https://mail-archives.apache.org/mod_mbox/flink-dev/)
JIRA: here (<- link to https://issues.apache.org/jira/browse/FLINK-XXXX)
Released: <Flink Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The current flink-conf.yaml parser in FLINK is not a standard YAML parser, which has some shortcomings. Firstly, it does not support nested structure configuration items and only supports key-value pairs, resulting in poor readability. Secondly, if the value is a collection type, such as a List or Map, users are required to write the value in a FLINK-specific pattern, which is inconvenient to use. Additionally, the parser of FLINK has some differences in syntax compared to the standard YAML parser, such as the syntax for parsing comments and null values. These inconsistencies can cause confusion for users, as seen in FLINK-15358 and FLINK-32740.
By supporting standard YAML, these issues can be resolved, and users can create a Flink configuration file using third-party tools and leverage some advanced YAML features. Therefore, this FLIP aims to introduce a standard YAML parser for parsing the FLINK configuration file.
Public Interfaces
Introduce the flink-config.yaml configuration file as the next generation of Flink configuration files. When flink-config.yaml exists in the Flink conf directory, Flink will use the standard YAML parser to parse it as the Flink configuration.
Modify the Flink packaging process so that when generating flink-dist, the flink-conf.yaml file is no longer generated in the conf directory. Instead, a flink-config.yaml file that conforms to the standard YAML syntax will be generated.
For compatibility reasons, in Flink 1.x, if the old configuration file flink-conf.yaml exists in the Flink conf directory, Flink will ignore the new configuration file flink-config.yaml and use the old parser to parse flink-conf.yaml as the Flink configuration. In Flink 2.x, Flink will no longer support parsing the old configuration file flink-conf.yaml.
To reduce the migration barrier for users and improve compatibility, we plan to extend the functionality of the standard YAML parser. The extended functionality will allow the standard YAML parser to parse List or Map type configOptions according to the FLINK-specific pattern.
Compatibility, Deprecation, and Migration Plan
Compatibility:
There is no compatibility issue here because we use different configuration file names to ensure compatibility. The default configuration file will be changed to "flink-config.yaml" and parsed by the standard YAML parser. If users prefer to use the old parser, they can create a "flink-conf.yaml" file in the conf directory.
Note that there are some behavior changes when using the standard YAML parser as follows:
Standard YAML parser
FLINK parser
ConfigOption key
The key of a ConfigOption cannot be a prefix of another option's key.
No requirements.
Comment
Comments must be separated from other tokens by white space characters.
Anything after the '#' symbol is considered a comment.
Null value
Parses null or blank values as null.
Anything after the first ':' symbol is treated as a string type value, except for blank values.
Special characters that need to be escaped when used in value
Standard YAML have some indicator characters and it is necessary to escape it in quotes to indicate that it is a string type when they used as part of a string value.
The indicator characters in standard YAML are as follows, and more details can be referred to at: https://yaml.org/spec/1.2.2/
- -
- ?
- :
- ,
- [
- ]
- {
- }
- #
- &
- *
- !
- |
- >
- '
- "
- %
- @
FLINK parser require escape values by quotes when the config option key is List type or Map type has some special characters:
- List type:
- When the semicolon (;) is part of the value rather than a separator for list elements.
- Map type:
- When the comma (,) is part of the value rather than a separator for Map elements
- When the colon (:) is part of the value rather than a separator for key-value pairs.
Deprecation:
The old configuration file "flink-conf.yaml" will be deprecated by announcing this change in the release notes and user documentation.
Migration Plan:
In FLINK-2.0, the old parser will no longer be supported, nor will the flink-conf.yaml file be used as a configuration file.
Test Plan
The change will be tested via UT cases and e2e tests.
Rejected Alternatives
Reject using a standard YAML parser to parse “flink-conf.yaml”
To avoid unexpected behavior changes for FLINK users, it is not reliable to use a standard YAML parser to parse the existing flink-conf.yaml file. This is because using different parsers to parse the same configuration file can result in potential changes in the parsing results, which can be difficult for users to notice. The following are examples that would cause a breaking change:
Comment
The FLINK parser considers anything after the '#' symbol as a comment, while the standard YAML parser required "Comments must be separated from other tokens by white space characters". If the user configures "key1: value1#comment1" in "flink-conf.yaml", the old parser will regard "value1" as the value and "#comment1" as a comment. However, the standard YAML parser will regard "value1#comment1" as the value.
2. Null value
The FLINK parser considers anything after the first ':' symbol as string type value. However, the standard YAML parser parses null or blank as null, unless it is enclosed in double quotes, such as "null".