...
Compatibility:
There is no compatibility issue here because we use different configuration file names to ensure compatibility. The default configuration file will be changed to "flink-config.yaml" and parsed by the standard YAML parser. If users prefer to use the old parser, they can create a "flink-conf.yaml" file in the conf directory.
Note that there are some behavior changes when using the standard YAML parser compared to using current Flink parser:
Standard YAML parser
FLINK parser
ConfigOption key
The key of a ConfigOption cannot be a prefix of another option's key.
No requirements.
Comment
Comments must be separated from other tokens by white space characters.
Anything after the '#' symbol is considered a comment.
Null value
Parses null or blank values as null value.
Anything after the first ':' symbol is treated as a string type value, except for blank values.
Special characters
The indicator characters in standard YAML are as follows, and more details can be referred to at: https://yaml.org/spec/1.2.2/.
If you want to use the following special characters as a part of a string value, you need to escape them by using quotation marks.
- -
- ?
- :
- ,
- [
- ]
- {
- }
- #
- &
- *
- !
- |
- >
- '
- "
- %
- @
Currently, there are some special characters sequences in the FLINK parser, as follows:
- ": " (a colon and a whitespace)
- In the FLINK parser, the first ": " in a key-value pair is considered as the delimiter between the key and value, while the remaining ": " will be treated as a part of the value.
- '#'
- In the FLINK parser, anything after the first '#' is considered as a comment. So the hash tag (#) cannot be included as a part of the key-value pair.
- ';'
- When using the List type config option, the semicolon (;) is used as the delimiter between List elements. If you want to include ';' as a part of an element value, you need to escape it using quotation marks.
- ','
- When using the Map type config option, the comma (,) is used as the delimiter between map elements, If you want to include ',' as a part of the value, you need to escape it using quotation marks.
- ':'
- When using the Map type config option, the colon (:) is used as the delimiter between key and value in the map, If you want to include ':' as a part of the value, you need to escape it using quotation marks.
Duplicate key
Standard YAML does not allow duplicate keys in a map. It states that "The content of a mapping node is an unordered set of key/value node pairs, with the restriction that each of the keys is unique." For more details, please refer to: https://yaml.org/spec/1.2.2/#nodes.
FLINK parser allows users to configure duplicate keys, where the key-value pair that appears later in the file will override the earlier key-value pair.
Sequences style
Standard YAML has two styles for sequences, as follows:
- Flow Style:
- Flowsequence content is denoted by surrounding “
[
” and “]
” characters. Sequence entries are separated by a “,” character. More details can refer to: https://yaml.org/spec/1.2.2/#741-flow-sequences
- Flowsequence content is denoted by surrounding “
For example:
Code Block language yml title Flow style example [A, B, C]
- Blocking Style:
A block sequence is simply a series of nodes, each denoted by a leading “
-
” indicator. The “-
” indicator must be separated from the node by white space. More details can refer to: https://yaml.org/spec/1.2.2/#821-block-sequences.
For example:
Code Block language yml title Blocking style example - A - B - C
The sequence style in FLINK parser is separated by ";" (semicolon).
For example: A;B;C
Mapping style
Standard YAML has two styles for Mapping, as follows:
- Flow Style:
Flow mappings are denoted by surrounding “
{
” and “}
” characters. Mapping entries are separated by a “,” character. More details can refer to: https://yaml.org/spec/1.2.2/#742-flow-mappings
For example:
Code Block language yml title Flow style example {k1: v1, k2: v2, k3: v3}
- Blocking Style:
A Block mapping is a series of entries, each presenting a key/value pair and more details can refer to: https://yaml.org/spec/1.2.2/#822-block-mappings.
For example:
Code Block language yml title Blocking style example k1: v1 k2: v2 k3: v3
The Mapping style in FLINK parser separates key-value pairs using ":" (colon), and different key-value pairs are separated by "," (comma).
For example: k1:v1, k2:v2, k3:v3
Deprecation:
The old configuration file "flink-conf.yaml" will be deprecated by announcing this change in the release notes and user documentation.
Migration Plan:
In FLINK-2.0, the old parser will no longer be supported, nor will the flink-conf.yaml file be used as a configuration file.
...