Discussion thread |
---|
https://lists.apache.org/thread/gv0ro4jsq4o206wg5gz9z5cww15qkvb9 |
Vote thread |
---|
https://lists.apache.org/thread/7xfc5pwlcs29vr4c8bm7kbx3lq8n7r4m | |||||||||||||
JIRA |
| ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Release | - |
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The proposed change aims to
Public Interfaces
The introduction
Proposed Changes
In a YARN environment, the current handling of classpath configuration using the yarn.provided.lib.dirs property has shown limitations. The yarn.ship-files property is utilized to add specific files to the classpath by including their parent directories. This approach is particularly useful when certain resources, such as hive-site.xml, need to be accessible to the application. However, there's an issue when the application employs Thread.currentThread().getContextClassLoader().getResource("hive-site.xml") to load hive-site.xml, as the parent directory of the resource is not included within Flink's classpath. This limitation prevents the application from effectively accessing the required configuration files.
To address this limitation, this proposal suggests extending the current approach. The goal is to include the parent directory of files specified in yarn.provided.lib.dirs in the classpath. This extension will ensure that the resources contained within these directories are accessible within the application's classpath, allowing access to the required configuration files. This change will not modify the inclusion of JAR files designated in yarn.provided.lib.dirs, which will continue to be added to the classpath as-is.
Public Interfaces
There will not be any programming interface change. However, the behavior of yarn.provided.lib.dirs will change to include the parent directory of resource files in the classpath for non-JAR resource files. JAR files will continue to be added to the classpath as they are.
Proposed Changes
The proposed change involves enhancing the classpath configuration mechanism when using the yarn.provided.lib.dirs property. Specifically, the current method of utilizing yarn.ship-files to add specific files' parent directories to the classpath will be extended. This extension will encompass the inclusion of the parent directory of non-JAR files designated in yarn.provided.lib.dirs as well. This improvement aims to fix the issue where resources required for application configuration are inaccessible due to their parent directories not being included in Flink's classpath. JAR files designated in yarn.provided.lib.dirs will still be added to the classpath as they are without modification.Add
Compatibility, Deprecation, and Migration Plan
- No impact on the existing users.
- The proposed changes are change is not introducing new parameters to the configuration file, and not changing any existing behavior. Therefore, there is no need to phase out any older behavior.
- No special migration tools required.
- Since there are no changes being made to existing behavior, there is no need to remove anythingThere is an enhancement to the existing behavior.
Test Plan
- Unit Test
...
- : New unit tests will be introduced to ensure proper inclusion of the parent directories of files specified in yarn.provided.lib.dirs to the classpath along with jar resource files.
Manual Test: Add following configuration to flink-conf.yaml and submit Flink job.
Code Block yarn.provided.lib.dirs: hdfs:///user/testuser/flinkTest/mapreduce/;hdfs:///user/testuser/flinkTest/lib/;hdfs:///user/testuser/flinkTest/plugins/;hdfs:///user/testuser/flinkTest/opt/;hdfs:///user/testuser/resources/;
Rejected Alternatives
No other ways to accomplish the same.