...

To address this limitation, this is a proposal to extend the existing approach. The intention is to also incorporate the parent directory of files specified in yarn.provided.lib.dirs to the classpath. This extension will enable the inclusion of the necessary resources in these directories within the application's classpath, thereby providing access to the required configuration files.

Public Interfaces

No new public interfaces are introduced by this proposalThere will not be any programing interface change. However, the behavior of yarn.provided.lib.dirs will change to also include the parent directory of each file in the class path.

Proposed Changes

The proposed changes involve change involves enhancing the classpath configuration mechanism when using the yarn.provided.lib.dirs property. Specifically, the current method of utilizing yarn.ship-files to add specific files' parent directories to the classpath will be extended. This extension will encompass the inclusion of the parent directory of files designated in yarn.provided.lib.dirs as well. This improvement aims to fix the issue where resources required for application configuration are inaccessible due to their parent directories not being included in Flink's classpath.This FLIP also addresses a concern related to treating all non-Flink dist-jar or plugin files as resource files. This approach might be considered excessive and could lead to resource overloading. An alternative approach that offers greater accuracy and explicitness is to follow a pattern similar to the handling of plugins. Specifically, introducing a reserved directory named resources can be considered. All files placed within the resources directory would be explicitly treated as resources. This approach offers a structured and controlled way to manage resources while avoiding potential overloading.

Compatibility, Deprecation, and Migration Plan

...