Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The proposed change aims to enhance the security and usability of Flink on YARN by introducing YARN ACLs to Flink containers. Presently, there exists a significant limitation where users who are not running the application cannot access the Flink application logs on the YARN resource manager. This creates significant hurdles in troubleshooting application issues and gaining insights into the application's behavior. Furthermore, non-application users are allowed to terminate running applications, leading to unexpected termination. By introducing YARN ACLs, we can provide a more secure and controlled environment for Flink on YARN, enabling only authorized users and groups to view logs and kill applications, thus avoiding potential mishandling.

Public Interfaces

The introduction of YARN ACLs aims to enhance the security and control over Flink containers and applications running on YARN. The two new parameters, yarn.view.acls and yarn.modify.acls, enable administrators to define access control rules for viewing Flink application logs and modifying (terminating) running applications, respectively. YARN ACLs are defined using comma-separated lists of users and groups. The format for defining ACLs is typically <comma-separated-users> <space> <comma-separated-groups>. For example, a rule could be specified as user1,user2 group1,group2.

Two new parameters will be introduced to the flink-conf.yaml:

• yarn.view.acls : This parameter determines which users and groups are allowed to view the Flink application logs on the YARN resource manager. Users or groups listed in yarn.view.acls will have read-only access to the logs.
• yarn.modify.acls : This parameter controls which users and groups can modify (terminate) running Flink applications. Users or groups specified in yarn.modify.acls will have the authority to kill running applications.

When a user attempts to access Flink application logs or terminate an application, YARN checks the ACLs defined in yarn.view.acls and yarn.modify.acls respectively. If the user is listed in the appropriate ACL and has the required permissions, the operation is allowed. Otherwise, access is denied. The proposed changes also mentions testing the wildcard * to grant access to everyone. This means that if * is included in an ACL, it grants access to all users or groups, effectively bypassing any specific user or group restrictions.

By leveraging YARN ACLs, FLIP-312 aims to provide a more secure environment for Flink on YARN, allowing only authorized users and groups to access Flink application logs and terminate applications. This helps prevent unauthorized access and ensures better control over the system.

Proposed Changes

Add two new parameters to the flink-conf.yaml :

• yarn.view.acls
• yarn.modify.acls 

Compatibility, Deprecation, and Migration Plan

• No impact on the existing users.
• The proposed changes are introducing new parameters to the configuration file, and not changing any existing behavior. Therefore, there is no need to phase out any older behavior.
• No special migration tools required.
• Since there are no changes being made to existing behavior, there is no need to remove anything.

Test Plan

Test the changes manually by adding the parameters in the flink-conf.yaml file and running a WordCount job on a Yarn cluster.

• Test yarn.view.acls when application in Running state:

  • User submitting the job can view the logs.
  • Another user in the yarn.view.acls can view the logs.
  • Another user not in the list of yarn.view.acls cannot view the logs.

• Test yarn.modify.acls when application in Running state:

  • User submitting the job can kill the application job.
  • Another user in the yarn.modify.acls can kill the application job..
  • Another user not in the list of yarn.modify.acls cannot kill the application job.

• Test with wildcard * giving access to everyone.

Rejected Alternatives

No other ways to accomplish the same.