Status

Current state: Under Discussion

Discussion thread: TBD

JIRA: TBD

Released: <Flink Version>

While we debate the proposal please review it in this GDoc, this facilitates direct comments for specific details. Please keep the discussion on the mailing list and in the GDoc rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As Flink clusters using Kerberos authenticated services (just talking to HDFS, HBase, Kerberos enabled Kafka or even running on YARN) scale to hundreds of nodes we will inevitably hit a scalability challenge with Kerberos authentication. In Spark Streaming this became a limitation at around 200 node clusters and led to job and cluster outages due to denied authentication by the Kerberos Key Distribution Center (KDC). We can build on the implementation and lessons learned by the Spark community before this becomes an issue.

Proposal

https://docs.google.com/document/d/1JzMbQ1pCJsLVz8yHrCxroYMRP2GwGwvacLrGyaIx5Yc/edit?fbclid=IwAR0vfeJvAbEUSzHQAAJfnWTaX46L6o7LyXhMfBUCcPrNi-uXNgoOaI8PMDQ#
As the community agrees on the approach we will move the content to this wiki page.