Status

...

...

...

...

Motivation

As Flink clusters using Kerberos authenticated services (just talking to HDFS, HBase, Kerberos enabled Kafka or even running on YARN) scale to hundreds of nodes we will inevitably hit a scalability challenge with Kerberos authentication. In Spark Streaming this became a limitation at around 200 node clusters and led to job and cluster outages due to denied authentication by the Kerberos Key Distribution Center (KDC). We can build on the implementation and lessons learned by the Spark community before this becomes an issue.

...

public interface DelegationTokenProvider {

  String name(serviceName();

  void init(Configuration configuration);

  boolean isTokenRequired();

  long obtainToken(Credentials credentials: Credentials);

}

With this new API any custom delegation token provider can be implemented easily.

...

• security.kerberos.relogin.period (default: 60000): The time in ms when keytab login happens automatically in order to always have a valid TGT.
• security.kerberos.tokens.retry-wait (default: 3600000): The time in ms how long to wait before retrying to obtain new delegation tokens after a failure.
• security.kerberos.tokens.renewal-ratio (default: 0.75): Ratio of the tokens's expiration time when new credentials should be re-obtained.
• security.kerberos.tokensdelegation.token.${providerserviceName}.enabled (default: true): Controls whether to obtain delegation token for a specific provider when security is enabled. By default, delegation tokens for all supported providers are retrieved when those services are configured, but it's possible to disable that behavior if it somehow conflicts with the application being run.

...