Status

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

At the moment there is no possibility to influence netty HTTP inbound/outbound requests. This could be useful from many perspectives. A good example (amongst others) is authentication/authorization. Here are the exact examples:

• With an outbound custom netty handler one can modify the HTTP request by adding authentication information.
• With an inbound custom netty handler one can do authentication/authorization.

The APIs are designed to be generic enough not to serve a simple use-case, so any other HTTP request modification is possible with them. 

In many secure enterprise environments having an explicit authentication layer for HTTP requests is part of the mandatory penetration testing procedure. Without adding this API layer, passing those penetration tests is unnecessarily complicated.

This API opens up the possibility to implement multiple authentication and authorization modules in the future. To keep the surface of Flink APIs and maintenance to a minimum our suggestion is to keep these authentication mechanisms outside of the main flink codebase and instead contribute them to the Flink Packages community site. To prove the viability of the API we commit to implementing HTTP Basic and Kerberos Authentication mechanism via it.

Proposed Changes

Important notes:

• The intended changes have no effect on Flink’s default behavior.
• Only HTTP requests can be custom handled.

For inbound/outbound HTTP requests new APIs are intended to be added. Please see New or Changed Public Interfaces for further details.

The mentioned handlers are intended to be loaded with Java Service Loader in the following places:

• RestClient -> OutboundChannelHandlerFactory
• WebFrontendBootstrap -> InboundChannelHandlerFactory
• RestServerEndpoint -> InboundChannelHandlerFactory

Multiple inbound/outbound handlers can be added but one must note in such cases the execution order of the handlers are random per direction. As a result, multiple handlers which are depending on each other is not advised. Such case handlers code can be merged instead.

When a new netty pipeline is built-up then each loaded factory is intended to be asked to create a custom handler. Here 3 options can happen:

• The mentioned factory realizes that no handler needs to be instantiated: This case createHandler returns null.
• The mentioned factory realizes that a handler needs to be instantiated but some required custom configurations are missing: This case ConfigurationException will be thrown.
• The mentioned factory realizes that a handler needs to be instantiated and all required custom configurations exist: This case a new instance of ChannelHandler will be returned.

As mentioned above we commit to publishing an example usage of these APIs (outside of Flink code) related the following protocols:

• Basic authentication
• Kerberos authentication

Public Interfaces

For inbound HTTP requests the following new API is intended to be added:

/**
 * Custom netty inbound handler factory in order to make custom changes on netty inbound data. Good
 * example usage of this API is custom authentication. When the user is not authenticated then the
 * instantiated channel handler can send back 401 to trigger negotiation. Since implementations are
 * loaded with service loader it's discouraged to store any internal state in factories.
 */

@Experimental
public interface InboundChannelHandlerFactory {
    /**
     * Gives back priority of the {@link ChannelHandler}. The more the value is, the earlier it is executed. If multiple handlers have the same priority then the order is not defined.
     *
     * @return the priority of the {@link ChannelHandler}.
     */
    int priority();

    /**
     * Creates new instance of {@link ChannelHandler}
     *
     * @param configuration The Flink {@link Configuration}.
     * @param responseHeaders The response headers.
     * @return {@link ChannelHandler} or null if no custom handler needs to be created.
     * @throws ConfigurationException Thrown, if the handler configuration is incorrect.
     */
    ChannelHandler createHandler(Configuration configuration, Map<String, String> responseHeaders)
            throws ConfigurationException;
}

For outbound HTTP requests the following new API is intended to be added:

/**
 * Custom netty outbound handler factory in order to make custom changes on netty outbound data.
 * Good example usage of this API is custom authentication. When the user wants to send
 * authentication information then the instantiated channel handler can modify the HTTP request.
 * Since implementations are loaded with service loader it's discouraged to store any internal state
 * in factories.
 */

@Experimental
public interface OutboundChannelHandlerFactory {

    /**
     * Gives back priority of the {@link ChannelHandler}. The more the value is, the earlier it is executed. If multiple handlers have the same priority then the order is not defined.
     *
     * @return the priority of the {@link ChannelHandler}.
     */
    int priority();

    /**
     * Creates new instance of {@link ChannelHandler}
     *
     * @param configuration The Flink {@link Configuration}.
     * @return {@link ChannelHandler} or null if no custom handler needs to be created.
     * @throws ConfigurationException Thrown, if the handler configuration is incorrect.
     */
    ChannelHandler createHandler(Configuration configuration) throws ConfigurationException;
}

This interface design is very minimal and straightforward, results in the least possible expected maintenance for enabling a large number of use cases.

Compatibility, Deprecation, and Migration Plan

Using this API is optional, does not affect compatibility.

Test Plan

As we are aiming to only contribute the API to Flink the tests should be contained with each respective implementation. The API itself can be validated via implementing and successfully using it for multiple request handlers, which we aim to do via Basic and Kerberos authentication.

Rejected Alternatives

There was a proposal to add one weak and one strong authentication into Flink but there was no agreement on that. Please see the rejected FLIP here: https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit?usp=sharing

The main concerns were that it is difficult to agree on a future proof strong authentication protocol and that it would add too much extra maintenance approach.