Jira: KNOX-1247

KIP: KIP-11 Cloud Usecases

Introduction

This feature which ships with Knox 1.2.0 allows federation from one Knox instance to another. This is done by using  Header Based Pre-Auth authentication provider. Typically this is useful in cases where one has a hybrid, on-prem - cloud model, so that on-prem Knox instance can federate requests to the cloud instance which can be useful in cases like:

• WebHDFS calls to on-prem Knox instance get re-dispatched to the cloud instance/s and results in files being put to or read from HDFS in the cloud.
• Spark jobs submitted to Livy through on-prem instances get re-dispatched and are submitted as cloud workloads.
• MapReduce jobs submitted to YARN RM through Knox will be submitted as workloads to the cloud.

The downside of this approach is that it adds an additional hop to the request which can slow things down in some cases. It is also extremely critical to make sure "two-way-ssl" is enabled and trust is properly established between on-prem and cloud Knox instances by provisioning certificates, since Header Based Pre Auth authentication by itself is not secure, perimeter security around cloud Knox instance is a must, e.g. VPC, ip whitelisting for on-prem Knox instance.

Setup

The following diagram describes the federated request flow.

We need to provision certificate for cloud Knox instance into on-prem Knox instance and vice versa and enable two-way-ssl, as shown below.

We will look at the setting for on-prem and cloud Knox topologies

On-prem

Authentication provider (topology): any

Dispatch (topology): org.apache.knox.gateway.dispatch.HeaderPreAuthFederationDispatch

Federation header name (gateway-site.xml): gateway.custom.federation.header.name

For authentication, since we will be authenticating locally, we can use any authentication provider we choose i.e.  local LDAP.

We can update the dispatch for the service that needs to be federated (WEBHDFS in the following example). You can override dispatch for a service in the topology itself e.g.

	  <service>
          <role>WEBHDFS</role>
          <url>https://my.cloudurl.com:8443/gateway/aws/webhdfs</url>
		  <dispatch>
             <classname>org.apache.knox.gateway.dispatch.HeaderPreAuthFederationDispatch</classname>
             <use-two-way-ssl>true</use-two-way-ssl>
          </dispatch>
      </service> 

Following property specifies custom header name to be added to outgoing federated request (gateway-site.xml) which will be checked by the cloud HeaderPreAuth authentication provider.

    <property>
        <name>gateway.custom.federation.header.name</name>
        <value>aws_header</value>
        <description>Custom header name to be used for federated requests.</description>
    </property>

Cloud

Authentication provider (topology): HeaderPreAuth

For cloud Knox instance, we need to use the HeaderPreAuth authentication provider and specify the "preauth.custom.header " parameter, "preauth.custom.header" should be exactly same as value of property "gateway.custom.federation.header.name" defined in on-prem gateway-site.xml (aws_header in our example above)

Following is the relevant topology snippet

     <provider>          
         <role>federation</role> 
         <name>HeaderPreAuth</name>          
         <enabled>true</enabled>         
         <param>
              <name>preauth.custom.header</name>
              <value>aws_header</value>
           </param>          
     </provider>

That's all there is, now you topology based federation should be all setup.