...

...

Knox is capable of monitoring Apache ZooKeeper for provider configurations and descriptors. To enable this, the following properties must be defined in gateway-site.xml.
(N.B., The address component of the gateway.remote.config.registry.sandbox-zookeeper-client property value must correspond to the ZooKeeper being used.)

<property<property>
    <name>gateway.remote.config.registry.sandbox-zookeeper-client</name>
    <value>type=ZooKeeper;address=localhost:2181</value>
    <description>ZooKeeper configuration registry client.</description>
</property>

<property>
    <name>gateway.remote.config.monitor.client</name>
    <value>sandbox-zookeeper-client</value>
    <description>Remote configuration monitor client name.</description>
</property>

...

<gateway>
    <provider>
        <role>authentication</role>
        <name>ShiroProvider</name>
        <enabled>true</enabled>
        <param>
       <param name="sessionTimeout" value="30"/> <name>sessionTimeout</name>
        <param name="main.ldapRealm" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm"/> <value>30</value>
        <param name="main.ldapContextFactory" value="org</param>
        <param>
         <name>main.ldapRealm</name>
         <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory"KnoxLdapRealm</>value>
        <param name="main.ldapRealm.contextFactory" value="$ldapContextFactory"/></param>
        <param name="main.ldapRealm.userDnTemplate" value="uid={0},ou=people,dc=hadoop,dc=apache,dc=org"/><param>
         <name>main.ldapContextFactory</name>
        <param name="main.ldapRealm.contextFactory.url" value="ldap://localhost:33389"/> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
        <param name="main.ldapRealm.contextFactory.authenticationMechanism" value="simple"/></param>
        <param name="urls./**" value="authcBasic"/>
<param>
      </provider>
    <provider>
<name>main.ldapRealm.contextFactory</name>
         <role>identity-assertion<<value>$ldapContextFactory</role>value>
        <name>Default<</name>param>
        <enabled>true</enabled>
<param>
         </provider>
</gateway>

...

Simple descriptors, as the name implies, are intended to simplify the declaration of which services should be included in a topology. By including a few discovery details, Knox can determine the endpoint URLs for the declared services.
The endpoint URLs can still be specified in these descriptors, for instance, if Ambari is not managing the target cluster or you need to override the endpoint for some reason.

Create discovery-sandbox.json with the following content:

In this descriptor, the discovery-user property value needs to match the Ambari user with at least Cluster User permissions. For this example, it is either maria_dev (for the HDP sandbox) or whatever user
you've defined for this purpose (see Ambari Configuration).

{
  "discovery-address":"http://localhost:8080",
  "discovery-user":"discovery",
  "provider-config-ref":"sandbox-providers",
  "cluster":"Sandbox",
  "services":[
    {"name":"NAMENODE"},
    {"name":"JOBTRACKER"},
    {"name":"WEBHDFS"},
    {"name":"WEBHCAT"},
    {"name":"OOZIE"},
    {"name":"WEBHBASE"},
    {"name":"RESOURCEMANAGER"}
  ]
}

...

These configuration files need to be deployed to the Knox instance, which will result in the generation and deployment of the discovery-sandbox.xml topology.
There are multiple means to accomplish this, including simple file copies (if you have access to the gateway host filesystem) and the Admin API.

File Copy

If you have access to the Knox host filesystem, you can simply copy the provider configuration and simple descriptor to the {GATEWAY_HOME}/conf/shared-providers and {GATEWAY_HOME}/conf/descriptors directories respectively.
For obvious reasons, it's important to deploy the provider configuration before any referencing descriptors.

cp sandbox-providers.xml {GATEWAY_HOME}/conf/shared-providers/
cp discovery-sandbox.json {GATEWAY_HOME}/conf/descriptors/

Admin API

Alternatively, the Knox Admin API can be used to deploy the configuration files

Replace these placeholders in the following URLs: {gateway-host} (e.g., localhost), {gateway-port} (e.g., 8443), {gateway-path} (e.g., gateway)

Deploy the Provider Configuration

curl -iku admin:admin-password https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/providerconfig/sandbox-providers \
     -X PUT \ 
     -H Content-Type:application/xml \
     -d "@sandbox-providers.xml"

Deploy the Descriptor

curl -iku admin:admin-password https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/descriptors/discovery-sandbox \
     -X PUT \
     -H Content-Type:application/json \
     -d "@discovery-sandbox.json"

Following either method of deployment, review the resulting topology file, discovery-sandbox.xml, and you'll find that it's a normal Knox topology, with all of the service URLs populated according to the cluster being proxied.

Ambari Cluster Monitoring

Knox is capable of monitoring Ambari instances, from which it has previously discovered cluster details, to identify changes that affect deployed topologies.

When the monitor discovers changes, it will trigger regeneration of the affected deployed topologies based on the changes, and redeploy the same.

...

To see this in action, try modifying a configuration property for a service in your cluster:

...

ZooKeeper Configuration Monitoring

We've seen two means for deploying provider configurations and simple descriptors to a Knox instance: The host filesystem and the Admin API.

Knox is also capable of monitoring Apache ZooKeeper for the addition/modification/removal of provider configurations and simple topology descriptors, and effecting gateway changes as a result.

When a provider configuration is added or modified under the monitored /knox/config/shared-providers znode, every Knox instance that is monitoring this znode will download the new file to its local {GATEWAY_HOME}/conf/shared-providers directory. Note that updates to a provider configuration will result in updates to every generated topology that references it.

When a descriptor is added or modified under the monitored /knox/config/descriptors znode, every Knox instance that is monitoring this znode will download the new file to its local {GATEWAY_HOME}/conf/descriptors directory, and attempt to generate a toploogy based on its contents.

Removals from these znodes are treated similarly, in that they result in the removal of the corresponding local files. When a descriptor is removed, it results in the undeployment of the corresponding topology.

KnoxCLI Support for Configuration in ZooKeeper

The Knox CLI provides some commands to faciliate easier management of these configuration files in ZooKeeper:

Try It

Invoke the Knox CLI commands to upload the configuration files you've created to ZooKeeper:

      {GATEWAY_HOME}/bin/knoxcli.sh upload-provider-config sandbox-providers.xml --registry-client sandbox-zookeeper-client --entry-name alt-providers.xml

...

<name>main.ldapRealm.userDnTemplate</name>
         <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
        </param>
        <param>
         <name>main.ldapRealm.contextFactory.url</name>
         <value>ldap://localhost:33389</value>
        </param>
        <param>
         <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
         <value>simple</value>
        </param>
        <param>
         <name>urls./**</name>
         <value>authcBasic</value>
        </param>
    </provider>
</gateway>

Create a Simple Descriptor

Simple descriptors, as the name implies, are intended to simplify the declaration of which services should be included in a topology. By including a few discovery details, Knox can determine the endpoint URLs for the declared services.
The endpoint URLs can still be specified in these descriptors, for instance, if Ambari is not managing the target cluster or you need to override the endpoint for some reason.

Create discovery-sandbox.json with the following content:

In this descriptor, the discovery-user property value needs to match the Ambari user with at least Cluster User permissions. For this example, it is either maria_dev (for the HDP sandbox) or whatever user
you've defined for this purpose (see Ambari Configuration).

{
  "discovery-address":"http://localhost:8080",
  "discovery-user":"discovery",
  "provider-config-ref":"sandbox-providers",
  "cluster":"Sandbox",
  "services":[
    {"name":"NAMENODE"},
    {"name":"JOBTRACKER"},
    {"name":"WEBHDFS"},
    {"name":"WEBHCAT"},
    {"name":"OOZIE"},
    {"name":"WEBHBASE"},
    {"name":"RESOURCEMANAGER"}
  ]
}

Deploy the Configuration

These configuration files need to be deployed to the Knox instance, which will result in the generation and deployment of the discovery-sandbox.xml topology.
There are multiple means to accomplish this, including simple file copies (if you have access to the gateway host filesystem) and the Admin API.

File Copy

If you have access to the Knox host filesystem, you can simply copy the provider configuration and simple descriptor to the {GATEWAY_HOME}/conf/shared-providers and {GATEWAY_HOME}/conf/descriptors directories respectively.
For obvious reasons, it's important to deploy the provider configuration before any referencing descriptors.

cp sandbox-providers.xml {GATEWAY_HOME}/conf/shared-providers/
cp discovery-sandbox.json {GATEWAY_HOME}/conf/descriptors/

Admin API

Alternatively, the Knox Admin API can be used to deploy the configuration files

Replace these placeholders in the following URLs: {gateway-host} (e.g., localhost), {gateway-port} (e.g., 8443), {gateway-path} (e.g., gateway)

Deploy the Provider Configuration

curl -iku admin:admin-password https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/providerconfig/sandbox-providers \
     -X PUT \ 
     -H Content-Type:application/xml \
     -d "@sandbox-providers.xml"

Deploy the Descriptor

curl -iku admin:admin-password https://{gateway-host}:{gateway-port}/{gateway-path}/admin/api/v1/descriptors/discovery-sandbox \
     -X PUT \
     -H Content-Type:application/json \
     -d "@discovery-sandbox.json"

Following either method of deployment, review the resulting topology file, discovery-sandbox.xml, and you'll find that it's a normal Knox topology, with all of the service URLs populated according to the cluster being proxied.

Ambari Cluster Monitoring

Knox is capable of monitoring Ambari instances, from which it has previously discovered cluster details, to identify changes that affect deployed topologies.

When the monitor discovers changes, it will trigger regeneration of the affected deployed topologies based on the changes, and redeploy the same.

Anchor
clustermonitortest clustermonitortest

Try It

To see this in action, try modifying a configuration property for a service in your cluster:

1. Navigate in Ambari to the Advanced yarn-site config, change the yarn.http.policy value to HTTPS_ONLY, and save the configuration.
2. Watch {GATEWAY_HOME}/logs/gateway.log for messages about noticing a cluster configuration change and regeneration/redeployment of the associated topology.
3. View the contents of the updated corresponding topology in {GATEWAY_HOME}/conf/topologies/, and note that the RESOURCEMANAGER URL now has an https scheme and a different port.
4. In Ambari, change the property value back to HTTP_ONLY, and note the subsequent update of the RESOURCEMANAGER URL in the re-generated topology.
   
   

ZooKeeper Configuration Monitoring

We've seen two means for deploying provider configurations and simple descriptors to a Knox instance: The host filesystem and the Admin API.

Knox is also capable of monitoring Apache ZooKeeper for the addition/modification/removal of provider configurations and simple topology descriptors, and effecting gateway changes as a result.

When a provider configuration is added or modified under the monitored /knox/config/shared-providers znode, every Knox instance that is monitoring this znode will download the new file to its local {GATEWAY_HOME}/conf/shared-providers directory. Note that updates to a provider configuration will result in updates to every generated topology that references it.

When a descriptor is added or modified under the monitored /knox/config/descriptors znode, every Knox instance that is monitoring this znode will download the new file to its local {GATEWAY_HOME}/conf/descriptors directory, and attempt to generate a toploogy based on its contents.

Removals from these znodes are treated similarly, in that they result in the removal of the corresponding local files. When a descriptor is removed, it results in the undeployment of the corresponding topology.

KnoxCLI Support for Configuration in ZooKeeper

The Knox CLI provides some commands to faciliate easier management of these configuration files in ZooKeeper:

Try It

Invoke the Knox CLI commands to upload the configuration files you've created to ZooKeeper:

      {GATEWAY_HOME}/bin/knoxcli.sh upload-provider-config sandbox-providers.xml --registry-client sandbox-zookeeper-client --entry-name alt-providers.xml

      {GATEWAY_HOME}/bin/knoxcli.sh upload-descriptor discovery-sandbox.json --registry-client sandbox-zookeeper-client --entry-name sandbox-copy.json

Knox will notice these additions, and the result will be sandbox-copy.xml in {GATEWAY_HOME}/conf/topologies.

Repeat the Ambari cluster configuration change test, and notice that both discovery-sandbox.xml and sandbox-copy.xml are regenerated and redeployed with updated RESOURCEMANAGER URLs.

Effect Policy Change Across Multiple Topologies

Add the hostmap provider to the local copy of sandbox-providers.xml you created earlier:

<gateway>
    <provider>
        <role>authentication</role>
        <name>ShiroProvider</name>
        <enabled>true</enabled>
        <param>
         <name>sessionTimeout</name>
         <value>30</value>
        </param>
        <param>
         <name>main.ldapRealm</name>
         <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
        </param>
        <param>
         <name>main.ldapContextFactory</name>
         <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
        </param>
        <param>
         <name>main.ldapRealm.contextFactory</name>
         <value>$ldapContextFactory</value>
        </param>
        <param>
         <name>main.ldapRealm.userDnTemplate</name>
         <value>uid=

Repeat the Ambari cluster configuration change test, and notice that both discovery-sandbox.xml and sandbox-copy.xml are regenerated and redeployed with updated RESOURCEMANAGER URLs.

Effect Policy Change Across Multiple Topologies

Add the hostmap provider to the local copy of sandbox-providers.xml you created earlier:

<gateway>
    <provider>
        <role>authentication</role>
        <name>ShiroProvider</name>
        <enabled>true</enabled>
        <param name="sessionTimeout" value="30"/>
        <param name="main.ldapRealm" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm"/>
        <param name="main.ldapContextFactory" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory"/>
        <param name="main.ldapRealm.contextFactory" value="$ldapContextFactory"/>
        <param name="main.ldapRealm.userDnTemplate" value="uid={0},ou=people,dc=hadoop,dc=apache,dc=org"/>,dc=org</value>
        </param>
        <param>
        <param name="main<name>main.ldapRealm.contextFactory.url" value="ldap://localhost:33389"/>url</name>
         <value>ldap://localhost:33389</value>
        </param>
        <param>
        <param name="main<name>main.ldapRealm.contextFactory.authenticationMechanism" value="simple"/>authenticationMechanism</name>
        <param name="urls./**" value="authcBasic"/>
<value>simple</value>
        </provider>param>
        <provider><param>
         <role>identity-assertion</role>
<name>urls./**</name>
         <name>Default<<value>authcBasic</name>value>
        <enabled>true<</enabled>param>
    </provider>
    <provider>
        <role>hostmap</role>
        <name>static</name>
        <enabled>true</enabled>
        <param>
       <param name="localhost" value="sandbox   <name>localhost</name>
          <value>sandbox,sandbox.hortonworks.com"com</>value>
        </param>
    </provider>
</gateway>

...