...

Discussion thread: 

JIRA: 

Motivation

Anchor
motivation motivation

There are a number of usability issues that are related to the current need to edit possibly large XML topology files within the Apache Knox Admin UI and within

...

This whole thing may be able to be eliminated or replaced by Service Discovery or may need to continue to be supported with some improvements for deployments that do not have another registry like Ambari available.

We will need to make sure that we continue to meet the needs that this existing mechanism provides.

...

to make sure that we continue to meet the needs that this existing mechanism provides.

2. Simplified Topology Descriptor

Addressing item #3 from the Motivation section, a simplified topology descriptor must allow the following details to be specified:

• Service discovery type
  An identifier indicating which type of discovery to apply (e.g., Ambari, etc...)
• Service discovery address
  The associated service registry address
• A provider configuration reference (a unique name, filename, etc...)
  A unique name representing a set of provider configuration
• A list of services to be exposed through Knox (with optional URL value)
• A list of UIs to be proxied by Knox (per KIP-9)

YAML offers more structure than a properties file, but the representation is not important so long as the necessary contents can be expressed clearly.

# Discovery info source
discovery-type : AMBARI
discovery-address : http://c6401.ambari.apache.org:8080

# Provider config reference, the contents of which will be
# included in (or referenced from) the resulting topology descriptor.
# The contents of this reference has a <gateway/> root, and
# contains <provider/> configurations.
provider-config-ref : ambari-cluster-policy.xml

# The cluster for which the service details should be discovered
cluster: mycluster

# The services to declare in the resulting topology descriptor,
# whose URLs will be discovered (unless a value is specified)
services:
    - NAMENODE
    - JOBTRACKER
    - WEBHDFS
    - WEBHCAT
    - OOZIE
    - WEBHBASE
    - HIVE
    - RESOURCEMANAGER
    - AMBARI : http://c6401.ambari.apache.org:8080
    - AMBARIUI : http://c6401.ambari.apache.org:8080

# UIs to be proxied through the resulting Knox topology (see KIP-9)
#uis:
#    - AMBARIUI : http://c6401.ambari.apache.org:8080

3.Topology Generation

In order to simplify what management UI's need to manage, we should support a much simpler descriptor that contains what is needed to generate an actual topology file at deployment time.

...

My proposal is that for such deployments we also implement a Zookeeper based discovery mechanism. Then all instances will pick up the change from ZK as the central source of truth.

...

4. Service Discovery

In order to transform the simple descriptor described in #2 into a full topology, we need some source for the relevant details and metadata.

...

Ambari should contain everything we need to flesh out a full topology from a minimal description of what resources that we want exposed and how we want access to those resources protected.

...

4.1 Apache Ambari API

These are some excerpts of responses from the Ambari REST API which can be employed for topology discovery:

...

"items" : [
  {
    "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/CLUSTER_NAME/components/HCAT",
    "ServiceComponentInfo" : {
      "cluster_name" : "CLUSTER_NAME",
      "component_name" : "HCAT",
      "service_name" : "HIVE"
    }
  },
  {
    "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/CLUSTER_NAME/components/HIVE_SERVER",
    "ServiceComponentInfo" : {
      "cluster_name" : "CLUSTER_NAME",
      "component_name" : "HIVE_SERVER",
      "service_name" : "HIVE"
    }
  },
  {
    "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/CLUSTER_NAME/components/NAMENODE",
    "ServiceComponentInfo" : {
      "cluster_name" : "CLUSTER_NAME",
      "component_name" : "NAMENODE",
      "service_name" : "HDFS"
    }
  },
  {
    "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/CLUSTER_NAME/components/OOZIE_SERVER",
    "ServiceComponentInfo" : {
      "cluster_name" : "CLUSTER_NAME",
      "component_name" : "OOZIE_SERVER",
      "service_name" : "OOZIE"
    }
  }
]

...

4.2 Zookeeper Service Registry

•  TODO: provide some sense of completeness, APIs and examples

...

5. Provider Configurations

KIP-1 touched on an improvement for Centralized Gateway Configuration for LDAP/AD and the ability to import topology fragments into other topologies by referencing them by name within the topology.

...