Page History

...

Just as is currently done for topology files, Knox can monitor a local directory for new or changed descriptors, and trigger topology generation and deployment upon such events.
This is great for development and small cluster deployments.

The Knox Topology Service will monitor two additional directories:

• conf/shared-providers
  • Referenced provider configurations will go in this directory; These configurations are the <gateway/> elements found in topology files.
  • When a file is modified (create/update) in this directory, any descriptors that reference it are updated to trigger topology regeneration to reflect any provider configuration changes.
  • Attempts to delete a file from this directory via the admin API will be prevented if it is referenced by any descriptors in conf/descriptors.
    
    

• conf/descriptors
  • Simple descriptors will go in this directory.
  • When a file is modified (create/update) in this directory, a topology file is (re)generated in the conf/topologies directory.
  • When a file is deleted from this directory, the associated topology file in conf/topologies is also deleted, and that topology is undeployed.
  • When a file is deleted from the conf/topologies directory, the associated descriptor in conf/descriptors is also deleted (if it exists), to prevent unintentional regeneration/redeployment of the topology.

3.1.2 Remote

For production and larger deployments, we need to be able to accommodate multiple instances of Knox better. My One proposal for such cases is a ZooKeeper-based discovery mechanism.
Then, all All Knox instances will pick up the changes from ZK as the central source of truth, and perform the necessary generation and deployment of the corresponding topology.

The location of these descriptors and their dependencies (e.g., referenced provider config) in ZK must be defined.

It would also be helpful to provide a means (e.g., Ambari, Knox admin UI, CLI, etc...) by which these descriptors can be easily published to the correct location in a znode.

...

Since the service URLs for a cluster will be discovered, Knox has the opportunity to respond dynamically to subsequent topology changes. For a Knox topology that has been generated and deployed, it's possible that the URL for a given service could change at some point afterward.
The host name could change. The scheme and/or port could change (e.g., http --> https). The potential and frequency of such changes certainly varies among deployments.
We should consider providing the option for Knox to detect topology changes for a cluster, and respond by updating its corresponding topology.

For example, Ambari provides the ability to request the active configuration versions for all the service components in a cluster. There could be a thread that checks this set, notices one or more version changes, and initiates the re-generation/deployment of that topology.

Another associated benefit is the capability for Knox to interoperate with Ambari instances that are unaware of the Knox instance. Knox no longer MUST be managed by Ambari.

5. Provider Configurations

...