THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Service Config | Property | Example Value | ||
|---|---|---|---|---|
| HIVE:hive-site | hive.zookeeper.quorum | host1:2181,host2:2181,host3:2181 | ||
| hive.server2.zookeeper.namespace | hiveserver2 | |||
| hive.server2.support.dynamic.service.discovery | true | |||
| HBASE:hbase-site | hbase.zookeeper.quorum | host1:2181,host2:2181,host3: | ||
| hbase.zookeeper.property.clientPort | 2181 | |||
| zookeeper.znode.parent | /hbase-unsecure | |||
| KAFKA:kafka-broker | zookeeper.connect | sandbox.hortonworks.com:2181 | ||
| HDFS:hdfs-site | ha.zookeeper.quorum | host1:2181,host2:2181,host3:2181 | ||
| dfs.ha.automatic-failover.enabled | true (only for "Auto HA") | |||
| OOZIE:oozie-site | oozie.zookeeper.connection.string | localhost:2181 | ||
| oozie.zookeeper.namespace | oozie | |||
| YARN:yarn-site | | true | ||
| host1:2181,host2:2181,host3:2181 | |||
| WEBHCAT:webhcat-site | templeton.zookeeper.hosts | host1:2181,host2:2181,host3:2181 | ||
| ATLAS:application-properties | atlas.kafka.zookeeper.connectyarn.resourcemanager.zk-address | host1:2181,host2:2181,host3:2181 | ||
| atlas.server.ha | webhcat-site | templeton.zookeeper.hostsconnect | host1:2181,host2:2181,host3:2181 | |
| atlas.server.ha.zookeeper.zkroot | /apache_atlas | |||
| atlas.server.ha.enabled | true |
Required Changes
- Knox will parse referenced provider configurations, rather than just blindly copying their contents.
This will serve multiple purposes:- It will provide a degree of validation (invalid provider configurations would fail parsing)
- It will allow a finer level of control over what is serialized to the generated topology file:
- Knox will have the opportunity to add cluster-specific details into provider configuration in the generated topologies
- Knox could add/modify the HaProvider config based on cluster details (e.g., HA enabled for a particular service)
- Knox could reference elements of the provider configuration (e.g., HaProvider) to inform the generation of service elements in the generated topologies.
...
- Name the service-level override param for the service-specific HaProvider param component named 'enabled'
- 'enabled'
- 'haEnabled'
- 'ha.enabled'
- Identify the nature of all the supported services
- Which services are currently purely topology-based (DefaultURLManager)?
- WEBHCAT
- OOZIE
- WEBHDFS
- ?
- Which services are currently ZooKeeper-based (BaseZooKeeperURLManager)?
- HIVE (HS2ZooKeeperURLManager)
- HBASE (HBaseZooKeeperURLManager)
- Kafka (KafkaZooKeeperURLManager)
- SOLR (SOLRZooKeeperURLManager)
- ATLAS (AtlasZookeeperURLManager)
- Could ZooKeeper support be added for any services which do no currently support it?
- These have zookeeper-related configuration (see table):
- WEBHDFS
- OOZIE
- YARN
- WEBHCAT
- RESOURCEMANAGER
- These have zookeeper-related configuration (see table):
- Which services are currently purely topology-based (DefaultURLManager)?
- For the ZooKeeper-based-HA services, determine if the ZooKeeper details are available from the service's configuration via Ambari.
- Can "HA mode" be determined for every service type from the cluster configuration details? Can Knox dynamically identify HA-configured services, and generate the topology accordingly?
- Determine how to leverage the cluster discovery data to generate the ZooKeeper HA configuration for the relevant declared topology services.
...