You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

With the new distributed configuration management feature of OODT, users now get the chance deploy large number of instances of OODT components without having to worry about manually configuring all those instances. This new feature allows any OODT component to retrieve configuration required from zookeeper on the fly.

Example Scenario

For example, lets consider the file manager component. Suppose that we want to deploy 20 file manager instances in a cluster. Configuring them one by one would be a tiresome task with the user repeating the same set of tasks 20 times, copying properties files, XML files and such other files related to configuration of that file manager instance. If we had to configure a file manager instance manually, 

  • First, we need to get the OODT distribution zip file and unzip it. Then we need to modify the content of the configuration files (ex: etc/filemgr.properties) to match our requirement.
  • Then upload filemgr directory to the corresponding remote server.
  • Then we need to start the file manager process in the remote server.

To configure all 20 nodes, this process needs to be repeated 20 times. This becomes even harder if we had to customize configuration files specifically to each node in the cluster. Distributed Configuration Management feature is the solution for this. All the required configuration files first need to be stored in zookeeper prior to the deployment. Once the files are stored properly, users can copy the executables to the remote servers and run that component. When being initialized, this component will retrieve and store all the configuration files and properties files from zookeeper making all the required configuration files available locally at runtime. This becomes even more useful when users need to change some configuration at runtime and wants them to be reflected at each node in the cluster without manually uploading them again. Therefore, the new distributed configuration management feature is capable of listening to configuration changes in runtime and retrieve the corresponding changes without the users having to upload those files to each server manually.

For all these to work, first we need to store the corresponding configuration files in zookeeper. Distributed Configuration Publisher is the component responsible for storing and managing configuration information in zookeeper. Let's see how should we store configuration in zookeeper.

Distributed Configuration Publishing

As mentioned in the example above, we need to store all the configuration files in zookeeper prior to deploying our OODT components in servers. Following steps need to be taken to store configuration in zookeeper.

Setting up Zookeeper

First, we need to setup a zookeeper cluster as described in zookeeper documentation. For testing purposes, we can use the standalone zookeeper server. But for production deployments we need to setup a zookeeper cluster as described in that documentation. Please make sure to use zookeeper version higher than 3.5.

Zookeeper Connect String

Once you setup zookeeper, please note down the zookeeper connect string which will be used by our Distributed Configuration Publisher to connect to zookeeper. The connect string looks like IP1:PORT1,IP2:PORT2,...,IPn:PORTn where IP is the ip address of each zookeeper node and port is the client port on which each zookeeper server is listening on.

Configuring configuration publisher

 

Now, download the latest distribution of apache OODT and extract it. There is a directory named config which contains the scripts required to publish configuration to zookeeper. Inside that directory will be bin, etc, examples and lib dirrectories. Out of those, bin contains conf-publisher executable. etc contains log4j.xml for logging configuration and config-publisher.xml for configuring the configuration to be stored in zookeeper. examples contains some example configuration files for resource manager and file manager. For us to publish/store configuration in zookeeper, we need to modify the config-publisher.xml in the etc directory. You will find the following default content in that file out of the box.

 

config-publisher.xml
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">

    <!-- Configuration publisher for File Manager OODT Component -->
    <bean id="filemgr-config-publisher" class="org.apache.oodt.config.distributed.cli.DistributedConfigurationPublisher">
        <constructor-arg value="filemgr"/>
        <property name="propertiesFiles">
            <map key-type="java.lang.String" value-type="java.lang.String">
                <entry key="../examples/filemgr/filemgr.properties" value="/etc/filemgr.properties"/>
            </map>
        </property>
        <property name="configFiles">
            <map key-type="java.lang.String" value-type="java.lang.String">
                <entry key="../examples/filemgr/mime-types.xml" value="/etc/mime-types.xml"/>
                <entry key="../examples/filemgr/cmd-line-actions.xml" value="/policy/cmd-line-actions.xml"/>
                <entry key="../examples/filemgr/cmd-line-options.xml" value="/policy/cmd-line-options.xml"/>
                <entry key="../examples/filemgr/oodt/elements.xml" value="/policy/oodt/elements.xml"/>
                <entry key="../examples/filemgr/oodt/product-types.xml" value="/policy/oodt/product-types.xml"/>
                <entry key="../examples/filemgr/oodt/product-type-element-map.xml" value="/policy/oodt/product-type-element-map.xml"/>
            </map>
        </property>
    </bean>

</beans>

 

This is a spring configuration file where beans of type DistributedConfigurationPublisher needs to be defined for each OODT component you wish to deploy with the help of distributed configuration management feature. There is only one bean defined in the above example, which is responsible fore storing configuration related to file manager OODT component. Let's look at how this should be configured. 

  1. First define a bean of type org.apache.oodt.config.distributed.cli.DistributedConfigurationPublisher and give it a meaningful id. In the above example, we have given filemgr-config-publisher for the bean id. 

  2. Then set the constructor argument value of that bean to the name of the OODT component you wish to publish configuration for. In the above example, it is filemgr which is the shorthand name used internally for OODT File Manager. List of shorthand names that should be used per component is listed below. 

    Shorthand names for OODT components

    Following shorthand names need to be used when configuring the config-publisher.xml to publish configuration for any OODT component. Using the following naming convention is mandatory because these shorthand names are being used internally for creating ZNodes in zookeeper.

    OODT Component NameShorthand name to be used when configuring config-publisher.xml
    File Managerfilemgr
    Resource Managerresmgr

  3. Now, we have to set the properties files to be stored in zookeeper for our selected OODT component. Since properties files and other configuration files need to be treated in a different manner at run time, properties files are considered separately. Following fragment from our example config-publisher.xml shows how we should define the configuration files to be stored in zookeeper.

    <property name="propertiesFiles">
    <map key-type="java.lang.String" value-type="java.lang.String">
        <entry key="../examples/filemgr/filemgr.properties" value="/etc/filemgr.properties"/>
    </map>
</property>

    Configuration takes a map of key value pairs where the key of each entry stands for the location (in your local machine) of the properties file to be stored in zookeeper. Value of each entry stands for the location where the corresponding properties file should be available once downloaded from zookeeper. For example,  

    <entry key="../examples/filemgr/filemgr.properties" value="/etc/filemgr.properties"/>

      says that the file located at ../examples/filemgr/filemgr.properties (relative to bin directory) needs to be available at ${OODT_COMPONENT_HOME}/etc/filemgr.properties location when being downloaded by the deployed instances. 

  4. Similarly we have to setup the configuration files other than properties files to be stored in zookeeper. Example configuration fragment is shown below. 

    <property name="configFiles">
    <map key-type="java.lang.String" value-type="java.lang.String">
        <entry key="../examples/filemgr/mime-types.xml" value="/etc/mime-types.xml"/>
        <entry key="../examples/filemgr/cmd-line-actions.xml" value="/policy/cmd-line-actions.xml"/>
        <entry key="../examples/filemgr/cmd-line-options.xml" value="/policy/cmd-line-options.xml"/>
        <entry key="../examples/filemgr/oodt/elements.xml" value="/policy/oodt/elements.xml"/>
        <entry key="../examples/filemgr/oodt/product-types.xml" value="/policy/oodt/product-types.xml"/>
        <entry key="../examples/filemgr/oodt/product-type-element-map.xml" value="/policy/oodt/product-type-element-map.xml"/>
    </map>
</property>

    As explained in the previous point, we have to give a mapping of local files to be stored in zookeeper along with the desired location of those files to be available relatively to ${OODT_COMPONENT_HOME} during run time. For example, 

    <entry key="../examples/filemgr/oodt/product-types.xml" value="/policy/oodt/product-types.xml"/>

    says that the file located at ../examples/filemgr/oodt/product-types.xml (relative to bin directory) needs to be available at ${OODT_COMPONENT_HOME}//policy/oodt/product-types.xml location when being downloaded by the deployed instances. 

  5. We have now finished configuring the configuration publisher for file manager OODT component in the above steps. Similarly we can configure configuration publisher beans for other OODT components as per our requirement.

Publishing Configuration - Configuration Publisher CLI

We have now configured our config-publisher.xml with the required corresponding spring beans to match the OODT components we need to deploy with the help of distributed configuration management. Now we can use the Configuration publisher CLI  to actually publish/store the contents in the configuration files in zookeeper for deployed OODT components to download. The configuration publisher executable is located in the bin directory and it is named conf-publisher. If you just run the conf-publisher as below without any arguments, it will print the usage as follows.

./conf-publisher
There's an error in your command
 -clear             : Unpublish any configuration which has been published
                      earlier using the same spring config file (default: false)
 -connectString VAL : Zookeeper connect string
 -publish           : Publishes configuration specified in the spring config
                      file to zookeeper. Any current similar config in
                      zookeeper will be overwritten. If not specified, command
                      will be assumed as a publish (default: false)
 -verify            : Verifies the content in the local files and the published
                      ones. Results will be printed. (default: false)
Finished

As shown above, the CLI provides 3 major functions,

  • publish - Stores configuration specified in the config-publisher.xml in zookeeper
  • verify - Verifies whether the content stored in zookeeper are identical to the content in the configuration files (original files which we just published to zookeeper) stored locally.
  • clear - Removes any configuration data stored in zookeeper. This will delete only the configuration stored in zookeeper using the same config-publisher.xml. Therefore, users do not have to worry about all configuration being removed from zookeeper.

Also note that we need to give the zookeeper connect string with the -connectString option to the CLI so that the CLI will know to which zookeeper ensemble to connect to. Our next step is running the conf-publisher. Executing 

conf-publisher -connectString <zookeeper-connect-string> -publish

in the bin directory will execute the configuration publisher and store all the configuration specified in zookeeper. Following is the possible output you will get.

ubuntu@ubuntu: bin$ ./conf-publisher -connectString 127.0.0.1:2181 -publish


Starting Distributed Configuration Publisher 
log4j:WARN Continuable parsing error 40 and column 23
log4j:WARN The content of element type "log4j:configuration" must match "(renderer*,appender*,(category|logger)*,root?,categoryFactory?)".
Starting configuration publishing
INFO ClassPathXmlApplicationContext - Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@6ed3ef1: display name [org.springframework.context.support.ClassPathXmlApplicationContext@6ed3ef1]; startup date [Sun Jul 09 18:11:50 IST 2017]; root of context hierarchy
INFO XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [etc/config-publisher.xml]
INFO ClassPathXmlApplicationContext - Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@6ed3ef1]: org.springframework.beans.factory.support.DefaultListableBeanFactory@5b464ce8
INFO DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5b464ce8: defining beans [filemgr-config-publisher]; root of factory hierarchy
INFO DistributedConfigurationPublisher - Using zookeeper connect string : 127.0.0.1:2181
INFO DistributedConfigurationPublisher - Curator framework start operation invoked
INFO DistributedConfigurationPublisher - Waiting to connect to zookeeper, startupTimeout : 30000
INFO DistributedConfigurationPublisher - CuratorFramework client started successfully


Processing commands for component : filemgr
Publishing configuration for : filemgr
INFO DistributedConfigurationPublisher - Publishing configuration ../examples/filemgr/filemgr.properties - /etc/filemgr.properties
INFO DistributedConfigurationPublisher - Replaced old published configuration at /etc/filemgr.properties with content of file : ../examples/filemgr/filemgr.properties
INFO DistributedConfigurationPublisher - Properties files published successfully
INFO DistributedConfigurationPublisher - Publishing configuration ../examples/filemgr/mime-types.xml - /etc/mime-types.xml
INFO DistributedConfigurationPublisher - Replaced old published configuration at /etc/mime-types.xml with content of file : ../examples/filemgr/mime-types.xml
INFO DistributedConfigurationPublisher - Publishing configuration ../examples/filemgr/cmd-line-actions.xml - /policy/cmd-line-actions.xml
INFO DistributedConfigurationPublisher - Replaced old published configuration at /policy/cmd-line-actions.xml with content of file : ../examples/filemgr/cmd-line-actions.xml
INFO DistributedConfigurationPublisher - Publishing configuration ../examples/filemgr/cmd-line-options.xml - /policy/cmd-line-options.xml
INFO DistributedConfigurationPublisher - Replaced old published configuration at /policy/cmd-line-options.xml with content of file : ../examples/filemgr/cmd-line-options.xml
INFO DistributedConfigurationPublisher - Publishing configuration ../examples/filemgr/oodt/elements.xml - /policy/oodt/elements.xml
INFO DistributedConfigurationPublisher - Replaced old published configuration at /policy/oodt/elements.xml with content of file : ../examples/filemgr/oodt/elements.xml
INFO DistributedConfigurationPublisher - Publishing configuration ../examples/filemgr/oodt/product-types.xml - /policy/oodt/product-types.xml
INFO DistributedConfigurationPublisher - Replaced old published configuration at /policy/oodt/product-types.xml with content of file : ../examples/filemgr/oodt/product-types.xml
INFO DistributedConfigurationPublisher - Publishing configuration ../examples/filemgr/oodt/product-type-element-map.xml - /policy/oodt/product-type-element-map.xml
INFO DistributedConfigurationPublisher - Replaced old published configuration at /policy/oodt/product-type-element-map.xml with content of file : ../examples/filemgr/oodt/product-type-element-map.xml
INFO DistributedConfigurationPublisher - Config files published successfully
Published configuration for : filemgr
INFO DistributedConfigurationPublisher - Configuration publisher destroyed
INFO DistributedConfigurationPublisher - Exiting CLI ...
Finished

 

Now we have published our configuration to zookeeper successfully.

Using Distributed Configuration Management

Once the configuration is published to zookeeper, we can deploy our OODT components to corresponding servers in the cluster. At this point, if we want to use the distributed configuration management in those instances, we have to follow the following steps.

  1. First, we should enable distributed configuration management by setting the org.apache.oodt.config.distributed system property to true.

  2. Then,  we must either specify the zookeeper connect string as the org.apache.oodt.config.zk.connectString system property or specify the properties file (file location as org.apache.oodt.config.zkProperties) in which we have set the aforementioned property for zookeeper connect string.

  3. When the above two steps are completed, we can run any OODT component with the support of distributed configuration management.

Example with file manager. We need to add the following two properties to filemgr executable.

-Dorg.apache.oodt.config.zk.connectString=<zookeeper-connect-string> \
-Dorg.apache.oodt.config.distributed=true \

Alternatively,

You can set the following two environment variables before executing your component. The bash script will pick those environment variables and set the above mentioned system properties accordingly.

export OODT_DISTRIBUTED_CONF=true
export ZK_CONNECT_STRING=<zookeeper-connect-string>
  • No labels