Short summary:
This document details the steps involved in Installing latest version of Apache Incubator Ranger independently on RHEL / Ubuntu / SUSE / Debian OS.
Ranger Admin support on Apache Components
Component name Version Reference HDFS 2.7.0 https://hadoop.apache.org/releases.html HIVE 1.2.0 https://hive.apache.org/downloads.html HBase 1.1.0.1 http://hbase.apache.org/ Knox 0.6.0 https://knox.apache.org/ Storm 0.10.0beta1 https://storm.apache.org/downloads.html Yarn 2.7.0 http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Zookeeper 3.4.6 https://zookeeper.apache.org/releases.html
Prerequisites
A. JDK 7 or above needs to be installed.
B. Install latest version of Database MySQL/ ORACLE/ Postgres/ SQL Server
Instructions
Preparing to install
Install Maven
cd /usr/local # Download maven latest distribution tar from apache maven site tar -xvf apache-maven-<Version>-bin.tar.gz export M2_HOME=/usr/local/apache-maven-<Version> export M2=$M2_HOME/bin export PATH=$M2:$PATH #Now to test your install of Maven, enter the following command mvn -version
Install git
yum install git
Install gcc and export JAVA_HOME
yum search gcc # Select the appropriate gcc and do 'yum install gcc' export JAVA_HOME=<Java Installation Directory>
Build Ranger Admin from source
Clone the ranger source code
mkdir ~/dev cd ~/dev git clone https://github.com/apache/incubator-ranger.git
Build the source
cd incubator-ranger export MAVEN_OPTS="-Xmx512M" mvn clean compile package assembly:assembly install
Verify all the tar files under target dir
Install Steps for Ranger Policy Admin on RHEL
a. Lay down the build into appropriate places. Let’s start with the Ranger web admin first.
cd /usr/local sudo tar zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-admin.tar.gz sudo ln -s ranger-0.5.0-admin ranger-admin cd /usr/local/ranger-admin
b. Verify the root password that you had picked while installing mysql. I had chosen root so the relevant section in my install.properties file looks as follows:
db_root_user=root db_root_password=root db_host=localhost
c.The install process would create a couple of users in the database for storing administration and audit information, pick passwords for those too. With my choices here’s how the relevant sections in the install.properties file look now.
# DB UserId used for the XASecure schema db_name=ranger db_user=rangeradmin db_password=rangeradmin # DB UserId for storing auditlog infromation audit_db_name=ranger audit_db_user=rangerlogger audit_db_password=rangerlogger
d. Ranger allows you to get fancier with security and authentication mode but for now let’s just leave rest of the things in this file as they are.
e. Once all the required properties are updated run the setup.sh script
./setup.sh
f. This will install the Ranger service.
g. After this Start Ranger Service by typing.
./ews/ranger-admin-services.sh start
h. After this Restart the Ranger Admin should work.
You can stop the Ranger service by typing : ./ews/ranger-admin-services.sh stop
To stop and start you can try this ./ews/ranger-admin-services.sh restart
i.You can verify by visiting the external URL of the server using browser, for example:
http://<Host Address>:6080/
- Configuring Ranger Admin Authentication Modes :
- ACTIVE DIRECTORY
To enable active directory authentication on Ranger admin, you need to configure following properties of install.properties
PROPERY | VALUE |
---|---|
authentication_method | ACTIVE_DIRECTORY |
xa_ldap_ad_domain | A sample value would be : “example.com” |
xa_ldap_ad_url | A sample value would be “ldap://127.0.0.1:389” |
xa_ldap_ad_base_dn | A sample value would be : “DC=example,DC=com” |
xa_ldap_ad_bind_dn | A sample value would be : |
xa_ldap_ad_bind_password | Password for the account that can search for users. |
xa_ldap_ad_referral | Possible values are “ignore ”, “follow ” and “throw ”. default value is “follow”. |
When searching a directory, the server might return several search results, in addition to
a few continuation references that show where to obtain further results. These results
and references might be interleaved at the protocol level. When property is set to
"follow", the AD service provider processes all the normal entries first, before following
the continuation references. When this property is set to "throw", all of normal entries are
returned in the enumeration first, before the ReferralException is thrown. By contrast, a
"referral" error response is processed immediately when property is set to "follow" or
"throw".
- LDAP
PROPERTY | VALUE |
---|---|
authentication_method | LDAP |
xa_ldap_url | A sample value would be : "ldap://127.0.0.1:389" Ldap server URL. |
xa_ldap_userDNpattern | A sample value would be : "uid={0},ou=users,dc=example,dc=com" User DN pattern is expanded when a user is being logged in. For example, if the user ‘ldapadmin’ attempted to log in, the LDAP Server would attempt to bind against the DN ‘uid=ldapadmin,ou=users,dc=example,dc=com’ using the password the user provided. |
xa_ldap_groupSearchBase | A sample value would be : "dc=example,dc=com" |
xa_ldap_groupSearchFilter | A sample value would be: "(member=cn={0},ou=users,dc=example,dc=com)" The filter which is used to search for group membership. The default is uniqueMember={0}, corresponding to the groupOfUniqueNames LDAP class. In case of Ranger authentication, the substituted parameter is the full distinguished name of the user. The parameter {0} can be used if you want to filter on the login name. |
xa_ldap_groupRoleAttribute | A sample value would be: "cn" The attribute which contains the name of the authority defined by the group entry. |
xa_ldap_base_dn | A sample value would be : " dc=example,dc=com" The Distinguished Name (DN) of the starting point for directory server searches. |
xa_ldap_bind_dn | A sample value would be : "cn=ldapadmin,ou=users,dc=example,dc=com" Full distinguished name (DN), including common name (CN), of an LDAP user account that has privileges to search for users. |
xa_ldap_bind_password | Password for the account that can search for users. |
xa_ldap_referral | default value is “follow” Possible values are “ignore ”, “follow ” and “throw ”. default value is “follow”. When searching a directory, the server might return several search results, in addition to a few continuation references that show where to obtain further results. These results and references might be interleaved at the protocol level. When property is setto "follow", the AD service provider processes all the normal entries first, before following the continuation references. When this property is set to "throw", all of normal entries are returned in the enumeration first, before the ReferralException is thrown. By contrast, a "referral" error response is processed immediately when property is set to "follow" or |
- UNIX
PROPERTY | VALUE |
---|---|
authentication_method | UNIX |
remoteLoginEnabled | true |
authServiceHostName | A sample value would be : localhost An Address of host where unixauth service is running |
authentication_method | 5151 |
port number on which unixauth service is running. default is 5151 |
- Configuring Ranger Admin HA Mode
a. Follow the ranger admin install steps above to install it on multiple hosts
b. Make sure to use the same configuration and policy DB details
c. Configure a load balancer to load balance among ranger admin instances and note down the load balancer URL.
- Software (e.g. Apache httpd) or hardware load balancer could be used.
- Details outside the scope of this document.
d. Update the policy manager external URL in all the clients of ranger admin (ranger user sync and ranger plugins) to point to the load balancer URL.
e. Restart all the clients.
Installing the Ranger User-Sync Process
- We’ll start by extracting out build at the appropriate place.
cd /usr/local sudo tar zxvf ~/dev/ incubator-ranger/target/ranger-0.5.0-usersync.tar.gz sudo ln -s ranger-0.5.0-usersync ranger-usersync sudo mkdir -p /var/log/ranger-usersync sudo chown ranger /var/log/ranger-usersync sudo chgrp ranger /var/log/ranger-usersync cd rangerusersyncb.
Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- POLICY_MGR_URL = http://localhost:6080
- SYNC_SOURCE = unix
- logdir = /var/log/ranger/usersync
- Now install the usersync by running the setup command
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64 ./setup.sh After installing rangerusersync, follow the same steps to start/stop services of usersync work. ./ranger-usersync-services.sh start
Configuring Ranger User-Sync process to use LDAP/AD server:
To Sync LDAP or AD users following properties should be configured in install.properties file before executing the setup.sh.
SYNC_SOURCE | sync source, “ldap” should be used for both LDAP or AD user |
---|---|
SYNC_LDAP_URL |
|
SYNC_LDAP_BIND_DN | Ldap bind dn used to connect to ldap and query for users and groups. Must specify a value if SYNC_SOURCE is ldap. |
SYNC_LDAP_BIND_PASSWORD | Ldap bind password for the bind dn specified above. Please ensure read access to this file is limited to root, to protect the password |
SYNC_LDAP_SEARCH_BASE | search base for users and groups |
SYNC_LDAP_USER_SEARCH_BASE | search base for users,overrides value specified in SYNC_LDAP_SEARCH_BASE. Must specify a value if SYNC_SOURCE is ldap and |
SYNC_LDAP_USER_SEARCH_SCOPE | search scope for the users, only base, one and sub are supported values. |
NEED TO BE CUSTOMIZE | PROPERTY | DESCRIPTION |
---|---|---|
#Please customize the value to suit your deployment. | SYNC_LDAP_USER_OBJECT_CLASS | object class to identify user entries. |
#Please customize the value to suit your deployment. | SYNC_LDAP_USER_SEARCH_FILTER | optional additional filter constraining the users selected for syncing. |
# please customize the value to suit your deployment. | SYNC_LDAP_USER_NAME_ATTRIBUTE | attribute from user entry that would be treated as user name |
# please customize the value to suit your deployment. | SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE | attribute from user entry whose values would be treated as group values to be pushed into Policy Manager database. You could provide multiple attribute names separated by comma. |
# possible values: none, lower, upper | SYNC_LDAP_USERNAME_CASE_CONVERSION | UserSync Case Conversion Flags. |
# possible values: none, lower, upper | SYNC_LDAP_GROUPNAME_CASE_CONVERSION | UserSync Case Conversion |
# any value other than true would be treated as false | SYNC_GROUP_SEARCH_ENABLED | do we want to do ldapsearch to find groups |
SYNC_GROUP_USER_MAP_SYNC_ENABLED | do we want to do ldapsearch to find groups instead of relying on user entry attributes and sync memberships of those groups valid values: true, false. any value other than true would be treated as false. | |
SYNC_GROUP_SEARCH_BASE | search base for groups. overrides value specified in SYNC_LDAP_SEARCH_BASE,SYNC_LDAP_USER_SEARCH_BASE. | |
# any value other than true would be treated as false | SYNC_GROUP_SEARCH_SCOPE | search scope for the groups, only base, one and sub are supported values |
# any value other than true would be treated as false | SYNC_GROUP_OBJECT_CLASS | object class to identify group entries. default value: groupofnames |
please customize the value to suit your deployment. | SYNC_LDAP_GROUP_SEARCH_FILTER | optional additional filter constraining the groups selected for syncing. default value is empty. |
# please customize the value to suit your deployment. | SYNC_GROUP_NAME_ATTRIBUTE | attribute from group entry that would be treated as group name. |
SYNC_GROUP_MEMBER_ATTRIBUTE_NAME | attribute from group entry that is list of members. default value: member. | |
SYNC_PAGED_RESULTS_SIZE | page size for paged results control.search results would be returned page by page with the specified number of entries per page default value: 500 | |
# please customize the value to suit your deployment. | SYNC_PAGED_RESULTS_ENABLED | do we want to use paged results control during |
Table: LDAP/AD Properties with sample values
PROPERTIES NAME | SAMPLE VALUES FOR LDAP USER SYNCH | SAMPLE VALUES FOR AD USERSYNCH |
---|---|---|
SYNC_LDAP_URL | ldap://127.0.0.1:389 | ldap://127.0.0.1:389 |
SYNC_LDAP_BIND_DN | cn=ldapadmin,ou=users,dc=example,dc=com | cn=adadmin,cn=Users,dc=exa |
SYNC_LDAP_BIND_PASSWORD | secret | secret |
SYNC_LDAP_SEARCH_BASE | dc=example,dc=com | dc=example,dc=com |
SYNC_LDAP_USER_SEARCH_BASE | ou=users,dc=example,dc=com | dc=example,dc=com |
SYNC_LDAP_USER_SEARCH_SCOPE | sub | sub |
SYNC_LDAP_USER_OBJECT_CLASS | person | person |
SYNC_LDAP_USER_SEARCH_FILTER | (objectcategory=person) | |
SYNC_LDAP_USER_NAME_ATTRIBUTE | uid or cn | sAMAccountName |
SYNC_LDAP_USER_GROUP_NAME_AT | memberof,ismemberof | memberof,ismemberof |
SYNC_LDAP_USERNAME_CASE_CONV | lower | lower |
SYNC_LDAP_GROUPNAME_CASE_CON | lower | lower |
SYNC_GROUP_SEARCH_ENABLED | false | false |
SYNC_GROUP_USER_MAP_SYNC_ENA | false | false |
SYNC_GROUP_SEARCH_BASE | ou=groups,dc=example,dc=com | dc=example,dc=com |
SYNC_GROUP_SEARCH_SCOPE | sub | sub |
SYNC_GROUP_OBJECT_CLASS | groupofnames | groupofnames |
SYNC_LDAP_GROUP_SEARCH_FILTER | ||
SYNC_GROUP_NAME_ATTRIBUTE | cn | cn |
SYNC_GROUP_MEMBER_ATTRIBUTE_ | member | member |
SYNC_PAGED_RESULTS_ENABLED | true | true |
SYNC_PAGED_RESULTS_SIZE | 500 | 500 |
Installing Apache Hadoop
Now let’s download and install hadoop. Following the excellent instructions available on the hadoop site itself. Follow steps given in pseudo distributed mode.
These instructions were written for version 2.7.0. So grab that tar (hadoop-2.7.0.tar.gz) and checksum file (hadoop-2.7.0.tar.gz.mds).
- Instructions on this page ask that java be installed. If java is not there, install JDK first.
sudo yum install java-1.7.0-openjdk-devel
- Make note of the location where you installed hadoop. Here I assume that you have installed it in
/usr/local/hadoop
- Create a user under which we could install and ultimately run the various hadoop processes. And login as that user.
sudo useradd --home-dir /var/hadoop --create-home --shell /bin/bash --user-group hadoop
- if you get below given message then try next command
sudo useradd --home-dir /var/hadoop --create-home --shell /bin/bash hadoop -g hadoop
sudo tar zxvf ~/dev/hadoop-2.7.0.tar.gz -C /usr/local cd /usr/local sudo ln -s hadoop-2.7.0 hadoop sudo chown hadoop -R hadoop hadoop-2.7.0 sudo chgrp hadoop -R hadoop hadoop-2.7.0
TO ADD HDFS USER
useradd hdfs to check whether user hadoop login works, try: -sudo su -hadoop
Enabling Ranger HDFS Plugins
a. We’ll start by extracting our build at the appropriate place (/usr/local).
cd /usr/local sudo tar zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz sudo ln -s ranger-0.5.0-hdfs-plugin ranger-hdfs-plugin cd ranger-hdfs-plugin
b. Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- Change the install.properties file
PROPERTY | VALUE |
---|---|
POLICY_MGR_URL | |
REPOSITORY_NAME | hadoopdev |
XAAUDIT.DB.IS_ENABLED | true |
XAAUDIT.DB.FLAVOUR | MYSQL |
XAAUDIT.DB.HOSTNAME | localhost |
XAAUDIT.DB.DATABASE_NAME | ranger_audit |
XAAUDIT.DB.USER_NAME | rangerlogger |
XAAUDIT.DB.PASSWORD | rangerlogger |
c. Now enable the hdfs-plugin by running the enable-hdfs-plugin.sh command (Remember to set JAVA_HOME)
Note: Hadoop conf and hadoop lib folder are not found at expected locations as per the script because of which,
Ranger hdfs plugin installation fails.
To resolve this issue create a symlink as conf dir of hadoop linking to hadoop conf dir
cd /usr/local/hadoop ln -s /usr/local/hadoop/etc/hadoop/ /usr/local/hadoop/conf
- Export HADOOP_HOME to bashrc
echo "export HADOOP_HOME=/usr/local/hadoop" >> /etc/bashrc cd /usr/local/ranger-hdfs-plugin ./ enable-hdfs-plugin.sh
- One more change that we need to do is copy all the jar files from ${hadoop_home}/lib
cp /usr/local/hadoop/lib/*.jar /usr/local/hadoop/share/hadoop/hdfs/lib/
- Provide required permission to logs directory
chown root:hadoop /usr/local/hadoop/logs chmod g+w /usr/local/hadoop/logs
- Provide required permission to users in OS file system and hdfs file system according to your environment and requirement.
d. Once these changes are done Restart hadoop.
- Stop NameNode, SecondaryNameNode and DataNode daemon:
su -l hdfs -c "/usr/local/hadoop/sbin/hadoop-daemon.sh stop namenode" su -l hdfs -c "/usr/local/hadoop/sbin/hadoop-daemon.sh stop secondarynamenode" su -l hdfs -c "/usr/local/hadoop/sbin/hadoop-daemon.sh stop datanode"
- Start NameNode, SecondaryNameNode and DataNode daemon:
su -l hdfs -c "/usr/local/hadoop/sbin/hadoop-daemon.sh start namenode" su -l hdfs -c "/usr/local/hadoop/sbin/hadoop-daemon.sh start secondarynamenode" su -l hdfs -c "/usr/local/hadoop/sbin/hadoop-daemon.sh start secondarynamenode"
e. This should start the association of rangerhdfsplugin with hadoop.
- You can verify by logging into the Ranger Admin Web interface > Audit > Agents.
Installing Apache Hive(1.2.0)
- Let’s download and install apache hive . Following the excellent instructions available on the apache hive site itself
sudo tar xzvf ~/dev/apache-hive-1.2.0-bin.tar.gz -C /usr/local cd /usr/local sudo ln -s apache-hive-1.2.0-bin hive useradd hive cd hive Export HIVE_HOME to bashrc echo "export HIVE_HOME=/usr/local/hive" >> /etc/bashrc
Note:HiveServer2 doesn’t start unless HADOOP_VERSION is exported to bashrc
Enabling Ranger Hive Plugin
- We’ll start by extracting our build at the appropriate place.
cd /usr/local sudo tar zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz sudo ln -s ranger-0.5.0-hive-plugin ranger-hive-plugin cd ranger-hive-plugin
- Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- Change the insall.properties file
PROPERTY | VALUE |
---|---|
POLICY_MGR_URL | |
REPOSITORY_NAME | hivedev |
XAAUDIT.DB.IS_ENABLED | true |
XAAUDIT.DB.FLAVOUR=MYSQL | MYSQL |
XAAUDIT.DB.HOSTNAME | localhost |
XAAUDIT.DB.DATABASE_NAME | ranger_audit |
XAAUDIT.DB.USER_NAME | rangerlogger |
XAAUDIT.DB.PASSWORD | rangerlogger |
- Now enable the hive-plugin by running the enable-hive-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-hive-plugin ./enable-hive-plugin.sh
- Once these changes are done Restart hive. This should start the association of ranger-hive-plugin with hive.
- You can verify by logging into the Ranger Admin Web interface > Audit Tab > Agents
- Provide required permission to users in OS file system and hdfs file system according to your environment and requirement..
NOTES: If /var/log/hive directory does not exist then create one and assign to user hive.
mkdir /var/log/hive chown -R hive: hive /var/log/hive
- Change properties file permission for hive user.
chown -R hive:hadoop /usr/local/apache-hive-1.2.0-bin/conf/hiveserver2-site.xml chown -R hive:hadoop /usr/local/apache-hive-1.2.0-bin/conf/hive-log4j.properties chown -R hive:hadoop /usr/local/apache-hive-1.2.0-bin/conf/hive-site.xml
- To start hive metastore :
su -l hive -c "env HADOOP_HOME=/usr/local/hadoop JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64 nohup hive --service metastore > /var/log/hive/hive.out 2> /var/log/hive/hive.log &”
- To start Hive server2 :
su -l hive -c "env HADOOP_HOME=/usr/local/hadoop JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64 nohup /usr/local/hive/bin/hiveserver2 hiveconf hive.metastore.uris=\" \" > /var/log/hive/hiveServer2.out 2>/var/log/hive/hiveServer2.log &”
- To Stop:
ps aux | awk '{print $1,$2}' | grep hive | awk '{print $2}' | xargs kill >/dev/null 2>&1
- To Login in Hive shell:
/usr/local/hive/bin/beeline -u "jdbc:hive2://localhost:10000" -n rituser -p rituser
- If hive metastore and hiveserver2 do not start then update below given key-values according to your environment in following files.
hiveserver2site.xml
<configuration> </property> |
---|
hivesite.xml
<property> |
---|
Installing Apache Hbase (1.1.0.1)
- Let’s download and install apache Hbase . Following the excellent instructions available on he apache Hbase site itself.
sudo tar xzvf ~/dev/hbase-1.1.0.1-bin.tar.gz -C /usr/local cd /usr/local sudo ln -s hbase-1.1.0.1 hbase useradd hbase cd hbase Export HBASE_HOME to bashrc echo "export HBASE_HOME=/usr/local/hbase" >> /etc/bashrc
- For HBase 0.98.5 and later, you are required to set the JAVA_HOME environment variable before starting HBase
Enabling Ranger Hbase Plugins
- We’ll start by extracting our build at the appropriate place.
cd /usr/local sudo tar zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-hbase-plugin.tar.gz sudo ln -s ranger-0.5.0-hbase-plugin ranger-hbase-plugin cd ranger-hbase-plugin
- Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- Change the insall.properties file
PROPERTY | VALUE |
---|---|
POLICY_MGR_URL | |
REPOSITORY_NAME | hbasedev |
XAAUDIT.DB.IS_ENABLED | true |
XAAUDIT.DB.FLAVOUR | MYSQL |
XAAUDIT.DB.HOSTNAME | localhost |
XAAUDIT.DB.DATABASE_NAME | ranger_audit |
XAAUDIT.DB.USER_NAME | rangerlogger |
XAAUDIT.DB.PASSWORD | rangerlogger |
- Now enable the hbase-plugin by running the enable-hbase-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-hbase-plugin ./enable-hbase-plugin.sh
- Once these changes are done Restart hbase. This should start the association of ranger-hbase-plugin with hbase.
- You can verify by logging into the Ranger Admin Web interface > Audit Tab > Agents
- To Stop master and regionserver try:
/usr/local/hbase/bin/hbase-daemon.sh stop master /usr/local/hbase/bin/hbase-daemon.sh stop regionserver
- Provide required permission to users in OS file system and hdfs file system according to your environment and requirement.
Installing Apache Knox Gateway
- Let’s download and install apache Knox from Apache Mirrors.
sudo tar -zxvf ~/dev/knox-0.6.0.tar.gz -C /usr/local cd /usr/local sudo ln -s knox-0.6.0 knox cd knox
- Following the instructions available on the apache knox site itself (To install Apache Knox Gateway).
Knox Master Secret : knox
Enabling Ranger Knox Plugins
- We’ll start by extracting our build at the appropriate place.
cd /usr/local tar -zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-knox-plugin.tar.gz sudo ln -s ranger-0.5.0-knox-plugin ranger-knox-plugin cd ranger-knox-plugin
- Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- Change the insall.properties file
PROPERY | VALUE |
---|---|
POLICY_MGR_URL | |
REPOSITORY_NAME | knoxdev |
KNOX_HOME | /usr/local/knox |
XAAUDIT.DB.IS_ENABLED | true |
XAAUDIT.DB.HOSTNAME | localhost |
XAAUDIT.DB.DATABASE_NAME | ranger |
XAAUDIT.DB.USER_NAME | rangerlogger |
XAAUDIT.DB.PASSWORD | rangerlogger |
- Now enable the knox-plugin by running the enable-knox-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-knox-plugin ./enable-knox-plugin.sh
- Once these changes are done Restart Knox ( Gateway / LDAP )
- if you get permission denied error during knox start please provide required privileges to knox user. for example :
chown -R knox:knox /usr/local/knox/data chown -R knox:knox /usr/local/knox/logs chown -R knox:knox /usr/local/knox/pids chown -R knox:hadoop /usr/local/knox/pids/*
- You can verify by logging into the Ranger Admin Web interface > Audit > Agents
Trusting Self Signed Knox Certificate
When Knox is listening on its SSL port with self signed certificate, you have to import SSL certificate of Knox into truststore used by XA PolicyManager. Here are steps for importing Knox SSL certificate in truststore used by XA PolicyManager.
- Log in the machine running Knox
- Export knox certificate
- cd $GATEWAY_HOME/data/security/keystores
- This is typically /usr/local/knox/data/security/keystores on Linux machine.
- keytool -exportcert -alias gateway-identity -keystore gateway.jks -file knox.crt
- Copy knox.crt file onto machine running Ranger Admin/PolicyManager to a working directory, for example /usr/local/ranger-admin
- Replicate cacerts
cd /usr/local/ranger-admin cp $JAVA_HOME/jre/lib/security/cacerts cacertswithknox
- Import Knox certificate into the replicated new keystore
keytool -import -trustcacerts -file <knox.crt created above> -alias knox -keystore cacertswithknox password: changeit
- Edit /usr/local/ranger-admin/ews/ranger-admin-services.sh
- Add parameter -Djavax.net.ssl.trustStore=<path to the cacertswithknox> to the java call in the script.
- Restart Ranger Admin/PolicyManager.
Installing Apache Storm (0.10.0)
- Let’s download and install apache Storm from Apache Mirrors
sudo tar -zxvf ~/dev/apache-storm-0.10.0-beta1.tar.gz -C /usr/local cd /usr/local sudo ln -s apache-storm-0.10.0beta1 storm cd storm
- Following the instructions available on the apache storm site itself(To install Apache Storm).
Enabling Ranger Storm Plugins
- We’ll start by extracting our build at the appropriate place.
cd /usr/local tar -zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-storm-plugin.tar.gz sudo ln -s ranger-0.5.0-storm-plugin ranger-storm-plugin cd ranger-storm-plugin
- Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- Change the insall.properties file
PROPERTY | VALUE |
---|---|
POLICY_MGR_URL | |
REPOSITORY_NAME | stormdev |
XAAUDIT.DB.IS_ENABLED | true |
XAAUDIT.DB.HOSTNAME | localhost |
XAAUDIT.DB.DATABASE_NAME | ranger |
XAAUDIT.DB.USER_NAME | rangerlogger |
XAAUDIT.DB.PASSWORD | XAAUDIT.DB.PASSWORD=rangerlogger |
- Now enable the storm-plugin by running the enable-storm-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-storm-plugin ./enable-storm-plugin.sh
- Once these changes are done Restart Storm
- You can verify by logging into the Ranger Admin Web interface > Audit > Agents
Installing Apache Yarn
You can run a MapReduce job on YARN in a pseudodistributed mode by setting a few parameters and running ResourceManager daemon and NodeManager daemon in addition
The following instructions assume that hadoop installations steps mentioned in Installing Apache Hadoop are already executed.
Enabling Ranger Yarn Plugin
- We’ll start by extracting our build at the appropriate place (/usr/local).
cd /usr/local sudo tar zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-yarn-plugin.tar.gz sudo ln -s ranger-0.5.0-yarn-plugin ranger-yarn-plugin cd ranger-yarn-plugin
- Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- Change the install.properties file
PROPERTY | VALUE |
---|---|
POLICY_MGR_URL | |
REPOSITORY_NAME | yarndev |
XAAUDIT.DB.IS_ENABLED | true |
XAAUDIT.DB.FLAVOUR | MYSQL |
XAAUDIT.DB.HOSTNAME | localhost |
XAAUDIT.DB.DATABASE_NAME | ranger_audit |
XAAUDIT.DB.USER_NAME | rangerlogger |
XAAUDIT.DB.PASSWORD | rangerlogger |
- Now enable the yarn-plugin by running the enable-yarn-plugin.sh command.
cd /usr/local/ranger-yarn-plugin ./ enable-yarn-plugin.sh
- One more change that we need to do is copy all the jar files from ${hadoop_home}/lib
cp /usr/local/ranger-yarn-plugin/lib/*.jar /usr/local/hadoop/share/hadoop/yarn/lib/
- if you get permission denied error during yarn start please provide required privileges to yarn user in local and hdfs file system. for example :
mkdir /var/log/yarn chown -R yarn:yarn /var/log/yarn
- Once these changes are done Start ResourceManager daemon and NodeManager daemon.
- Start the ResourceManager on ResourceManager hosts.
su yarn -c "/usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager" ps -ef | grep -i resourcemanager
- Start the NodeManager on NodeManager hosts.
su yarn -c "/usr/local/hadoop/sbin/yarn-daemon.sh start nodemanager" ps -ef | grep -i nodemanager
- Stop the ResourceManager on ResourceManager hosts.
su yarn -c "/usr/local/hadoop/sbin/yarn-daemon.sh stop resourcemanager" ps -ef | grep -i resourcemanager
- Stop the NodeManager on NodeManager hosts.
su yarn -c "/usr/local/hadoop/sbin/yarn-daemon.sh stop nodemanager" ps -ef | grep -i nodemanager
- This should start the association of ranger-yarn-plugin with hadoop.
- You can verify by logging into the Ranger Admin Web interface > Audit > Agents
Installing Ranger KMS (0.5.0)
Prerequisites: (Need to done for all host on which Ranger KMS needs to be installed)
- Download “Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files” zip using below link depending upon the Java version used
- unzip the above downloaded zip file to java’s security folder (Depending upon the java version used)
unzip UnlimitedJCEPolicyJDK7.zip into $JDK_HOME/jre/lib/security unzip jce_policy-8.zip into $JDK_HOME/jre/lib/security
STEPS FOR RANGER KMS:
- We’ll start by extracting our build at the appropriate place(/usr/local).
cd /usr/local sudo tar -zxvf ~/dev/incubator-ranger/target/ranger-0.5.0-kms.tar.gz sudo ln -s ranger-0.5.0-kms ranger-kms cd ranger-kms
- Please note that Ranger KMS plugin is integrated with Ranger KMS and will be installed automatically when KMS is installed.
- Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
- Change the install.properties file
- DB_FLAVOR
- SQL_CONNECTOR_JAR
- db_root_user
- db_root_password
- db_host
- db_name
- db_user
- db_password
PROPERTY | VALUE |
---|---|
POLICY_MGR_URL | |
REPOSITORY_NAME | kmsdev |
KMS_MASTER_KEY_PASSWD | enter master key password |
XAAUDIT.DB.IS_ENABLED | true |
XAAUDIT.DB.FLAVOUR | MYSQL |
XAAUDIT.DB.HOSTNAME | localhost |
XAAUDIT.DB.DATABASE_NAME | ranger_audit |
XAAUDIT.DB.USER_NAME | rangerlogger |
XAAUDIT.DB.PASSWORD | rangerlogger |
- Edit “hdfs-site.xml”( Need to give provider else it will not support hadoop commands)Replace localhost with <internal host name>
Go to path cd /usr/local/hadoop/conf/
vim hdfssite.xml
- For property “dfs.encryption.key.provider.uri” ,enter the value “kms://http@<internal host name>:9292/kms”
save and quit
- Edit “coresite.xml”( Need to give provider else it will not support hadoop commands)
- Replace localhost with <internal host name>
- Go to path cd /usr/local/hadoop/conf/
- vim coresite.xml
- For property “hadoop.security.key.provider.path” ,enter the value “kms://http@<internal host name>:9292/kms”
- Once these changes are done Restart hadoop.
- Stop NameNode, SecondaryNameNode and DataNode daemon:
su -l hdfs -c "/usr/local/hadoop/sbin/hadoopdaemon.sh stopnamenode" su -l hdfs -c "/usr/local/hadoop/sbin/hadoopdaemon.sh startnamenode"
- Run setup
./setup.sh
- Start the kms server
ranger-kms start
- You can verify the plugin is communicating to Ranger admin in Audit->plugins tab.
- If kmsdev service is not created in Ranger Admin then kms-plugin will not able to connect to Ranger admin.
- To Create the Kms service
PROPERTY | VALUE |
---|---|
REPOSITORY_NAME | name specified in installed.properties (e.g |
KMS URL | kms://http@<internal host name>:9292/kms |
Username | <username> (for e.g keyadmin) |
Password | <password> |
Check Test Connection |
ENABLING AUDIT LOGGING TO HDFS:
- To enable Audit to HDFS for a plugin do the below:
- set XAAUDIT.HDFS.ENABLE = true for respective component plugin in the install.properties file which may be found in /usr/local/ranger<component>plugin/ directory.
- configure NameNode host in the XAAUDIT.HDFS.HDFS_DIR.
- create a policy in HDFS service from Ranger Admin for individual component users (hive/hbase/knox/storm/yarn/kafka/kms) to give READ+ WRITE permission for the particular audit folder. i.e for enabling Hive component to log Audits to HDFS , we need to create a policy for hiveuser with READ+ WRITE permissions to respective audit directory
- Audit to HDFS caches logs in local directory, which can be specified in XAAUDIT.HDFS.LOCAL_BUFFER_DIRECTORY ( this can be like ‘/var/log/<component>/**), which is the path where audit is stored temporarily, likewise for archived logs we need to update XAAUDIT.HDFS.LOCAL_ARCHIVE_DIRECTORY value ( this can be like ‘/var/log/<component>/**), before enabling the plugin for the component.
Note that, HDFS audit logging is for archive purposes. For seeing audit report in the Ranger Admin UI, recommended option is Solr.
ENABLING AUDIT LOGGING TO SOLR:
- To enable Audit report from Solr, do the below change in Ranger admin:
- Set following properties in install.properties of ranger service to work audit to solr in Ranger
PROPERTIES | VALUE |
---|---|
audit_store | solr |
audit_solr_urls | |
audit_solr_user | ranger_solr |
audit_solr_password | NONE |
Restart Ranger.
2. To enable Audit to Solr for a plugin do the below:
- Set following properties in install.properties of plugin to start logging audit to Solr : for eg Hbase
PROPERTY | VALUE |
---|---|
XAAUDIT.SOLR.IS_ENABLED | true |
XAAUDIT.SOLR.ENABLE | true |
XAAUDIT.SOLR.URL | |
XAAUDIT.SOLR.USER | ranger_solr |
XAAUDIT.SOLR.PASSWORD | NONE |
XAAUDIT.SOLR.FILE_SPOOL_DIR | var/log/hadoop/hdfs/audit/solr/spool |
- Enable ranger plugin for Hbase.
- Restart Hbase component.
Configuring Kerberos Authentication
Installing and Configuring the KDC
To use Kerberos with Hadoop you can either use an existing KDC or install a new one just for Hadoop's use. The following gives a very high level description of the installation process.
- To install a new version of the server.
yum install krb5-server krb5-libs krb5-auth-dialog krb5-workstation
- Increase the entropy of the linux kernel. This can be done by installing rng-tools on CentOS:
yum install rng-tools -y
- Edit /etc/sysconfig/rngd to add EXTRAOPTIONS="-r /dev/urandom"