Note | ||
---|---|---|
| ||
Please refer to Apache Ranger 0.5.0 Installation |
Table of Contents
Note | ||
---|---|---|
| ||
Please refer to Apache Ranger 0.5.0 Installation |
Summary
...
This document details the steps involved in installing and configuring Apache Ranger.
These instructions are for installing Ranger on CentOS/RHEL (release 6).
Prerequisites
...
JDK 7+ needs to be installed. Both OpenJDK and Oracle JDK are supported.
- MySQL (5.6+) or ORACLE DB (11g+) for Policy/Audit DB. <TBD: Link on database settings for Ranger>
- DB server can be installed on the same host. Or Ranger services need to have access to DB server host
- For production, ensure appropriate capacity planning is done for the database size
- MavenMaven. If not installed, please follow below steps
wget http://mirrors.gigenet.com/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz (Use different mirror if needed)
su -c "tar -zxvf apache-maven-3.0.5-bin.tar.gz -C /opt/"
su -c "vi /etc/profile.d/maven.sh" # Add the following lines to maven.sh
export M2_HOME=/opt/apache-maven-3.0.5
export M2=$M2_HOME/bin
export PATH=$M2:$PATH
Now test your install of Maven. Logout of the system and then log back into it. Enter the following command:
- mvn -version
- Ranger Admin process requires approximately 1.5GB of RAM
Building Ranger from source
...
Get the ranger source
mkdir ~/dev
cd ~/dev
git clone -b master git@githubranger-0.4 git@github.com:apache/incubator-ranger.git ranger
Now build the source
cd ranger
export MAVEN_OPTS="-Xmx512M"
yum search gcc and Install gcc (yum install gcc.x86_64)
mvn clean compile package assembly:assembly
Verify all the tar files under target dir
ls -ltr *tar.gz
-rw-r--r-- 1 root root 15068844 Dec 1 04:30 ranger-0.4.0-hdfs-plugin.tar.gz
-rw-r--r-- 1 root root 14480716 Dec 1 04:30 ranger-0.4.0-hive-plugin.tar.gz
-rw-r--r-- 1 root root 14349626 Dec 1 04:30 ranger-0.4.0-hbase-plugin.tar.gz
-rw-r--r-- 1 root root 17763192 Dec 1 04:30 ranger-0.4.0-knox-plugin.tar.gz
-rw-r--r-- 1 root root 21243470 Dec 1 04:31 ranger-0.4.0-storm-plugin.tar.gz
-rw-r--r-- 1 root root 126143540 Dec 1 04:31 ranger-0.4.0-admin.tar.gz
-rw-r--r-- 1 root root 7677999 Dec 1 04:31 ranger-0.4.0-usersync.tar.gz
Install/Configure Ranger Admin
...
What is done here?
Ranger Admin bits are laid out
Database and database users for Ranger Admin policy store and Ranger Audit database are created.
Ranger Admin properties are configured
Ranger Admin startup service files are created and updated
Steps
Lay down the binaries into appropriate places.
cd /usr/local
sudo tar zxf
...
Lay down the binaries into appropriate places.
cd /usr/local
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-admin.tar.gz
sudo ln -s ranger-0.4.0-admin ranger-admin
- Open install.properties in ranger root folder
cd ranger-admin and open install.properties using your text editor (e.g. vi install.properties)
Verify the root
Verify the rootpassword that you had picked while installing mysql. I had chosen root so the relevant section in my install.properties file looks as follows
- db_root_user=root
- db_root_password=root
db_host=localhost
The install process would create a couple of users in the database for storing administration and audit information, pick passwords for those too. With my choices here’s how the relevant sections in the install.properties file look now.
Ranger allows you to get different authentication modes but for now let’s just leave rest of the things in install.properties file as they are.
Once all the required properties are updated, execute the below scripts to install ranger admin service.
Execute : ./setup.sh
Execute : ./set_globals.sh- # DB information Ranger Policy Store schema
# - db_name=ranger
- db_user=rangeradmin
- db_password=rangeradmin
- # DB UserId for storing auditlog infromation
- #
- audit_db_name=ranger (Note the database details used here. This is needed while configuring plugins in later steps)
- audit_db_user=rangerlogger
- audit_db_password=rangerlogger
- # DB information Ranger Policy Store schema
Once all the required properties are updated, execute the below scripts to install ranger admin service.
Execute : ./setup.sh (This will configure the properties)
Execute : ./set_globals.sh (This will create the soft links and other folders for logs, etc)Create a valid symlink in /usr/bin/ for start/stop of ranger admin
cd /usr/bin
ln -sf /usr/local/ranger-admin/ews/startcreate a valid symlink in /usr/bin/ for start/stop of ranger admin
cd /usr/bin
ln -sf /usr/local/ranger-admin/ews/start-ranger-admin.sh ranger-admin-start
ln -sf /usr/local/ranger-admin/ews/stop-ranger-admin.sh ranger-admin-stopstart
ln -sf /usr/local/ranger-admin/ews/stop-ranger-admin.sh ranger-admin-stopUpdate ranger-admin update ranger-admin service file to link to the start and stop scripts
vim /etc/init.d/ranger-admin ( Update the Start and Stop commands to point to the created symlinks )Start the Ranger Admin
service ranger-admin startYou can verify by visiting the external URL of the server using browser, for example :
http://<Host Address>:6080/
Install/Configure Ranger User Sync:
- Start by extracting out binaries at the appropriate place.
cd /usr/local
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-usersync.tar.gz
sudo ln -s ranger-0.4.0-usersync ranger-usersync
sudo mkdir -p /var/log/ranger-usersync
sudo chown ranger /var/log/ranger-usersync; sudo chgrp ranger /var/log/ranger-usersync
cd ranger-usersync - Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
POLICY_MGR_URL=http://localhost:6080
SYNC_SOURCE=unix
logdir=/var/log/ranger/usersync - Now install the usersync by running the setup command
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64 ./setup.sh create a valid symlink in /usr/bin/ for start/stop of ranger usersync
cd /usr/bin
ln -sf /usr/local/ranger-usersync/start.sh ranger-usersync-start
ln -sf /usr/local/ranger-admin/ews/stop.sh ranger-usersync-stopupdate ranger-usersync service file to link to the start and stop scripts
vim /etc/init.d/ranger-usersync ( Update the Start and Stop commands to point to the created symlinks )
Start the Ranger Usersync
service ranger-usersync startYou can verify by looking at the users tab in Ranger Admin. Unix host users should be sync'ed to ranger.
Install/Configure Ranger HDFS Plugin:
- These instructions were written for Hadoop 2.5.2. So grab that tar (hadoop-2.5.2.tar.gz) and checksum file (hadoop-2.5.2.tar.gz.md5).
- Follow the instructions available on the hadoop site http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html.
- Follow steps given in pseudo distributed mode.
- Make note of the location where you installed hadoop. Here I assume that you have installed it in /usr/local/hadoop.
- Create a user under which we could install and ultimately run the various hadoop processes. And login as that user.
...
Defaut user and password is admin/admin. Please change it post update.
Logs are in ews/logs folder. The path is relative to where you have installed ranger-admin. Check xa_portal.log and catalina.out files for ERROR and WARN log messages
Next Steps
Ranger Admin authentication can be configured to use LDAP or Linux system. Consider configuring one of them in production environment. TBD: Provide link to configure LDAP or Linux for authentication
Review database capacity for Audit database. It can grow dramatically in HBase or high volume environment. TBD: Provide link DB capacity planning
Install/Configure Ranger User Sync
What is done here?
Ranger UserSync bits are laid out
Configure Ranger UserSync to synchronize with LDAP/AD or Unix system.
Configure Ranger UserSync to be the authentication server for Linux users (optional)
Other Ranger UserSync properties are configured
Ranger UserSync startup service files are created and updated
Steps
Start by extracting out binaries at the appropriate place.
cd /usr/local
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-usersync.tar.gz
sudo ln -s ranger-0.4.0-usersync ranger-usersync
sudo mkdir -p /var/log/ranger-usersync
sudo chown ranger /var/log/ranger-usersync; sudo chgrp ranger /var/log/ranger-usersync
cd ranger-usersync
Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
POLICY_MGR_URL=http://localhost:6080 (This should point to the server where Ranger Admin has been installed)
SYNC_SOURCE=unix (This assumes that the user from the linux/unix system will be synchronized). If you want to use LDAP users, you can find here how to Configure Ranger UserSync for LDAP
logdir=/var/log/ranger/usersync
Now install the usersync by running the setup command
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64 (Change this to your Java install folder)
./setup.sh
create a valid symlink in /usr/bin/ for start/stop of ranger usersync
cd /usr/bin
ln -sf /usr/local/ranger-usersync/start.sh ranger-usersync-start
ln -sf /usr/local/ranger-admin/ews/stop.sh ranger-usersync-stopupdate ranger-usersync service file to link to the start and stop scripts
vim /etc/init.d/ranger-usersync ( Update the Start and Stop commands to point to the created symlinks )
Start the Ranger Usersync
service ranger-usersync startYou can verify by looking at the users tab in Ranger Admin. Unix host users should be sync'ed to ranger.
Logs are in logs folder. It is relative to the location where ranger-usersync was installed. Look for usersync.log file for User Sync related errors and auth.log for remote login errors.
Next Steps
If you are using LDAP or AD, then you will have to configure using to synchronize with LDAP/AD
Install/Configure Ranger HDFS Plugin
Ranger HDFS plugin helps to centralize HDFS authorization policies. To verify that, first Apache Hadoop needs to be installed. If Hadoop is not already installed, follow below steps.
These instructions were written for Hadoop 2.5.2. So grab that tar (hadoop-2.5.2.tar.gz) and checksum file (hadoop-2.5.2.tar.gz.md5).
Follow the instructions available on the hadoop site http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html.
Follow steps given in pseudo distributed mode.
Make note of the location where you installed hadoop. Here I assume that you have installed it in /usr/local/hadoop.
Create a user under which we could install and ultimately run the various hadoop processes. And login as that user.
sudo useradd --home-dir /var/hadoop --create-home --shell /bin/bash --user-group hadoop
sudo tar xzf hadoop-2.5.2.tar.gz -C /usr/local
cd /usr/local
sudo ln -s hadoop-2.5.2 hadoop
sudo chown hadoop -R hadoop hadoop-2.5.2
sudo chgrp hadoop -R hadoop hadoop-2.5.2
sudo su - hadoop
Now let's follow the below steps to install/configure Ranger HDFS plugin.
Create a repository in Ranger Policy Manager. E.g. "local_hdfs". The same name needs to be configured during plugin setup
Extract binaries at the appropriate place (/usr/local).
cd /usr/local
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-hdfs-plugin.tar.gz
sudo ln -s ranger-0.4.0-hdfs-plugin ranger-hdfs-plugin
cd ranger-hdfs-plugin
Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
POLICY_MGR_URL=http://localhost:6080
REPOSITORY_NAME=local_hdfs
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger
XAAUDIT.DB.USER_NAME=rangerlogger
XAAUDIT.DB.PASSWORD=rangerlogger (Change password as it was set during Ranger Admin install)
Now enable the hdfs-plugin by running the enable-hdfs-plugin.sh command (Remember to set JAVA_HOME)
- Create a symlink as conf dir of hadoop linking to hadoop conf dir
- cd /usr/local/hadoop
- ln -s /usr/local/hadoop/etc/hadoop conf
- Enable Ranger HDFS plugin
- export HADOOP_HOME=/usr/local/hadoop
- cd /usr/local/ranger-hdfs-plugin
- ./enable-hdfs-plugin.sh
- Copy all the jar files from ${hadoop_home}/lib
- cp /usr/local/hadoop/lib/* /usr/local/hadoop/share/hadoop/hdfs/lib/
- Create a symlink as conf dir of hadoop linking to hadoop conf dir
Now edit the xasecure-audit.xml file.
- cd /usr/local/hadoop/conf
- change the xasecure-audit.xml file to look like the below. Make sure the JDBC properties are correct.
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.url</name>
<value>jdbc:mysql://localhost/ranger</value>
</property>
<property>
<name>xasecure.audit.jpa.javax.persistence.jdbc.user</name>
<value>rangerlogger</value>
</property>
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.password</name>
<value>rangerlogger</value>
</property>
Once these changes are done Restart Hadoop namenode. This should start the association of ranger-hdfs-plugin with hadoop.
You can verify by logging into the Ranger Admin Web interface -> Audit -> Agents
Now HDFS resources will be authorized via Ranger policies.
Install/Configure Ranger Hive Plugin
Ranger Hive plugin integrates with Hive to enforce authorization policies. To verify that, first Apache Hive needs to be installed. If not already installed, follow below steps. Version 0.14 is required.
Download and install Hive by following the instructions available on the apache hive website. Hive Install instructions
cd /usr/local
sudo tar zxf ~/dev/apache-hive-0.14.0-bin.tar.gz
sudo ln -s ~/dev/apache-hive-0.14.0-bin.tar.gz hive
cd hive
Note: Hive Server 2 doesn’t start unless HADOOP_VERSION is exported to bashrc
Now let's follow the below steps to install/configure Ranger
...
Hive plugin.
...
Create a repository in Ranger Policy Manager. E.g. "hivedev". The same name needs to be configured during plugin setup
Extract binaries at the appropriate place
...
.
...
cd /usr/local
...
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-
...
hive-plugin.tar.gz
...
sudo ln -s ranger-0.4.0-
...
hive-plugin ranger-
...
hive-plugin
...
cd ranger-
...
hive-plugin
Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
POLICY_MGR_URL=http://localhost:6080
REPOSITORY_NAME=
...
hivedev
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger
XAAUDIT.DB.USER_NAME=rangerlogger
XAAUDIT.DB.PASSWORD=rangerlogger
Now enable the
...
hive-plugin by running the enable-
...
hive-plugin.sh command (Remember to set JAVA_HOME)
...
- cd /usr/local/hadoop
- ln -s /usr/local/hadoop/etc/hadoop conf
...
- echo “export HADOOP_HOME=/usr/local/hadoop” >> /etc/bashrc
cd /usr/local/ranger-
...
hive-plugin
./enable-
...
hive-plugin.sh
...
Now edit the xasecure-audit.xml
...
file.Once these changes are done Restart hive. This should start the association of ranger-hive-plugin with hive.
- cd /usr/local/
...
- hive/conf
...
- Change the xasecure-audit.xml file
...
- . Make sure the JDBC properties are correct.
...
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.url</name>
...
<value>jdbc:mysql://localhost/ranger</value>
...
</property>
...
<property>
...
<name>xasecure.audit.jpa.javax.persistence.jdbc.user</name>
...
<value>rangerlogger</value>
...
</property>
...
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.password</name>
...
<value>rangerlogger</value>
...
</property>
...
You can verify by logging into the Ranger Admin Web interface -> Audit Tab -> Agents
...
Install/Configure Ranger
...
HBase Plugin
...
Ranger
HiveHBase plugin integrates with
HiveHBase to enforce authorization policies. To verify that, first Apache
HiveHBase needs to be installed. If not already installed
, follow below steps. Version 0.14 is required....
, follow below steps. Ranger will work only with HBase version 0.99.2 or above.
- Build Hbase > 0.99.2 (0.99.2RC0 is used for this document)
- cd ~/dev && mkdir hbase-trunk && cd hbase-trunk/
- git clone https://github.com/apache/hbase.git
- cd hbase && git checkout 0.99.2RC0
- mvn clean compile install package -DskipTests=true assembly:single
- Untar the build Hbase assembly to /usr/local directory
- cd /usr/local && cp /root/dev/hbase-trunk/hbase/hbase-assembly/target/hbase-0.99.2-bin.tar.gz .
- tar xzf hbase-0.99.2-bin.tar.gz
...
- sudo ln -s
...
- hbase-0.
...
- 99.2 hbase
- cd hbase
- Now follow the instructions in Apache HBase site to configure in Pseudo mode. http://hbase.apache.org/book/getting_started.html
...
Now let's follow the below steps to install/configure Ranger
...
HBase plugin.
Create a repository in Ranger Policy Manager. E.g. "hbasedev". The same name needs to be configured during plugin setup
Extract
...
binaries at the appropriate place.
cd /usr/local
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-
...
hbase-plugin.tar.gz
sudo ln -s ranger-0.4.0-
...
hbase-plugin ranger-
...
hbase-plugin
cd ranger-
...
hbase-plugin
Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
POLICY_MGR_URL=http://localhost:6080
REPOSITORY_NAME=
...
hbasedev
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger
XAAUDIT.DB.USER_NAME=rangerlogger
XAAUDIT.DB.PASSWORD=rangerlogger
Now enable the
...
hbase-plugin by running the enable-
...
hbase-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-
...
hbase-plugin
./enable-
...
hbase-plugin.sh
Now edit the xasecure-audit.xml file.Once these changes are done Restart HBase. This should start the association of ranger-hbase-plugin with HBase.
- cd /usr/local/
...
- hbase/conf
- Change the xasecure-audit.xml file. Make sure the JDBC properties are correct.
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.url</name>
<value>jdbc:mysql://localhost/ranger</value>
</property>
<property>
<name>xasecure.audit.jpa.javax.persistence.jdbc.user</name>
<value>rangerlogger</value>
</property>
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.password</name>
<value>rangerlogger</value>
</property>
...
You can verify by logging into the Ranger Admin Web interface -> Audit Tab -> Agents
Install/Configure Ranger
...
Knox Plugin
...
Let’s download and install apache Knox from Apache Mirrors. http://mirrors.koehn
5.0/
cd /usr/local
sudo tar zxf ~/dev/knox-0.5.0.tar.gz
sudo ln -s
knox-0.
5.
0 knox
cd knox- Now follow the instructions in Apache
- Knox site to configure
- Knox Gateway/LDAP. Knox Install Instructions
Now let's follow the below steps to install/configure Ranger
...
Knox plugin.
- Create a repository in Ranger Policy Manager. E.g. "knoxdev". The same name needs to be configured during plugin setup
- Set the URL as https://knox_host:8443/gateway/admin/api/v1/topologies (Replace the knox_host with your fully qualified host name)
- Extract
...
- binaries at the appropriate place.
cd /usr/local
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-
...
- knox-plugin.tar.gz
sudo ln -s ranger-0.4.0-
...
- knox-plugin ranger-
...
- knox-plugin
cd ranger-
...
- knox-plugin
- Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
POLICY_MGR_URL=http://localhost:6080
REPOSITORY_NAME=
...
- knoxdev
KNOX_HOME=/usr/local/knox
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger
XAAUDIT.DB.USER_NAME=rangerlogger
XAAUDIT.DB.PASSWORD=rangerlogger - Now enable the
...
- knox-plugin by running the enable-
...
- knox-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-
...
- knox-plugin
./enable-
...
- knox-plugin.sh
- Now edit the xasecure-audit.xml file.
- cd /usr/local/
...
- knox/conf
- Put in /usr/<hadoop-installs>/<your version>/knox/ext/ a symbolic link to mysql-connector-java.jar
- Change the xasecure-audit.xml file. Make sure the JDBC properties are correct.
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.url</name>
<value>jdbc:mysql://localhost/ranger</value>
</property>
<property>
<name>xasecure.audit.jpa.javax.persistence.jdbc.user</name>
<value>rangerlogger</value>
</property>
<property> <name>xasecure.audit.jpa.javax.persistence.jdbc.password</name>
<value>rangerlogger</value>
</property>- Once these changes are done Restart
...
- Knox (Gateway/LDAP). This should start the association of ranger-
...
- knox-plugin with
...
- Knox.
- You can verify by logging into the Ranger Admin Web interface -> Audit Tab -> Agents
Install/Configure Ranger
...
Storm Plugin
...
Ranger
...
Storm plugin integrates with
...
Storm to enforce authorization policies. To verify that, first Apache
...
Storm needs to be installed. If not already installed, follow below steps.
Let’s download and install apache
...
storm from Apache Mirrors.
...
cd /usr/local
...
sudo tar zxf ~/dev/
...
apache-storm-0.
...
9.
...
3.tar.gz
...
sudo ln -s
...
apache-storm-0.
...
9.
...
3 storm
cd storm
Following the instructions available on the apache storm site itself (To install Apache Storm).
Now let's follow the below
...
steps to install/configure Ranger Storm Plugin.
...
Create a repository in Ranger Policy Manager. E.g. "stormdev". The same name needs to be configured during plugin setup
cd /usr/local
...
sudo tar zxf ~/dev/ranger/target/ranger-0.4.0-
...
storm-plugin.tar.gz
...
sudo ln -s ranger-0.4.0-
...
storm-plugin ranger-
...
storm-plugin
...
- cd ranger-
...
- storm-plugin
Now let’s edit the install.properties file.
...
Here are the relevant lines that you should edit:
...
Change the insall.properties file
POLICY_MGR_URL=http://localhost:6080
...
REPOSITORY_NAME=
...
stormdev
...
KNOX_HOME=/usr/local/
...
storm
XAAUDIT.DB.IS_ENABLED=true
XAAUDIT.DB.HOSTNAME=localhost
...
XAAUDIT.DB.DATABASE_NAME=ranger
...
XAAUDIT.DB.USER_NAME=rangerlogger
...
XAAUDIT.DB.PASSWORD=rangerlogger
Now enable the
...
storm-plugin by running the enable-
...
storm-plugin.sh command (Remember to set JAVA_HOME)
...
cd /usr/local/ranger-
...
storm-plugin
...
./enable-
...
storm-plugin.sh
Now edit the xasecure-audit.xml file. Here are the relevant lines that you should edit :
cd /usr/local/
knoxstorm/conf
- Change
change the xasecure-audit.xml
file. Make sure the JDBC properties are correct. <property>file.
<property>
<name>xasecure.audit.jpa.javax.persistence.jdbc.url</name>
...
<value>jdbc:mysql://localhost/ranger</value>
...
</property>
<property>
<name>xasecure.audit.jpa.javax.persistence.jdbc.user</name>
...
<value>rangerlogger</value>
...
</property>
<property>
<name>xasecure.audit.jpa.javax.persistence.jdbc.password</name>
...
<value>rangerlogger</value>
...
</property>
Once these changes are done Restart Storm
- Knox (Gateway/LDAP).
This should start the association of ranger-
knoxstorm-plugin
with Knoxwith storm.
You can verify by logging into the Ranger Admin Web interface ->
Audit TabAudit -> Agents
Install/Configure Ranger Storm Plugin:
...