TL;DR
If all services in the minicluster are set up correctly and are up, run this to restart the Impala cluster with Ranger authorization:
Code Block |
---|
bin/start-impala-cluster.py --impalad_args="--server_name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger" --catalogd_args="--server_name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger" |
If you want to investigate Hive behaviors in Ranger authorization, run this to restart Hive:
Code Block |
---|
testdata/bin/run-hive-server.sh -with_ranger |
Details
When starting an Impala minicluster on a local machine using '$IMPALA_HOME/bin/start-impala-cluster.py', by default there is no authorization service, e.g., Sentry or Ranger, enabled on this cluster. We could enable the Ranger service initialized with the default policies on an Impala minicluster by the following steps (assuming that we have already executed '$IMPALA_HOME/testdata/bin/run-all.sh').
- Execute '$IMPALA_HOME/testdata/bin/kill-allranger-server.sh'
- Execute 'source $IMPALA_HOME/bin/impala-config.sh'
- Execute '$IMPALA_HOME/buildall.sh -noclean -notests -ninja'
- Execute '$IMPALA_HOME/bin/create-test-configuration.sh -create_ranger_policy_db'
- Execute '$IMPALA_HOME/testdata/bin/run-ranger-allserver.sh'
- Execute '$IMPALA_HOME/testdata/bin/createsetup-load-dataranger.sh'
- Execute the following command. We note that the arguments passed into 'start-impala-cluster.py' could also be found at '$IMPALA_HOME/tests/authorization/test_ranger.py'. The first 4 arguments are passed to 'start-impala-cluster.py' directly and those surrounded by a pair of single quotation marks are passed to statestore, impalad, and catalogd, respectively.
$IMPALA_HOME/bin/start-impala-cluster.py \
--cluster_size=3 \
--num_coordinators=3 \
--log_dir=/tmp/ \
--log_level=1 \
'--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' \
'--impalad_args=--server-_name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger' \
'--catalogd_args=--server-_name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger'
Once the minicluster has been started, we could log into the Impala shell as an administrator with username 'admin' by executing '$IMPALA_HOME/bin/impala-shell -u admin'. The account of 'admin' was added by the function 'setup-ranger' in 'create-load-data.sh' above. To tell whether or not the Impala is Ranger-enabled, we could try to execute 'refresh authorization' in the Impala shell. If the Ranger is enabled in Impala, we would see some output similar to the following.
Code Block |
---|
[localhost:21000] default> refresh authorization; |
...
Query: refresh authorization |
...
Query submitted at: 2019-08-29 15:17:37 (Coordinator: http://fangyurao-OptiPlex-9020:25000) |
...
Query progress can be monitored at: http://fangyurao-OptiPlex-9020:25000/query_plan?query_id=374567f0bf4ca48b:0769d23700000000 |
...
Fetched 0 row(s) in 0.02s |
If the Ranger service is not correctly enabled, then after executing '$IMPALA_HOME/bin/impala-shell -u admin' followed by 'refresh authorization' in the Impala shell, we may see the following error message.
Code Block |
---|
[localhost:21000] default> refresh authorization; |
...
Query: refresh authorization |
...
Query submitted at: 2019-08-29 15:23:37 (Coordinator: http://fangyurao-OptiPlex-9020:25000) |
...
ERROR: AnalysisException: Authorization is not enabled. To enable authorization restart Impala with the --server_name=<name> flag. |
We also note that currently to create 'admin' in the Ranger service before this account is created, we have to run '$IMPALA_HOME/testdata/bin/create-load-data.sh' (the 5th step above to start a Ranger-enabled Impala minicluster), which does much more than needed because this script will also load the whole test datasets, which is time-consuming. A better approach is thus to only call the function 'setup-ranger' in 'create-load-data.sh' only. To achieve this, we may consider moving the function 'setup-ranger' out of 'create-load-data.sh' and then we make 'create-load-data.sh' call 'setup-ranger'.
...
Code Block | ||
---|---|---|
| ||
RANGER_SETUP_DIR="${IMPALA_HOME}/testdata/cluster/ranger/setup" perl -wpl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' \ "${RANGER_SETUP_DIR}/impala_group_owner.json.template" > \ "${RANGER_SETUP_DIR}/impala_group_owner.json" export GROUP_ID_OWNER=$(wget -qO - --auth-no-challenge --user=admin --password=admin \ --post-file="${RANGER_SETUP_DIR}/impala_group_owner.json" \ --header="accept:application/json" \ --header="Content-Type:application/json" \ http://localhost:6080/service/xusers/secure/groups | python -c "import sys, json; print json.load(sys.stdin)['id']") export GROUP_ID_NON_OWNER=$(wget -qO - --auth-no-challenge --user=admin \ --password=admin --post-file="${RANGER_SETUP_DIR}/impala_group_non_owner.json" \ --header="accept:application/json" \ --header="Content-Type:application/json" \ http://localhost:6080/service/xusers/secure/groups | python -c "import sys, json; print json.load(sys.stdin)['id']") perl -wpl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' \ "${RANGER_SETUP_DIR}/impala_user_owner.json.template" > \ "${RANGER_SETUP_DIR}/impala_user_owner.json" perl -wpl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' \ "${RANGER_SETUP_DIR}/impala_user_non_owner.json.template" > \ "${RANGER_SETUP_DIR}/impala_user_non_owner.json" wget -O /dev/null --auth-no-challenge --user=admin --password=admin \ --post-file="${RANGER_SETUP_DIR}/impala_user_owner.json" \ --header="Content-Type:application/json" \ http://localhost:6080/service/xusers/secure/users wget -O /dev/null --auth-no-challenge --user=admin --password=admin \ --post-file="${RANGER_SETUP_DIR}/impala_user_non_owner.json" \ --header="Content-Type:application/json" \ http://localhost:6080/service/xusers/secure/users wget -O /dev/null --auth-no-challenge --user=admin --password=admin \ --post-file="${RANGER_SETUP_DIR}/impala_service.json" \ --header="Content-Type:application/json" \ http://localhost:6080/service/public/v2/api/service curl -u admin:admin -H "Accept: application/json" -H "Content-Type: application/json" \ -X PUT http://localhost:6080/service/public/v2/api/policy/4 \ -d @"${RANGER_SETUP_DIR}/policy_4_revised.json" |
Then you should be able to see the "test_impala" service in your Ranger portal (default to http://localhost:6080) like this
...
If you encounter errors in executing the wget commands, try restart Ranger by testdata/bin/run-ranger-server.sh. If Ranger fails to start, try reconfigure the ranger db by "bin/create-test-configuration.sh -create_ranger_policy_db".
2. Failed to start Ranger Admin server due to "Permission denied"
This error usually happens when you first set up the ranger service or after we bump the CDP build number (so you need to set up the ranger service again!). The errors are
Code Block |
---|
find: ‘/home/quanlong/workspace/Impala/toolchain/cdp_components-7049391/ranger-2.1.0.7.2.7.0-44-admin/ews/webapp/WEB-INF/classes/conf/’: No such file or directory
mkdir: cannot create directory ‘/var/run/ranger’: Permission denied
chmod: cannot access '/var/run/ranger': No such file or directory
Restarting Apache Ranger Admin
Apache Ranger Admin Service is not running
Starting Apache Ranger Admin Service
Apache Ranger Admin Service failed to start! |
The script should not creating '/var/run/ranger'. It reveals that RANGER_PID_DIR_PATH is not set correctly. It's default value is '/var/run/ranger'.
You should run bin/create-test-configuration.sh to set up the ranger service. It will copy some scripts to the ranger dir so some vars like RANGER_PID_DIR_PATH can be set correctly. After these, testdata/bin/run-ranger-server.sh will succeed.
Remember to run bin/create-test-configuration.sh when you bump the CDP build number. Because a new Ranger dir will be created in toolchain and it should be set up as well.