The Knox Gateway provides a single access point for all REST interactions with Hadoop clusters. It will be valuable to monitor the access events happening in knox gateway and see if there is an anomaly and generate an alert.
Apache Knox Audit Log Format
...
Code Block | ||
---|---|---|
| ||
EVENT_PUBLISHING_TIME ROOT_REQUEST_ID|PARENT_REQUEST_ID|REQUEST_ID|LOGGER_NAME|TARGET_SERVICE_NAME|USER_NAME|PROXY_USER_NAME|SYSTEM_USER_NAME|ACTION|RESOURCE_TYPE|RESOURCE_NAME|OUTCOME|LOGGING_MESSAGE |
...
Component | Description |
---|---|
EVENT_PUBLISHING_TIME | Time when audit record was published. |
ROOT_REQUEST_ID | The root request ID if this is a sub-request. Currently it is empty. |
PARENT_REQUEST_ID | The parent request ID if this is a sub-request. Currently it is empty. |
REQUEST_ID | A unique value representing the current, active request. If the current request id value is different from the current parent request id value then the current request id value is moved to the parent request id before it is replaced by the provided request id. If the root request id is not set it will be set with the first non-null value of either the parent request id or the passed request id. |
LOGGER_NAME | The name of the logger |
TARGET_SERVICE_NAME | Name of Hadoop service. Can be empty if audit record is not linked to any Hadoop service, for example, audit record for topology deployment. |
USER_NAME | Name of user that initiated session with Knox |
PROXY_USER_NAME | Mapped user name. |
SYSTEM_USER_NAME | Currently is empty. |
ACTION | Type of action that was executed. Following actions are defined: authentication, authorization, redeploy, deploy, undeploy, identity-mapping, dispatch, access. |
RESOURCE_TYPE | Type of resource for which action was executed. Following resource types are defined: uri, topology, principal. |
RESOURCE_NAME | Name of resource. For resource of type topology it is name of topology. For resource of type uri it is inbound or dispatch request path. For resource of type principal it is a name of mapped user. |
OUTCOME | Action result type. Following outcomes are defined: success, failure, unavailable. |
LOGGING_MESSAGE | Logging message. Contains additional tracking information. |
...
Sample log events with DEBUG level logging,
Code Block | ||||
---|---|---|---|---|
| ||||
16/02/04 12:28:29 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|unavailable|Request method: GET
16/02/04 12:28:30 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|
16/02/04 12:28:30 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|Groups: []
16/02/04 12:28:30 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||dispatch|uri|http://<hadoop_host>:50070/webhdfs/v1/user/guest?op=LISTSTATUS&user.name=guest|unavailable|Request method: GET
16/02/04 12:28:31 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||dispatch|uri|http://<hadoop_host>:50070/webhdfs/v1/user/guest?op=LISTSTATUS&user.name=guest|success|Response status: 200
16/02/04 12:28:31 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|Response status: 200 |
Code Block | ||||
---|---|---|---|---|
| ||||
16/02/04 12:29:58 ||24cb20a1-0287-44b5-bafd-afea8dd333a5|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|unavailable|Request method: GET 16/02/04 12:29:58 ||24cb20a1-0287-44b5-bafd-afea8dd333a5|audit|WEBHDFS||||authentication|principal|guest|failure|LDAP authentication failed. 16/02/04 12:29:58 ||24cb20a1-0287-44b5-bafd-afea8dd333a5|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|Response status: 401 |
Code Block | ||||
---|---|---|---|---|
| ||||
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|unavailable|Request method: GET
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|success|
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|success|Groups: []
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||dispatch|uri|http://<hadoop_host>:50070/webhdfs/v1/user/invalid-guest?op=LISTSTATUS&user.name=guest|unavailable|Request method: GET
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||dispatch|uri|http://localhost:50070/webhdfs/v1/user/invalid-guest?op=LISTSTATUS&user.name=guest|success|Response status: 404
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|success|Response status: 404 |
Code Block | ||||
---|---|---|---|---|
| ||||
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE||||access|uri|/gateway/sandbox/hbase/version/cluster|unavailable|Request method: GET
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||authentication|uri|/gateway/sandbox/hbase/version/cluster|success|
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||authentication|uri|/gateway/sandbox/hbase/version/cluster|success|Groups: []
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||dispatch|uri|http://<hadoop_host>:60080/version/cluster?user.name=guest|unavailable|Request method: GET
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||dispatch|uri|http://<hadoop_host>:60080/version/cluster?user.name=guest|failure|
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||access|uri|/gateway/sandbox/hbase/version/cluster|failure| |
/ TODO
Sending Audit Messages to Kafka
...
Code Block | ||
---|---|---|
| ||
# add KAFKA appender to the audit logger log4j.logger.audit=INFODEBUG, auditfile, KAFKA # kafka log4j appender configuration log4j.appender.KAFKA_KNOX_AUDIT_LOG=kafka.producer.KafkaLog4jAppender log4j.appender.KAFKA_KNOX_AUDIT_LOG.layout=org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout log4j.appender.KAFKA_KNOX_AUDIT_LOG.BrokerList=<HOST>:<PORT> # default port - 6667 log4j.appender.KAFKA_KNOX_AUDIT_LOG.Topic=knox_audit_log log4j.appender.KAFKA_KNOX_AUDIT_LOG.Serializer=kafka.test.AppenderStringSerializer log4j.appender.KAFKA_KNOX_AUDIT_LOG.compressionType=none log4j.appender.KAFKA_KNOX_AUDIT_LOG.requiredNumAcks=0 log4j.appender.KAFKA_KNOX_AUDIT_LOG.syncSend=true |
...