The Knox Gateway provides a single access point for all REST interactions with Hadoop clusters. It will be valuable to monitor the access events happening in knox gateway and see if there is an anomaly and generate an alert.
EAGLE-144
Apache Knox Audit Log Format
Apache Knox logs are audited in the following format by default,
EVENT_PUBLISHING_TIME ROOT_REQUEST_ID|PARENT_REQUEST_ID|REQUEST_ID|LOGGER_NAME|TARGET_SERVICE_NAME|USER_NAME|PROXY_USER_NAME|SYSTEM_USER_NAME|ACTION|RESOURCE_TYPE|RESOURCE_NAME|OUTCOME|LOGGING_MESSAGE
Component | Description |
|---|
EVENT_PUBLISHING_TIME | Time when audit record was published. |
ROOT_REQUEST_ID | The root request ID if this is a sub-request. Currently it is empty. |
PARENT_REQUEST_ID | The parent request ID if this is a sub-request. Currently it is empty. |
REQUEST_ID | A unique value representing the current, active request. If the current request id value is different from the current parent request id value then the current request id value is moved to the parent request id before it is replaced by the provided request id. If the root request id is not set it will be set with the first non-null value of either the parent request id or the passed request id. |
LOGGER_NAME | The name of the logger |
TARGET_SERVICE_NAME | Name of Hadoop service. Can be empty if audit record is not linked to any Hadoop service, for example, audit record for topology deployment. |
USER_NAME | Name of user that initiated session with Knox |
PROXY_USER_NAME | Mapped user name. |
SYSTEM_USER_NAME | Currently is empty. |
ACTION | Type of action that was executed. Following actions are defined: authentication, authorization, redeploy, deploy, undeploy, identity-mapping, dispatch, access. |
RESOURCE_TYPE | Type of resource for which action was executed. Following resource types are defined: uri, topology, principal. |
RESOURCE_NAME | Name of resource. For resource of type topology it is name of topology. For resource of type uri it is inbound or dispatch request path. For resource of type principal it is a name of mapped user. |
OUTCOME | Action result type. Following outcomes are defined: success, failure, unavailable. |
LOGGING_MESSAGE | Logging message. Contains additional tracking information. |
Sample log events with DEBUG level logging,
16/02/04 12:28:29 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|unavailable|Request method: GET
16/02/04 12:28:30 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|
16/02/04 12:28:30 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|Groups: []
16/02/04 12:28:30 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||dispatch|uri|http://<hadoop_host>:50070/webhdfs/v1/user/guest?op=LISTSTATUS&user.name=guest|unavailable|Request method: GET
16/02/04 12:28:31 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||dispatch|uri|http://<hadoop_host>:50070/webhdfs/v1/user/guest?op=LISTSTATUS&user.name=guest|success|Response status: 200
16/02/04 12:28:31 ||af043c01-1289-458f-b264-63a1686a585a|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|Response status: 200
16/02/04 12:29:58 ||24cb20a1-0287-44b5-bafd-afea8dd333a5|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|unavailable|Request method: GET
16/02/04 12:29:58 ||24cb20a1-0287-44b5-bafd-afea8dd333a5|audit|WEBHDFS||||authentication|principal|guest|failure|LDAP authentication failed.
16/02/04 12:29:58 ||24cb20a1-0287-44b5-bafd-afea8dd333a5|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/guest?op=LISTSTATUS|success|Response status: 401
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|unavailable|Request method: GET
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|success|
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|success|Groups: []
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||dispatch|uri|http://<hadoop_host>:50070/webhdfs/v1/user/invalid-guest?op=LISTSTATUS&user.name=guest|unavailable|Request method: GET
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||dispatch|uri|http://localhost:50070/webhdfs/v1/user/invalid-guest?op=LISTSTATUS&user.name=guest|success|Response status: 404
16/02/04 12:32:17 ||47914988-4558-428c-988f-8e7d54daa71e|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/invalid-guest?op=LISTSTATUS|success|Response status: 404
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE||||access|uri|/gateway/sandbox/hbase/version/cluster|unavailable|Request method: GET
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||authentication|uri|/gateway/sandbox/hbase/version/cluster|success|
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||authentication|uri|/gateway/sandbox/hbase/version/cluster|success|Groups: []
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||dispatch|uri|http://<hadoop_host>:60080/version/cluster?user.name=guest|unavailable|Request method: GET
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||dispatch|uri|http://<hadoop_host>:60080/version/cluster?user.name=guest|failure|
16/02/04 12:31:11 ||d8b85bb1-29c9-4d67-81b3-b13a56ce22c7|audit|WEBHBASE|guest|||access|uri|/gateway/sandbox/hbase/version/cluster|failure|
Sending Audit Messages to Kafka
Using the KafkaLog4jAppender, the gateway-audit.log messages for Knox can be send directly to Kafka with the following configurations,
# add KAFKA appender to the audit logger
log4j.logger.audit=DEBUG, auditfile, KAFKA
# kafka log4j appender configuration
log4j.appender.KAFKA_KNOX_AUDIT_LOG=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA_KNOX_AUDIT_LOG.layout=org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout
log4j.appender.KAFKA_KNOX_AUDIT_LOG.BrokerList=<HOST>:<PORT> # default port - 6667
log4j.appender.KAFKA_KNOX_AUDIT_LOG.Topic=knox_audit_log
log4j.appender.KAFKA_KNOX_AUDIT_LOG.Serializer=kafka.test.AppenderStringSerializer
log4j.appender.KAFKA_KNOX_AUDIT_LOG.compressionType=none
log4j.appender.KAFKA_KNOX_AUDIT_LOG.requiredNumAcks=0
log4j.appender.KAFKA_KNOX_AUDIT_LOG.syncSend=true
Following dependencies are required,
- kafka-<version>.jar
- kafka-clients-<version>.jar
- scala-library-<version>.jar
References
Apache Knox User Guide
