YARN can be configured to retain logs for debugging purposes. You can set "yarn.nodemanager.delete.debug-delay-sec" to some non-zero value (for example, 3600 for an hour). The Slider dev team is working on a feature to make all logs available through YARN at one location (SLIDER-116). 

Application Logs

Application logs are produced by the application component instances (e.g. HBASE_MASTER for HBASE on YARN). A component is free to store logs anywhere. Component instances are activated as the YARN user (e.g. yarn). Therefore, you need to ensure that the component instances have write permission. The best practice is to log at within ${AGENT_LOG_ROOT}. This symbolic name resolves to the container specific log folder  - <yarn.nodemanager.log-dirs>/application_id/container_id. The default packages available for Slider are by default configured to log at ${AGENT_LOG_ROOT}

Why you may look into Application Logs:

  • Application instance not behaving as expected
  • Application component instances go down after start

Agent Command Logs

Agent command logs are the logs produced by an agent when commands are executed. Currently, the agent execute INSTALL, CONFIGURE, and START. These logs are available at <yarn.nodemanager.log-dirs>/application_id/container_id folder. You will see three types of files here:

  • command-N.json - input to the command, N is a unique command id assigned by the Slider AppMaster

  • output-N.txt - output of the command

  • errors-N.txt - error reported by the command

Why you may look into Agent Command Logs:

  • Failures during command (INSTALL, CONFIGURE, START) execution

Agent Logs

Agent log is the log produced by the agent itself. This log is available at <yarn.nodemanager.log-dirs>/application_id/container_id/slider-agent.log

Why you may look into Agent Logs:

  • AppMaster starts containers but is not able to heartbeat with the Agent (typically communication issues)
  • AppMaster issues commands but commands do not get scheduled for execution

AppMaster Logs

AppMaster runs within its own container. The log files are: <yarn.nodemanager.log-dirs>/application_id/container_id/slider-out.txt, and slider-err.txt. The AppMaster is activated in the first container.  

Why you may look into AppMaster Logs:

  • No containers are being requested from YARN
  • AppMaster fails after start

YARN logs

At times you may need to look into the nodemanager and resourcemanager logs. They are in the YARN log folder which is by default /var/log/hadoop-yarn/yarn. 

  • No labels