Apache Kylin : Analytical Data Warehouse for Big Data
Page History
Table of Contents |
---|
In order to facilitate users to troubleshoot problems, kylin will collect all logs to kylin.log in the running process, including the logs generated by spark in the build job and query job, which will be managed by kylin's log system.
...
In order to solve this problem, kylin4.0.0-beta refactored the log of the build job. After refactor, the build log in kylin4 will be separated from kylin.log and uploaded to hdfs. The log4j configuration files of this log include the following two: ${KYLIN_HOME}/conf/spark-driver-log4j.properties and ${KYLIN_HOME}/conf/spark-executor-log4j.properties.
2、Spark driver log of build job
2.1 Output log to hdfs file
spark-driver-log4j.properties is used to configure the output path, appender, layout, etc. of spark driver log in the build job. By default, spark driver log of a step of a build job will be output to a file in hdfs. The file path is spliced by kylin.env.hdfs-working-dir, kylin.metadata.url, project name, step id, etc., where step id is spliced by job Id and two digits counting from 00, for example, a build job's first step's step id is jobId-00, the second step's step id is jobId-01, and the specific path of log file is: ${kylin.env.hdfs-working-dir}/{kylin.metadata.url}/${project_name}/spark_logs/driver/${step_id}/execute_output.json.timestamp.log.
2.2 View logs through kylin WebUI
Users can view logs through the Job Output on the Monitor page on kylin's WebUI. By default, the Output will only show the contents of the first and last 100 lines of all logs of this step. if you need to view all logs, you can click "download the log file" at the top of the Output window to download all logs, and then the complete spark driver log file of this step will be downloaded locally by the browser.
2.3 Output log to local file
If the user does not want to upload the log of spark driver to hdfs during the build job, the configuration item in spark-driver-log4j.properties can be changed:
...
After modifying the configuration, restart kylin, and then the spark driver log of one step of a job will be output to the local file: ${KYLIN_HOME}/logs/spark/${step_id}.log.
3、Spark executor log of build job
3.1 Log output to hdfs file
spark-executor-log4j.properties is used to configure the output path, appender, layout, etc. of spark executor log in the build job. Similar to spark driver log, spark executor log of one step of a build job will be output to a folder in hdfs. Each file in this folder corresponds to an executor log. The specific path is ${kylin.env.hdfs-working-dir}/{kylin.metadata.url}/${project_name}/spark_logs/executor/yyyy-mm-dd/${job_id}/${step_id}/executor-x.log
4、Troubleshooting
When the spark job submitted by kylin is submitted to the yarn cluster for execution, the user who uploads the spark executor log to HDFS may be yarn. At this time, the user of yarn may not have write permission to the hdfs directory ${kylin.env.hdfs-working-dir}/${kylin.metadata.url}/${project _ name}/spark_logs, which leads to the failure of uploading spark executor log. At this time, when viewing the task log with "yarn logs -applicationId <Application ID>", you will see the following error:
...