...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public interface HAURLSelector { boolean checkUrl(String url); void reSelectUrl() throws IOException; String getSelectedUrl(); } |
Support Spark Job Monitoring
Eagle running job spout pick up MR job monitoring as its first case, and consider to support spark job monitoring as well,
Spark on Yarn Environment Setup
...
1) prepare: install hdfs, yarn, java7, scala2.10
...
3) unzip and put it in /opt/spark-1.5.2-bin-hadoop2.6
export SPARK_HOME=/opt/spark-1.5.2
export PATH=$PATH:$SPARK_HOME/bin
Code Block | ||
---|---|---|
| ||
spark.yarn.max_executor.failures 3
spark.yarn.applicationMaster.waitTries 10
spark.history.kerberos.keytab none
spark.yarn.preserve.staging.files False
spark.yarn.submit.file.replication 3
spark.history.kerberos.principal none
spark.yarn.historyServer.address <hostname>:18080
spark.yarn.scheduler.heartbeat.interval-ms 5000
spark.yarn.queue default
spark.yarn.containerLauncherMaxThreads 25
spark.yarn.driver.memoryOverhead 384
spark.history.ui.port 18080
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.yarn.max.executor.failures 3
spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.yarn.executor.memoryOverhead 384
spark.eventLog.enabled true
spark.eventLog.dir hdfs://<hostname>:8020/directory |
Spark Restful API for monitoring
Following are some spark restful APIs
Spark on Yarn Environment Setup
Following are some steps for setup a test spark on yarn env
...
6) ./sbin/start-master.sh hdfs://druid-test-host1-556191.slc01.dev.ebayc3.com<hostname>:8020
./sbin/start-slave.sh spark://localhost:7077
./sbin/start-history-server.sh hdfs://<hostaname>:8020
Spark Restful API for monitoring
Following are some spark restful APIs
...
Code Block | ||||
---|---|---|---|---|
| ||||
[ { jobId: 1, name: "sum at <stdin>:1", submissionTime: "2016-01-13T09:56:43.335GMT", completionTime: "2016-01-13T09:56:43.710GMT", stageIds: [ 1 ], status: "FAILED", numTasks: 2, numActiveTasks: 1, numCompletedTasks: 0, numSkippedTasks: 0, numFailedTasks: 7, numActiveStages: 0, numCompletedStages: 0, numSkippedStages: 0, numFailedStages: 1 }, { jobId: 0, name: "count at <stdin>:1", submissionTime: "2016-01-13T09:56:07.496GMT", completionTime: "2016-01-13T09:56:09.299GMT", stageIds: [ 0 ], status: "SUCCEEDED", numTasks: 2, numActiveTasks: 0, numCompletedTasks: 2, numSkippedTasks: 2, numFailedTasks: 0, numActiveStages: 0, numCompletedStages: 1, numSkippedStages: 0, numFailedStages: 0 } ] |
Notes
Spark History Server reply on logs written by spark applications to report applications' status
But sometime logs may not be correctly updated by spark jobs, for example the following job is actually completed, but the logs on hdfs shows it's still in progress(not completed), which cause spark history server report wrong status
ID | User | Name | Application Type | Queue | StartTime | FinishTime | State | FinalStatus | Progress | Tracking UI |
---|---|---|---|---|---|---|---|---|---|---|
application_1452593058395_0006 | root | PySparkShell | SPARK | default | Tue, 12 Jan 2016 15:27:54 GMT | Tue, 12 Jan 2016 18:05:49 GMT | FINISHED | SUCCEEDED | History |
hdfs dfs -ls /directory/
Found 4 items
-rwxrwx--- 3 root supergroup 13227 2016-01-12 15:27 /directory/application_1452593058395_0005
-rwxrwx--- 3 root supergroup 13227 2016-01-12 18:05 /directory/application_1452593058395_0006.inprogress
-rwxrwx--- 3 root supergroup 51025 2016-01-13 09:48 /directory/application_1452593058395_0007
-rwxrwx--- 3 root supergroup 67994 2016-01-13 09:57 /directory/application_1452593058395_0008