THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
public interface HAURLSelector {
boolean checkUrl(String url);
void reSelectUrl() throws IOException;
String getSelectedUrl();
} |
Support Spark Job Monitoring
Eagle running job spout pick up MR job monitoring as its first case, and consider to support spark job monitoring as well
Spark on Yarn Environment Setup
Following are some steps for setup a test spark on yarn env
...
6) ./sbin/start-master.sh hdfs://druid-test-host1-556191.slc01.dev.ebayc3.com<hostname>:8020
./sbin/start-slave.sh spark://localhost:7077
./sbin/start-history-server.sh hdfs://<hostaname>:8020
Spark Restful API for monitoring
Following are some spark restful APIs
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
[
{
jobId: 1,
name: "sum at <stdin>:1",
submissionTime: "2016-01-13T09:56:43.335GMT",
completionTime: "2016-01-13T09:56:43.710GMT",
stageIds: [
1
],
status: "FAILED",
numTasks: 2,
numActiveTasks: 1,
numCompletedTasks: 0,
numSkippedTasks: 0,
numFailedTasks: 7,
numActiveStages: 0,
numCompletedStages: 0,
numSkippedStages: 0,
numFailedStages: 1
},
{
jobId: 0,
name: "count at <stdin>:1",
submissionTime: "2016-01-13T09:56:07.496GMT",
completionTime: "2016-01-13T09:56:09.299GMT",
stageIds: [
0
],
status: "SUCCEEDED",
numTasks: 2,
numActiveTasks: 0,
numCompletedTasks: 2,
numSkippedTasks: 2,
numFailedTasks: 0,
numActiveStages: 0,
numCompletedStages: 1,
numSkippedStages: 0,
numFailedStages: 0
}
] |
Notes
Spark History Server reply on logs written by spark applications to report applications' status
...