Problem
If user wants to use Eagle for general metric/log-based monitoring like hadoop jmx monitoring, user has to setup-up development environment, rewrite code though we provided high-level fluent DSL and re-build, package and submit the topology manually, which is not good for user experience nor re-use eagle's existing capability for more monitoring cases, so we propose a dynamical monitoring framework.
The dynamical Monitoring Framework is consist of four parts:
- Pipeline Builder: Declarative Pipeline Modeling, DSL, Parser and Compiler
- Pipeline Runner: Provide tool/CLI to read pipeline definition, parse, compile and submit pipeline
- Pipeline Scheduler: Automatically pipeline scheduler
- Pipeline Designer: Graphic-baed Pipeline designer
Features
- Compile DSL Configure to Pipeline model
- Compile Pipeline model to Stream Execution Graph
- Submit Stream Execution Graph to actual running environment say storm
- Support Alert and Persistence for metric monitoring
- Extensible stream module management
- Automatically scan and register module
- Pipeline runner CLI tool and shell script
- Decouple pipeline compiler and scheduler into individual modules
- Stream Pipeline Scheduler
- Graph editor to define streaming graph in UI
- JSON/Config & Scala Case Class Mapping (https://github.com/scala/pickling)
- Raw message structure oriented programing is a little ugly, we should define a generic message/event consist of [payload:stream/timestamp/serializer/deserializer,data:message]
- Provide stream schema inline and send to metadata when submitting
- UI should support specify executorId when defining new stream
- Lack of a entity named StreamEntity for the workflow of defining topology&policy end-to-end
- Fix multi-configure bug
- Fix configuration conflict
Components
Dynamical Pipeline Builder
Pipeline ::= config + schema + dataflow
Configuration based DSL
Tickets
Definition
pipeline { context { // pipeline configuraiton } schema{ ${schemaName} { ${field} : ${type} } } dataflow{ // 1. DAG Node definition ${ModuleType}.${streamId} { // module confiruation } // 2. DAG Connector definition ${streamId} -> ${streamId} { // Connection configuration like grouping = "shuffle" } } }
Example
Finally compile to follow DAG in runtime.
Dynamical Pipeline Runner
- Pipeline Runner CLI
Dynamical Pipeline Scheduler
Dynamical Pipeline Designer
Requirements
- User could define the monitoring DAG pipeline in UI by DSL
- User could define the monitoring DAG pipeline in UI with UI actions
Use Case
Hadoop JMX Metric Monitoring