Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

livy integration

Griffin uses livy as job-submitter of spark, all jobs will be submitted via livy client.

...

SparkSubmitJob.java leverages to quartz scheduler and call livy client to submit jobs


DSL parser

Griffin gives a new sql-like language to define data quality computation, there is an implementation in measure project.

GriffinDslParser.scala is responsible for parsing rules in data quality.


Sink

Data sink is a term used to describe a computer or any other medium capable of receiving data.

Sink.scala is an abstract data sink type in griffin which deals with all data processed.