Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

What Griffin should focus

Griffin is a generic framework to enable user to measure and monitor the data quality in an easy and extensive manner.

Old Griffin architecture problems

...

What's the painpoints we are facing in the current edition of Griffin from the architectural perspective?

...

  1. To have an abstraction layer of query engine
    The current spark implementation limited Griffin's adoption, since a company could use other query engine in company level. It's too heavy to set up a spark environment to run Griffin.

  2. To support a common data quality workflow: measure-monitor-alert
    In most of real scenarios, measuring is not the goal, but measure-monitor-alert workflow.
      user should be able to define an use case, including:
        1. to measure a dq metrics
        2. to define the monitoring trigger
        3. to define the alerting action
        4. to integrate the 3-steps into a single job/UoW (by scheduler)


Next generation Griffin architecture considerations

...