THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
We can discuss roadmap in dev@griffin.incubator.apache.org
Features | Apache Release |
---|---|
Accuracy-Batch | griffin-0.1.5-incubating
|
Accuracy-Streaming | griffin-0.1.6-incubating |
Profiling | griffin-0.1.6-incubating |
Uniqueness | griffin-0.2.0-incubating |
Timeliness | griffin-0.2.0-incubating |
Batch Job Schedule | griffin-0.2.0-incubating |
Streaming Job Schedule | |
Completeness | |
Consistency | |
Validity |
7 Comments
jianwen.fan
质量监控方面,有如下需求:
1、 波动检查
2、 空值检查
3、 枚举值检查
4、 主键冲突
5、 Missing value
6、 ETL 数据延迟
7、 平衡性检查
8、基于历史趋势的异常检测
结合开发的进度,帮忙给排一下期
jianwen.fan
Data quality monitoring, the following requirements:
1.Period check
2.Null value check
3.Enumeration value check
4.Primary Key conflict
5.Missing value
6.ETL data delay
7.Balance check
8.Anomaly detection based on historical trend
Combined with the progress of the development, to help schedule
William Guo
hi Fan,
Could you elaborate more about item 1/7/8, adding more to describe the cases?
Thanks,
William
诸葛子房
hi ,i see the project and find you use the framework of spring boot, in front of project ,i also use the spring boot,but i find the frame is not mature. I think springmvc is more better .
William Guo
could you specify what is the problem in current solution?
诸葛子房
vip Data Quality Platform(DQP)
1.datasources:
(1)hive,postgresql,hive,oracle
(2)api
(3)redis
(4)file(excle ect)
2.measure
we use python and sql ,it may be flexible;but it also has problem.it need the person who use the platform has high skills.
3.project
(1)hive:For simple rules, we adopt the adoption of direct adoption on hive
(2)mysql:For some of the more complex business scenarios, we need to extract the data from the hive through the ETL to the MySQL and then perform it
4.etl
(1)database
(2)api
(3)redis
(4)file
5.authorization
6.alert
send email and message
Soldier Shen
William Guo thanks for building such a great framework. So do we have a rough plan on new feature/design? I also see the "Griffin Improvement Proposals"