...
To further the initial vision from https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop through - to define "incremental processing" - through use cases, patterns of functionality, designs, applications (like feature extraction, model training), algorithms, code, etc. and by building on other technologies and state of the art research.
General context
While "analytics" covers a broad area of interest functionality, here we distinguish between `experimental` and `production` and we focus on `production`.
- `experimental` = is defined as
- exploratory data analysis
- development in notebooks
- essentially ad-hoc choice of tools
- generally batch only, "one off", manual execution
- small data, manual sampling
- models are trained offline
- the end result being reports, diagrams, etc,
- `production` = pretty much the opposite
- end result is are enterprise data science applications
- ran in production
- with large `data , multi-dimensional data set`s that do not fit in RAM RAM, logically infinite
- hence they the algorithms / analysis must be incremental
- use of managed `data set`s : `data lake`s, `feature store`s
- models are trained online incrementally
- offline periodically and refreshed / deployed every few hours/days
- with awareness of `concept drift`, `distribution drift`, `adversarial attacks` and adaptationable to adapt
- use complex orchestration between core analysis and decision layer, model monitoring and other application logic and business processes, some involving human interactions
...
source https://github.com/productml/blurr
Why "continuous"
- Not about "batch" or "stream" - this (only) defines how data arrives first time (in the `data lake`) but not about fully define how analysis code uses data programmatically
- A more appropriate mindset might `sliding window` from `Apache Beam` which provides powerful "window` semantics
- Not about "pipelines" - this implies a an unidirectional, one-pass over data
- Most complex Complex algorithms / analyses / machine learning applications in a production/commercial setting are multi-pass, iterative and must tend to run "online".This means that each component of the analysis produces data that others consume.
Putting this data in kafka, separate databases, etc breaks not just the logical cohesion of the analysis but also leaves no room for global optimization.
See the premise of Weld : https://www.weld.rs/assets/weld-strata.pdf
- Lastly, there are "too many frameworks"and the programming model is too varied
- No! Not Another Deep Learning Framework
- https://www.mosharaf.com/wp-content/uploads/deepstack-hotos17.pdf
Hypothesis : We need "continuous" analytics in both the time dimension and , in the physical dimension.data dimension and in terms of programming model
- relational algebra
- linear algebra
- differential programming
- probabilistic programming
- computational graphs
- differential dataflow
- functional programming
- lazy evaluation
- monadic comprehensions
- etc
- etc
Why "deep"
- In terms of algos we have deep learning, ofc.
- But also in terms of "data fabric" we need to handle multi-dimensional, heterogeneous, business rich meaning data and abstractions over data.
- relational data - need #RelationalAlgebra
- arrays / matrices / tensors - need #LinearAlgebra
- graph data - need graph representations and algos (https://www.slideshare.net/oracle4engineer/using-graphs-for-data-analysis)
...
- "Particle physics, 10,000 times faster" by Jim Pivarski
- "Our data is deeply nested and cross linked"
- https://youtu.be/jvt4v2LTGK0?t=455
Tip | title | Hudi keeps data in files
---|
This is where `Hudi` comes into the picture by allowing data to be kept in files, not just input data but also output data. Hypothesis *: the ability to keep data in the basic files is key to building the aboveParquet files throughout the analysis is key to building the above vision. |
Activity
This page about Uber #Michelangelo https://eng.uber.com/michelangelo/ suggests there is still a distinction in the architecture and implementation (and programming model) between "batch" and "streaming" and the data is placed in distinct kinds of physical repositories between batch and continuous analyses.
...
Here we will list a growing list of use cases that we find useful in the above context.
...
Some of these may lead to `Hudi` HIPs, some to extensions and others to more broad solutions, beyond `Hudi` itself but where `Hudi` plays a part.
Use cases
(UC) Ability to support deletes
- Application to GDPR "right to be forgotten" requirement.
- Context : https://databricks.com/blog/2019/03/19/efficient-upserts-into-data-lakes-databricks-delta.html
- Status : wip https://github.com/apache/incubator-hudi/pull/635
(UC) Integrate Hudi with Apache Beam so that the sliding data window abstractions of beam can run on top of Parquet files incrementally updated through `Hudi`
source: https://qcon.ai/system/files/presentation-slides/simplifying_ml_workflows_with_apache_beam.pdf
Hypothesis The `sliding data window` abstraction from Apache Beam (also present in Spark and Flink) can eliminate most (all ) of the ad-hoc attempts to handle incremental data inside analysis code.
(UC) Use Hudi to build file based data lakes that are incrementally self updating (data fabric)
`Hudi` works with one `data set` at a time but when building a `data lake` we need to relate `data set`s structurally and logically (business semantics) so that `feature store`s can be built from raw data
(UC) Use Hudi to build file based feature stores (data fabric)
The first kind is relational data but we also need graph, array and other forms of relations in data, ideally in an unified `data fabric`.
Resources about how Dremio relational cache works for inspiration on how `Hudi` might play in
- https://www.slideshare.net/dremio/using-apache-arrow-calcite-and-parquet-to-build-a-relational-cache-81440786
- https://www.dremio.com/webinars/apache-arrow-calcite-parquet-relational-cache/
Technologies on the radar
- Apache Arrow
- Dremio
- Project DAWN Weld (https://www.weld.rs/)
Resources / reading list
- "Evaluating End-to-End Optimization for Data Analytics Applications in Weld"
- "Bridging the Gap: Towards Optimization Across Linear and Relational Algebra"
- "Accessible Machine Learning through Data Workflow Management"