Page History

...

Concepts and Primitives Lack Organization

Currently,

...

DataStream API provides primitives that corresponds to concepts of many different levels. This makes it difficult to maintain consistency across different interfaces and breaks our principle of make it as flexible as possible and easy to understand and use. The main reasons are as follows:

• It contains the most fundamental concepts that defines a stateful stream processing application: data streams, transforms, states, etc.

• It also allows defining arbitrary transforms via process functions and custom operators.

• It provides high-level implementations and utilities for common data processing patterns: map, aggregate, join, window, etc.

• There are also short-cuts for specific source / sink implementations, mainly for testing purpose.

...