THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- IOChannelFactory Redesign [doc]
- Configurable BeamFileSystem [doc]
- New API for writing files in Beam [doc]
- Dynamic file-based sinks [doc]
- Event Time and Watermarks in KafkaIO [doc]
- Exactly-once Kafka sink [doc]
Metrics
- Defining and Adding SDK Metrics via FN API [doc]
- Metrics architecture inside the runners
- Get Metrics API: Metric Extraction via proto RPC API. [doc]
- Metrics API [doc]
- I/O Metrics [doc]
- Metrics extraction independent from runners / execution engines [doc]
- Watermark Metrics [doc]
- Support Dropwizard Metrics in Beam [doc]
...
- Streams and Tables [doc]
- Streaming SQL [doc]
- Schema-Aware PCollections [doc]
- Pubsub to Beam SQL [doc]
- Apache Beam Proposal: design of DSL SQL interface [doc]
- Calcite/Beam SQL Windowing [doc]
- Reject Unsupported Windowing Strategies in JOIN [doc]
- Beam DSL_SQL branch API review [doc]
- Complex Types Support for Beam SQL DDL [mail]
- [SQL] Reject unsupported inputs to Joins [mail]
- Integrating runners & IO [doc]
- Beam SQL Pipeline Options [doc]
- Unbounded limit [doc]
- Portable Beam Schemas [doc]
- Cost Based Optimizer [doc1, doc2]
- ZetaSQL as a dialect in BeamSQL [doc]
- Project and predicate push-down [doc]
Portability
- Portability Framework
- The model protos contain all aspects of the portability API and is the truth on the ground. The proto definitions supercede any design documents. The main design documents are the following:
Runner API. Pipeline representation and discussion on primitive/composite transforms and optimizations.
Job API. Job submission and management protocol.
Fn API. Execution-side control and data protocols and overview.
Container contract. Execution-side docker container invocation and provisioning protocols. See CONTAINERS.md for how to build container images.
- Cross language. Options and tradeoffs for how to handle various kinds of multi-language/multi-SDK pipelines.
- Fn API
- Apache Beam Fn API Overview [doc]
- Processing a Bundle [doc]
- Progress [doc]
- Graphical view of progress [doc]
- Fn State API and Bundle Processing [doc]
- Checkpointing and splitting of Beam bundles over the Fn API, with application to SDF [doc]
- How to send and receive data [doc]
- Defining and adding SDK Metrics [doc]
- SDK harness container contract [doc]
- Structure and Lifting of Combines [doc]
- Cross-language Beam Pipelines [doc]
- SDK X with Runner Y using Runner API [doc]
- Flink Portable Runner Overview [doc]
- Launching portable pipeline on Flink Runner [doc]
- Portability support [table]
- Portability Prototype [doc]
- Portable Artifact Staging [doc]
- Portable Beam on Flink [doc]
- Portability API: How to Checkpoint and Split Bundles [doc]
- Portability API: How to Finalize Bundles [doc]
- Side Input in Universal Reference Runner [doc]
- Spark Portable Runner Overview [doc]
- Cross-Language Pipelines & Legacy IO [doc]
- Artifact Staging in Cross-Language Pipelines [doc]
...