Examples Refurbisch

Status

Motivation

Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:

• Serve as tutorials to learn Airflow DAG implementation
• Serve with code snippets for documentation
• Serve for testing the setup
• (some) service for CI integration testing

Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.

There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.

So in the Airflow 3 Dev Calls there was a demand named to clean-up and optimize example DAGs.

Considerations / Targets

• The number of examples should be reduced to 20-30
  • If possible examples from docs should be represented in examples. Some code examples which are stand-alone in code should be moved into examples if possible.
  • Otherwise example content not referenced in documentation might be questioned if beneficial
• Examples should be arranged along a story-line if possible which might represent a virtual company and support real-life use cases
• Examples should follow best practices in coding
• Existing examples should be reviewed which DAGs are just used for testing. Testing DAGs should be separated and not pollute the example collection
• Some examples are specific for providers. They should be moved to provider packages
• A mechanism is to be created that uses DAG bundle loading mechanism to load example DAGs from providers w/o need to copy them to global examples.
• Same like today if loading of example DAGs is enabled also needed plugins e.g. timetables should be usable out-of-the-box
• (more tbd?)

Storyline

Technical Work Packages → Features

• --load-example-dags must load examples from standard provider at least in Airflow 3.1 (same like in breeze hack today)
• Testing DAGs must be loadable (at least in breeze) to be able to remove them from example tree
• DAG Budles must be extended depending on installed/available providers to extend examples - allowing to move examples from core to providers (e.g. example_kubernetes_executor.py → cncf.kubernetes)

Proposed Technical Excellence in Example DAGs

• All code has documentation (pydoc)
• All DAGs have DAG MD docs and task MD docs
• All DAGs and Tasks use Typing
• The Examples use tags mapping to use cases of storyline.
  • All examples carry the the tag "Example"
  • If the DAG is serving as a tuutorial, it is having the tag "Tutorial"
  • Current examples that we move off for testing (or which are dual-use) get a tag "Testing"
• Ruff + Mypy checks are enabled and examples follow code quality guidelines
• DAGs and Tasks have nice display names
• Examples do not use deprecated functions
• Examples do not carry top-level code
• All examples are running in the standard setup out-of-the-box. They need to run out-of-the-box (e.g. no connections, SW packages need to be created prior run)
• Examples should integrate into the storyline and not just stand-alone to showcase a technical feature

Current Examples → Example DAG Target Panning

See https://github.com/apache/airflow/tree/main/airflow-core/src/airflow/example_dags