Examples Refurbish

Status

Motivation

Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:

• Serve as tutorials to learn Airflow DAG implementation
• Serve with code snippets for documentation
• Serve for testing the setup
• (some) service for CI integration testing

Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.

There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.

So in the Airflow 3 Dev Calls there was a demand named to clean-up and optimize example DAGs.

Considerations / Targets

• The number of examples should be reduced to 20-30
  • If possible examples from docs should be represented in examples. Some code examples which are stand-alone in code should be moved into examples if possible.
  • Otherwise example content not referenced in documentation might be questioned if beneficial
• Examples should be arranged along a story-line if possible which might represent a virtual company and support real-life use cases
• Examples should follow best practices in coding
• Existing examples should be reviewed which DAGs are just used for testing. Testing DAGs should be separated and not pollute the example collection
• Some examples are specific for providers. They should be moved to provider packages
• A mechanism is to be created that uses DAG bundle loading mechanism to load example DAGs from providers w/o need to copy them to global examples.
• Same like today if loading of example DAGs is enabled also needed plugins e.g. timetables should be usable out-of-the-box
• (more tbd?)

Storyline

Current ideas collection:

• "Tailwind" - A virtual / non existing wind park energy company that powers a farm of win-mills to produce clean energy. The company has a strong demand to ETL sensor data from the windmills as well as need to act on data events when base data changes or contracts with customers renew. The company values also the DEI rules and has sustainable targets for clean energy and CO2 reduction.

Technical Work Packages → Features

• --load-example-dags must load examples from standard provider at least in Airflow 3.1 (same like in breeze hack today)
• Testing DAGs must be loadable (at least in breeze) to be able to remove them from example tree
• DAG Budles must be extended depending on installed/available providers to extend examples - allowing to move examples from core to providers (e.g. example_kubernetes_executor.py → cncf.kubernetes)

• We should figure out a way how to show examples from a provider - possibly "load_core_examples=true/false" and "load_provider_examples=[list of providers]" would be a nice way how to do it

  
  

Proposed Technical Excellence in Example DAGs

• All code has documentation (pydoc)
• All DAGs have DAG MD docs and task MD docs
• All DAGs and Tasks use Typing
• The Examples use tags mapping to use cases of storyline.
  • All examples carry the the tag "Example"
  • If the DAG is serving as a tuutorial, it is having the tag "Tutorial"
  • Current examples that we move off for testing (or which are dual-use) get a tag "Testing"
• Ruff + Mypy checks are enabled and examples follow code quality guidelines
• DAGs and Tasks have nice display names
• Examples do not use deprecated functions
• Examples do not carry top-level code
• All examples are running in the standard setup out-of-the-box. They need to run out-of-the-box (e.g. no connections, SW packages need to be created prior run)
• Examples should integrate into the storyline and not just stand-alone to showcase a technical feature

Current Examples → Example DAG Target Panning

See https://github.com/apache/airflow/tree/main/airflow-core/src/airflow/example_dags