Page History

Status

Motivation

Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:

...

• "Tailwind" - A virtual / non existing wind park energy company that powers a farm of win-mills to produce clean energy. The company has a strong demand to ETL sensor data from the windmills as well as need to act on data events when base data changes or contracts with customers renew. The company values also the DEI rules and has sustainable targets for clean energy and CO2 reduction.

  • Event driven case: New data is dropped on a file system (S3 would be great but can not be executed w/o S3, can be added to AWS provider as extension) that triggers data processing for wind energy. Data is loaded  and Asset events are generated (see Github Issue 52481)

  • Asset driven pipeline around reporting,  data is split up per city of wind turbine and reports about production are distributed to shareholders. Branching can be used to check if a notification is sent via email or a custom notifier. This can also use branch labels. A third notification channel might be broken and as these shareholders are important we inform the admin in case of any task fails (trigger rule) and start a recovery task.
  • Scheduled nightly use case, example reporting is written to file system (where event pipeline is picking up!). As it would be too easy some tasks migth fail and then a custom weight rule is used for retries.
  • Manual correction trigger: Correction wind production counters can be submitted which then also are written to file system
  • Timetable example for maintenance schedule (selected calendar dates) where maintenance notifications are sent (see Github Issue 52479)
  • Scheduled hourly check for wind turbines state. This requires som some special infrastructure to start and stop, using setup+teardown to open a VPN tunnel to the remote machines. This is using a generator pattern and produces the same logic for 3 counties. (see Github Issue 52480)

As for demos and examples a lot of functionality is needed in both decorator as well as classic Dag implementation it would be good to have two similar use cases. Or alternatively describe that the Tailwind south branch prefers to implement all in Pythonic manner whereas the Tailwind North branch data engineers like the classic implementation?

...

•  --load-example-dags must load examples from standard provider at least in Airflow 3.1 (same like in breeze hack today) (see Github Issue 52469)
•  Testing DAGs must be loadable (at least in breeze) to be able to remove them from example tree (See Github Issue 52474)
  Likely we should have a dedicated "test_dags" folder/bundle that should contain dags used for testing only. We can automatically add such dags to be used in unit test via auto-fixture in the shared conftest.py or similar - this way it will work in both breeze and local venv. I think we should aim to have breeze == local env
  
•  DAG Bundles must be extended depending on installed/available providers to extend examples - allowing to move examples from core to providers (e.g. example_kubernetes_executor.py → cncf.kubernetes)
  Likely we can add "examples" section in provider.yaml and move the example dags from "tests" to separate "examples" folder that will be also embedded in the .whl file (so that you can also conditionally enable examples from a given provider). This means that "system" tests will not be a separate "system" folder, but something separate.
  
•  

  We should figure out a way how to show examples from a provider - possibly "load_core_examples=true/false" and "load_provider_examples=[list of providers]" would be a nice way how to do it

•  

  The example have to be reviewed with  "security" point of view. We often have security reports that are pointing to RCE , lack of sanitizations etc. in our examples - which is very important as those examples can be used by others and bad practices propagated to production code.

•  Add a review checklist into the repo to remember the qulity gates we defined for future reviews and extensions after the examples have been cleaned-up. (see Github Issue 52476)
•  Add a check that links in example dags/doc_md are valid to currest RST (See Github Issue 52477)

Proposed Technical Excellence in Example DAGs

...