Versions Compared

Old Version 13

changes.mady.by.user Unknown User (jscheffl)

Saved on

compared with

New Version 14

changes.mady.by.user Aritra

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • "Tailwind" - A virtual / non existing wind park energy company that powers a farm of win-mills to produce clean energy. The company has a strong demand to ETL sensor data from the windmills as well as need to act on data events when base data changes or contracts with customers renew. The company values also the DEI rules and has sustainable targets for clean energy and CO2 reduction.
    • Event diven driven case: New data is dropped on a file system (S3 would be great but can not be executed w/o S3, can be added to AWS provider as extension) that triggers data processing for wind energy. Data is loaded  and Asset events are generated
    • Asset driven pipeline around reporting,  data is split up per city of wind turbine and reports about production are distributed to shareholders. Branching can be used to check if a notification is sent via email or a custom notifier. This can also use branch labels. A third notification channel might be broken and as these shareholders are important we inform the admin in case of any task fails (trigger rule) and start a recovery task.
    • Scheduled nightly use case, example reporting is written to file system (where event pipeline is picking up!). As it would be too easy some tasks migth fail and then a custom weight rule is used for retries.
    • Manual correction trigger: Correction wind production counters can be submitted which then also are written to file system
    • Timetable example for maintenance schedule (selectec selected calendar dates) where maintenance notifications are sent
    • Scheduled hourly check for wind turbines state. This requires som special infrastructure to start and stop, using setup+teardown to open a VPN tunnel to the remote machines. This is using a generator pattern and produces the same logic for 3 counties.

As for demos and examples a lot of functionality is needed in both decorator as well as classic Dag implementation it woul dbe would be good to have two similar use cases. Or alternatively dscribe that the Tailwind south branch prefers to implement all in Pythonic manner whereas the Tailwind North branch data engineers like the classic implementation?

...

See https://github.com/apache/airflow/tree/main/airflow-core/src/airflow/example_dags

NameGapsReferenced inFeatures usedProposed Change

Core - airflow-core/src/airflow/example_dags

example_asset_alias.py


-AssetAlias, taskflow

Documentation in https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/assets.html#dynamic-data-events-emitting-and-asset-creation-through-assetalias does not use the examples. Either ned to add to real-world example, refernce it in docs or delete them. Unknown User (uranusjr) Do you have an ida how to map this to a real world example (question)

example_asset_alias_with_no_taskflow.py


-AssetAlias

example_asset_decorator.py


-assets, decoratorsBasic examples should be integrated into the storyline. Else too many dags w/o business example. Then drop (error)

example_assets.py


-assets

example_asset_with_watchers.py


-AssetWatcherNeeds to be intgrated into case, not standalone. Then drop (error)

example_branch_labels.py


airflow-core/docs/core-concepts/dags.rst
Needs to be intgrated into case, not standalone. Then drop (error)

example_branch_python_dop_operator_3.py


-
Needs to be intgrated into case, not standalone, then drop (error)

example_complex.py


airflow-core/docs/howto/usage-cli.rst
Lags a real business case. But complexity might be still a good show case. Check for the resulting example, if similar complexity then drop (error)

example_custom_weight.py


airflow-core/docs/administration-and-deployment/priority-weight.rstpriority weights, Just a technical example. Needs to be kept but would be best to
intgrate
integrate in a real use case. (question)

example_dag_decorator.py


airflow-core/docs/core-concepts/dags.rst
Not needed as standalone example if the business example contains similar code. Rework into a real business example then Drop (error)

example_display_name.py


-
Drop standalone example. Ral display names should be added to all examples. → Drop (error)

example_dynamic_task_mapping.py


airflow-core/docs/authoring-and-scheduling/dynamic-task-mapping.rstDynamic task mappingNice example. But use one alternative from real life and drop the individual dag then (error)

example_dynamic_task_mapping_with_no_taskflow_operators.py


airflow-core/docs/authoring-and-scheduling/dynamic-task-mapping.rstDynamic task mapping

example_inlet_event_extra.py


-

Needs a proper example to showcase something useful. Unknown User (uranusjr) do you have a real world example to add to the storyline (question) Then rework on this together with outlet events. Then drop (error) this example as standaone file.

example_kubernetes_executor.py


providers/cncf/kubernetes/docs/kubernetes_executor.rst
Move to K8s Provider

example_latest_only_with_trigger.py


airflow-core/docs/core-concepts/dags.rst

I (= Unknown User (jscheffl)) do not understand the example as well as not the benefit of this Operator. Is there anybody who can provide a real life example of use? Else drop (error)

example_local_kubernetes_executor.py


-
Move to K8s Provider

example_nested_branch_dag.py


-
Not of any use standalone. Drop (error)

example_outlet_event_extra.py


-
Same like example_inlet_event_extra.py (question)

example_params_trigger_ui.py


airflow-core/docs/core-concepts/params.rst
Migrate to storyline, then drop the individual example (error)

example_params_ui_tutorial.py


airflow-core/docs/core-concepts/params.rst
If not all features can be transferred into storyline, keep this as individual example to
eb
be able to test all form elements.

example_passing_params_via_test_command.py


-
Migrate to storyline, then drop the individual example (error)

example_setup_teardown.py


-
Migrate to storyline, then drop the individual example (error)

example_setup_teardown_taskflow.py


-

example_simplest_dag.py


-
Merge with tutorial. No value standalone (error)

example_skip_dag.py


-
Should be integrated into storyline and be added as code reference in docs. Then Drop (error) this example

example_task_group_decorator.py


airflow-core/docs/core-concepts/dags.rstTask GroupMigrate to storyline, then drop the individual example (error)

example_task_group.py


-Task Group

example_time_delta_sensor_async.py


-
Merge with tutorial. No value standalone (error)

example_trigger_target_dag.py


-
Ups, this DAG was forgotten to be moved to standard provider (warning) → But anyway: Migrate to storyline, then drop the individual example (error)

example_workday_timetable.py


code copied, not inlined in airflow-core/docs/howto/timetable.rst
Use as a base for one of the examples in the storyline, then Drop (error)

example_xcomargs.py


-
Use as a base for one of the examples in the storyline, then Drop (error)

example_xcom.py


-
Use as a base for one of the examples in the storyline, then Drop (error)

tutorial_dag.py


-

Consolidate with tutorial.py

(warning) Note is referenced in some tests and needs to be replaces with the other tutorial

tutorial_objectstorage.py


airflow-core/docs/tutorial/objectstorage.rst
(question)

tutorial.py


airflow-core/docs/core-concepts/dag-run.rst

airflow-core/docs/tutorial/fundamentals.rst


Keep as starter tutorial, beautify using technical rules (tick)

tutorial_taskflow_api.py


airflow-core/docs/tutorial/taskflow.rst

tutorial_taskflow_api_virtualenv.py


-

Check if features can be added to storyline for technical completeness, then drop (error)

tutorial_taskflow_templates.py


-
Check if features can be added to storyline for technical completeness, then drop (error)

Standard - providers/standard/tests/system/standard

example_bash_decorator.py


providers/standard/docs/operators/bash.rst
Keep

example_bash_operator.py


providers/standard/docs/operators/bash.rst

airflow-core/docs/core-concepts/debug.rst

contributing-docs/quick-start-ide/contributors_quick_start_vscode.rst


Keep

example_branch_datetime_operator.py


providers/standard/docs/operators/datetime.rst
Keep, but need a better idea

example_branch_day_of_week_operator.py


providers/standard/docs/operators/datetime.rst

example_branch_operator_decorator.py


providers/standard/docs/operators/python.rst

example_branch_operator.py


providers/standard/docs/operators/python.rst

example_external_task_child_deferrable.py


-

example_external_task_marker_dag.py


providers/standard/docs/sensors/external_task_sensor.rst

example_external_task_parent_deferrable.py


providers/standard/docs/sensors/external_task_sensor.rst

example_latest_only.py


providers/standard/docs/operators/latest_only.rst

example_python_decorator.py


airflow-core/docs/tutorial/taskflow.rst

providers/standard/docs/operators/python.rst



example_python_operator.py


airflow-core/docs/best-practices.rst

providers/standard/docs/operators/python.rst



example_sensor_decorator.py


airflow-core/docs/tutorial/taskflow.rst

providers/standard/docs/sensors/python.rst



example_sensors.py


providers/standard/docs/sensors/bash.rst

providers/standard/docs/sensors/datetime.rst

providers/standard/docs/sensors/file.rst

providers/standard/docs/sensors/python.rst



example_short_circuit_decorator.py


providers/standard/docs/operators/python.rst

example_short_circuit_operator.py


providers/standard/docs/operators/python.rst

example_trigger_controller_dag.py


providers/standard/docs/operators/trigger_dag_run.rst
See example_trigger_target_dag.py

ArangoDB - providers/arangodb/src/airflow/providers/arangodb/example_dags

example_arangodb.py


providers/arangodb/docs/operators/index.rst

Oracle - providers/oracle/src/airflow/providers/oracle/example_dags

example_oracle.py


providers/oracle/docs/operators.rst

Edge - providers/edge3/src/airflow/providers/edge3/example_dags

integration_test.py


-

win_notepad.py


-

win_test.py


-


List of features to cover in new examples

...