Page History

...

• Fundamentally improving the scheduler
  • Improving the scheduler by making dag runs more coherent
    • DescriptionProposal and design
    • AIRFLOW-20
    • PR 

• Ensure correct handling of Skipped tasks - Champion : Sid

  • depends_on_past=True and Skipped : 1155 (Fixed)
  • DAGs or Tasks that we would like to manually skip as described in 262
  • DAGs that specific only_run_latest as in 59
    • In these cases, we want to skip DAG Runs except for the latest. Unfortunately, if we skip all but the latest, the latest will not run if depends_on_past=True

• Concurrency Limits Not Honored : max active, concurrency, pool

  • Some overlap with the following item!
    • https://github.com/airbnb/airflow/issues/1057 (pool over-subscription)
    • https://github.com/airbnb/airflow/issues/1085 (concurrency not honored)

• Backfill offers a parallel code path to scheduling - Champion : Jeremiah & Sid

  • As such, it may not honor concurrent parameters
    • https://github.com/airbnb/airflow/issues/1057 (pool over-subscription)
    • https://github.com/airbnb/airflow/issues/1085 (concurrency not honored)
  • Backfill should add DagRuns and defer to the scheduler.
    • This will avoid incomplete duplication of logic between Backfill and Scheduler.
    • This will also expose all runs in the DAGRuns view, not just the ones created/used by the scheduler

• Only Run Latest - Champion : Sid

  • For cases where we need to only run the latest in a series of task instance runs and mark the others as skipped. For example, we may have job to execute a DB snapshot every day. If the DAG is paused for 5 days and then unpaused, we don’t want to run all 5, just the latest. With this feature, we will provide “cron” functionality for task scheduling that is not related to ETL

• Backfill Oddity : needs one successful run!

...