Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • there's no logical date uniqueness constraint on dag_run
  • users could end up with multiple runs for a given logical date if user triggers manually, via api, or through trigger dag run operator
  • in part because of this, backfill currently will always add new dag runs and never reuse existing dag runs
  • as a result, use of backfill will always result in multiple runs per logical date
  • when there are multiple runs for a given logical date, they will all be shown on the UI.  e.g. a red run might falsely indicate partition non-fulfillment
  • there is nothing to prevent concurrent runs of a given logical date.  e.g. if there's a backfill running, and then a user clears an old failed dag run in the same range, or if to users or API calls trigger a dag run at around the same time, nothing will prevent that.
  • what backfill does now is look at the state of the latest dag run (by start date) to ascertain state of the partition.  and for any date for which there's currently a running dag run, backfill will not create a dag run for that date.

5. Provide configurability at backfill run time or more "depends on past" options

...

Here are my concerns with it:

  • these options would not maintain these aspects of old behavior
    • concurrent runs of a one logical date a.k.a. partition are disallowed
    • grid view is a partition-fulfillment view
  • "choose when doing a backfill"
    • what you're talking about i think is, let user choose, at the time of backfill, whether to rerun all runs or, say, only the latest.  but "the latest" is a fraught concept as a partition inference strategy.  because (1) there could be a tie and (2) even if there isn't a tie, if say user created 3 runs at same time, it would be subject to randomness which one is the lucky one that we use. 

Proposal

I would be in favor of 2 or 3.

...