DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Another option would be to disallow conversion.
4.
...
This is essentially the do nothing approach. We more or less look the other way and don't worry about it. Leave it how it is. Backfill will look at the latest (by start_date) run for a given logical date to determine state. Similar elsewhere. No protection against concurrent runs. No well-defined view of data completeness.
Backfill always only adds new rows to the dag run table. It never "clears" an existing row to be rerun.
5. Distribute the logic elsewhere
...
Do nothing
Here's where things stand right now
- there's no logical date uniqueness constraint on dag_run
- users could end up with multiple runs for a given logical date if user triggers manually, via api, or through trigger dag run operator
- in part because of this, backfill currently will always add new dag runs and never reuse existing dag runs
- as a result, use of backfill will always result in multiple runs per logical date
- when there are multiple runs for a given logical date, they will all be shown on the UI. e.g. a red run might falsely indicate partition non-fulfillment
- there is nothing to prevent concurrent runs of a given logical date. e.g. if there's a backfill running, and then a user clears an old failed dag run in the same range, or if to users or API calls trigger a dag run at around the same time, nothing will prevent that.
5. Provide configurability at backfill run time or more "depends on past" options
Here's one phrasing of the question:
> Why is this not something that you choose when doing a backfill or when setting depends on past?
Here are my concerns with it:
- these options would not maintain these aspects of old behavior
- concurrent runs of a one logical date a.k.a. partition are disallowed
- grid view is a partition-fulfillment view
- "choose when doing a backfill"
- what you're talking about i think is, let user choose, at the time of backfill, whether to rerun all runs or, say, only the latest. but "the latest" is a fraught concept as a partition inference strategy. because (1) there could be a tie and (2) even if there isn't a tie, if say user created 3 runs at same time, it would be subject to randomness which one is the lucky one that we use.
Proposal
I would be in favor of 2 or 3.
...