...

• Users are not protected against concurrent runs of the same logical date
• Suppose you backfill a range containing 10 runs with date X
  • What if you ask for "rerun failed"? How do you determine the state of that logical date?
  • Suppose they all failed.  Should we rerun all of them?  Only the one that started most recently?
    • What if there is more than one "latest" one? I.e. with same start date?
    • Suppose that their start dates differ by a few milliseconds, so there is no such ambiguity – do you really want your workflow to be driven by that randomness / race condition?
  • In the old Airflow semantics, backfilling a range would never result in more than one run per logical date.
• If a task is marked "depends on past", and there are 3 runs for the prior logical date, which run should we look at?
• If you try to retrieve xcom of the task from the prior run, which run should it choose?
• We've lost our view of data completeness.  Previously grid view functioned as data completeness view, because every partition's state was visible there.  So if you saw a red bar you knew that partition did not load properly.  But now if it was later re-run successfully, that red bar would not be telling the truth anymore.

Side problem: data intervals

Data intervals are not a specific target of this proposal.  But, in any of these solutions we have to think about what are the implications for data intervals and what to do about them.  And, in some of the solutions considered, the proposal is to not have data intervals defined in some scenarios.  Generally speaking, I think they are confusing and misleading and we would probably do well, as is planned for assets, to move away from them.

With that let's briefly discuss some of the  issues.

...

Data intervals were layered onto the existing (albeit implicit) partitioning concept represented by execution_date / logical_date.  They assume more about what the user is doing than they really need to.  So e.g. rather than saying this execution_date is simply the partition key for this run (whatever that may mean) they say this logical date represents an interval.  But what if a M-F dag really does process only the M-F data, i.e. really only deals witth with one date at a time, and doesn't care about weekends?

...

1. restore uniqueness and keep old dag run semantics
2. restore uniqueness but make logical_date nullable
   1. this would allow users to create arbitrary runs without worrying about logical date collisions, while keeping the old semantics for normally scheduled dags
3. add a partition-esque conceptadd a param to DAG object to allow users to optionally keep the old dag run semantics
4. leave it sorta fuzzy and ambiguous
5. distribute the responsibility for what backfill means etc

...

2. Restore uniqueness but make logical_date nullable and / or optional

To me this is an interesting approach.  We could largely a very interesting approach that on one hand, let's us keep the dag run semantics unchanged.  But allow users who want to trigger runs manually with no logical date to optionally set it null thereby avoiding any uniqueness constraint.

But there are multiple paths here.  The simplest one is, keep everything the same but allow manual triggering of runs with null logical date and then these runs have no data interval.  Another path is to add timetables for which logical_date is always null and there are never data intervals.

...

But it also makes it possible to just fully dispense with logical dates and data intervals for those dags which don't use them.

With this approach we sort of have our cake and eat it too.

2.a. timetables that don't use logical date

If you don't care about logical date or data intervals, then maybe indicate that with the timetable you use:

DAG(dag_id="blah", schedule=MyNoLogicalDateNonDataIntervalTimetable(...))

Or, if we want to let users continue to use a cron expression, we could add a DAG level param.

DAG(dag_id="blah", schedule="0 0 * * *", logical_dates=False)

2.b. don't add timetables with no logical date, but let user manually trigger with no such date

A more limited version of this one, nullable logical date, is that we don't actually add timetables that don't use logical date, but we allow users to trigger runs with no logical date.  These would be unaffected by the uniqueness constraint on logical date.

Technical note

Null values are allowed with uniqueness constraint.  So the unique constraint on logical date in dag run is not a problem here.

Other note

I think with option 2, it might make sense to introduce another date field, run_date  which would be populated for all runs.  It would be the date after which the dag run should be free to be scheduled.  This word is generic enough to work for both schedule-driven dags and event-driven dags.  It would always mean the earliest time that the dag should run.  For manually triggered e.g. it would be "right now".  When the scheduler is not overloaded, it should always be close to start_time . 

3. Add a partition concept, or a partition-esque concept

We could allow users to declare whether the dag should use old semantics or new semantics.

...

With partition=None, no partition records would be created in the dag_partition table.  No uniqueness would be enforced w.r.t. logical date.  

...

Other modes

We have explored adding a newer, preferred mode of partition-driven scheduling.  With this approach, (1) there would be no data intervals (2) logical date would always align with the "run at" date.

...

...

other configurability to the partition paramater.  E.g. something to allow us to not have data intervals, or to change when the partition is run – e.g. on the partition date and not on the start of next partition.

Aside: I think making dag partitioning an explicit thing in airflow would be an overall good thing, beyond the whole uniqueness issue.  It clarifies what exactly is going on, i.e. why execution X does not run until the next partition – cus it would not be complete otherwise.  It also makes it possible for the user to say, every day rerun the last N partitions.  This is essentially a partition-driven workflow with a configurable lookback.

4. Leave it sorta fuzzy and ambiguous

This is essentially the do nothing approach.  We more or less look the other way and don't worry about it.  Leave it how it is.  Backfill will look at the latest (by start_date) run for a given logical date to determine state. Similar elsewhere.  No protection against concurrent runs.  No well-defined view of data completeness.

5. Distribute the logic elsewhere

...

Notes on conversion to and from partition-driven

One thing to consider is, what happens when user changes the partition scheme or schedule.  People will obviously do this, either intentionally or on accident. 

One option is to only track partition fulfillment when the dag is in partition mode.  Then, if dag is later changed to non-partition mode, then we would just leave the old partition records there, and ignore them.  Any runs created while dag not in partition mode would not update the partition records.  If the dag is switched back to partition-driven, then the partition store would not have knowledge of runs that happened when the dag ran in non-partition mode.  If user wants to migrate the history they would have to do so manually or via the api.

Another option would be that when we convert modes, we could try to infer the current partition state.

Another option would be to disallow conversion.

4. Do nothing

Here's where things stand right now

• there's no logical date uniqueness constraint on dag_run
• users could end up with multiple runs for a given logical date if user triggers manually, via api, or through trigger dag run operator
• in part because of this, backfill currently will always add new dag runs and never reuse existing dag runs
• as a result, use of backfill will always result in multiple runs per logical date
• when there are multiple runs for a given logical date, they will all be shown on the UI.  e.g. a red run might falsely indicate partition non-fulfillment
• there is nothing to prevent concurrent runs of a given logical date.  e.g. if there's a backfill running, and then a user clears an old failed dag run in the same range, or if to users or API calls trigger a dag run at around the same time, nothing will prevent that.
• what backfill does now is look at the state of the latest dag run (by start date) to ascertain state of the partition.  and for any date for which there's currently a running dag run, backfill will not create a dag run for that date.

5. Provide configurability at backfill run time or more "depends on past" options

Here's one phrasing of the question:

> Why is this not something that you choose when doing a backfill or when setting depends on past?

Here are my concerns with it:

• these options would not maintain these aspects of old behavior
  • concurrent runs of a one logical date a.k.a. partition are disallowed
  • grid view is a partition-fulfillment view
• "choose when doing a backfill"
  • what you're talking about i think is, let user choose, at the time of backfill, whether to rerun all runs or, say, only the latest.  but "the latest" is a fraught concept as a partition inference strategy.  because (1) there could be a tie and (2) even if there isn't a tie, if say user created 3 runs at same time, it would be subject to randomness which one is the lucky one that we use. 

Proposal

I would be in favor of 1, 2 , or 3.  Mild preference for 3

Additionally, if we do something like 3, we should make non-partition-driven dags the default, with no data interval.