Page History

...

2. Restore uniqueness but make logical_date nullable and / or optional

To me this is an a very interesting approach.  We could largely keep the dag run semantics unchanged.  But allow users who want to trigger runs manually with no logical date to optionally set it null thereby avoiding any uniqueness constraint.

But there are multiple paths here.  The simplest one is, keep everything the same but allow manual triggering of runs with null logical date and then these runs have no data interval.  Another path is to add timetables for which logical_date is always null and there are never data intervals.

With this approach we sort of have our cake and eat it too.

If you don't care about logical date or data intervals, then maybe indicate that with the timetable you use:

DAG(dag_id="blah", schedule=MyNoLogicalDateNonDataIntervalTimetable(...))

Or, if we want to let users continue to use a cron expression, we could add a DAG level param.

...

DAG(dag_id="blah", schedule="0 0 * * *", logical_dates=False)

3. Add a partition concept, or a partition-esque concept

We could allow users to declare whether the dag should use old semantics or new semantics.

...