Page History

This is a working document to clarify proposal option number 2 originally presented in AIP-83 amendment to support classic Airflow authoring style.

We’re on a tight timeline to make a decision and finish implementation for 3.0. PLEASE LEAVE YOUR OPINIONS AS SOON AS POSSIBLE. We should reach a decision on everything before 19th January.

Proposal: restore uniqueness but make logical_date nullable and / or optional

...

Question 1. For manual runs, should logical date be null by default or only optionally?

1.a. user manually triggers from UI

1.b. triggering run via REST API

1.c. TriggerDagRunOperator (and maybe other Python interface?)

1.d airflow dags trigger

Question 2. For asset-triggered runs, should logical date be populated?

...

There is still a unique constraint on run_id.  Currently it is generated deterministically from logical date plus run type.  What should we do for runs with no logical date?  How can we support the use case with many runs triggered in close succession such that perhaps even utcnow()  would collide.  What shall we do?

Question 7: How do we deal with APIs that select a logical date range?

For example, xcom_pull has include_prior_dates. Taskinstance.clear allows clearing all tis in a date range. Those use logical dates. What do runs with null logical date fit into these?

Resolution of questions

Here will be where I work out the consensus view based on feedback.  This will be amended as we go.

Question 1a: manual runs in UI

• In legacy UI, behavior Manual runs (question 1)behavior in legacy UI and connexion API will be unchanged
• in new REST API, when triggering a run logical date will be required field and may be null
in new UI, when In new UI, when triggering a run, it will be default None but you can select a date with date pickerif you try to trigger a date that already exists, you will not be able to.  you can use backfill to rerun a historical date

Question 1b: triggering through REST API

• legacy connexion API will be removed and will not be changed as part of this
• in new API, when triggering a run logical date will be required field, and user may choose NULL

Question 1c: TriggerDagRunOperator

• TriggerDagRunOperator will have a default of None for logical date, but user may supply

Question 1d: CLI dag run trigger

• CLI dag run trigger will be default none, but you can add supply a logical date if you like.

Question 2: Asset triggered dag runs

• Asset-triggered dag runs will not have a logical date or a data interval
• Dags which are a schedule + asset triggered will have the same behavior as just asset-triggered (i.e. no logical date or data interval)
• For asset-triggered dags, using logical date in a template will result in key error

Question 3: How to sort runs in UI

• we add a new date field run_after which means, the earliest time after which this dag run may be scheduled
• Airflow will sort by run_after  by default

Question 4a: data intervals for asset triggered

• Asset-triggered dags will have neither logical date nor data_interval_start / data_interval_end

Question 4b: data intervals for manual runs

• For manual runs, if the dag is schedule-driven, then there there will be a data interval if logical date is not null
• for manual runs, if the dag is asset-driven, there will be no data interval

Question 5: for runs with no logical date, what happens if accessed from template / context

• key error

Question 6 run_id logic for null logical date: 

• when null logical date, use run_after + random string

Question 7 How do we deal with APIs that select a logical date range?

• For xcom pull, setting `include_prior_dates` to True will have no effect.  The reason why is, prior refers to prior logical date, and we do not propose to change this.
• For clear, same thing – adding those filters will have no effect, for the same reason.
• When you use a range, runs with null logical date will never be included in the result.
• In other cases, when there existing filters that apply to logical date, we don't plan to change it.