• Created by Unknown User (dstandish), last updated on Jan 17, 2025 2 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

This is a working document to clarify proposal option number 2 originally presented in AIP-83 amendment to support classic Airflow authoring style.

We’re on a tight timeline to make a decision and finish implementation for 3.0. PLEASE LEAVE YOUR OPINIONS AS SOON AS POSSIBLE. We should reach a decision on everything before 19th January.

Proposal: restore uniqueness but make logical_date nullable and / or optional

Uncontroversial elements

  • restore the uniqueness constraint on logical date
  • make logical date nullable
  • we will not immediately add timetables that don't use logical date

Quetions of controversy

Question 1. For manual runs, should logical date be null by default or only optionally?

1.a. user manually triggers from UI

1.b. triggering run via REST API

1.c. TriggerDagRunOperator (and maybe other Python interface?)

1.d airflow dags trigger

Question 2. For asset-triggered runs, should logical date be populated?

And what about dags that are asset-or-schedule?  

Question 3. How to sort in the UI dag runs with no logical date?

Logical date is the default sort field currently.  Start date would be a natural candidate when no logical date.  But it is not populated initially.

One proposal is to add a run_after  field.  This concept already exists in timetables.  But it is not persisted to the dag run record.

Alternatively we could do coalesce(logical_date, start_date, updated_at)

Question 4: Should data_interval_start and data_interval_end be populated?

The question must be answered for both manual runs and asset-triggered.

Question 5: For runs without logical date (or data intervals) what should happen when a user has a template that references them?

Question 6: Implications for run_id

There is still a unique constraint on run_id.  Currently it is generated deterministically from logical date plus run type.  What should we do for runs with no logical date?  How can we support the use case with many runs triggered in close succession such that perhaps even utcnow()  would collide.  What shall we do?

Question 7: How do we deal with APIs that select a logical date range?

For example, xcom_pull has include_prior_dates. Taskinstance.clear allows clearing all tis in a date range. Those use logical dates. What do runs with null logical date fit into these?

Resolution of questions

Here will be where I work out the consensus view based on feedback.  This will be amended as we go.

Question 1: manual runs

  • behavior in legacy UI and connexion API will be unchanged
  • in new REST API, when triggering a run logical date will be required field and may be null
  • in new UI, when triggering a run, it will be default None but you can select a date with date picker
    • if you try to trigger a date that already exists, you will not be able to.  you can use backfill to rerun a historical date
  • TriggerDagRunOperator will have a default of None for logical date
  • CLI dag run trigger will be default none, but you can add a logical date if you like.

Question 2: Asset triggered

  • no logical date
  • data interval?
  • No labels