Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Here will be where I work out the consensus view based on feedback.  This will be amended as we go.

Question 11a: manual runs in UI

  • behavior in In legacy UI and connexion API , behavior will be unchanged
  • in new REST API, when triggering a run logical date will be required field and may be null
  • in In new UI, when  when triggering a run, it will be default None but you can select a date with date picker
  • if you try to trigger a date that already exists, you will not be able to.  you can use backfill to rerun a historical date

Question 1b: triggering through REST API

  • legacy connexion API will be removed and will not be changed as part of this
  • in new API, when triggering a run logical date will be required field, and user may choose NULL

Question 1c: TriggerDagRunOperator

  • TriggerDagRunOperator TriggerDagRunOperator will have a default of None for logical date, but user may supply

Question 1d: CLI dag run trigger

  • CLI dag run trigger will be default none, but you can add supply a logical date if you like.

Question 2: Asset triggered dag runs

  • Asset-triggered dag runs will not have a logical date or a data interval
  • Dags which are a schedule + asset triggered will have the same behavior as just asset-triggered (i.e. no logical date or data interval)
  • For asset-triggered dags, using logical date in a template will result in key error

Question 3: How to sort runs in UI

  • we add a new date field run_after which means, the earliest time after which this dag run may be scheduled
  • Airflow will sort by run_after  by default

Question 4a: data intervals for asset triggered

  • Asset-triggered dags will have neither logical date nor data_interval_start / data_interval_end

Question 4b: data intervals for manual runs

  • For manual runs, if the dag is schedule-driven, then there there will be a data interval if logical date is not null
  • for manual runs, if the dag is asset-driven, there will be no data interval

Question 5: for runs with no logical date, what happens if accessed from template / context

  • key error

Question 6 run_id logic for null logical date: 

  • when null logical date, use run_after + random string

Question 7 How do we deal with APIs that select a logical date range?

  • For xcom pull, setting `include_prior_dates` to True will have no effect.  The reason why is, prior refers to prior logical date, and we do not propose to change this.
  • For clear, same thing – adding those filters will have no effect, for the same reason.
  • When you use a range, runs with null logical date will never be included in the result.
  • In other cases, when there existing filters that apply to logical date, we don't plan to change it.