Page History

...

We could allow users to declare whether the dag should use old semantics or new semantics.

  This old vs new distinction is just shorthand; in reality we should not describe it as old vs new, since this implies a judgment about it, whereas I think we should recognize the old way as a valid way of designing pipelines, even as we make it just one option instead of the only option.

With this approach, one option would be we propose to add a partition  attr param to the dag object.  At this time, we would propose three available behaviors.

Full backcompat – legacy partition-driven semantics

E.g. we could declare partition="implicit" to say that the dag should have partitions defined by the schedule. 

...

With implicit partitioning, we keep the old Airflow semantics.  When the scheduler schedules a dag run, it creates a record in a dag_partition table.  This table would ensure uniqueness, within the scope of the dag, of partition keys.  Each partition key would be associated with only one dag run.  So this would make well defined the questions "what is the state for this partition?", and "what is the run prior to this run?".  Partition info would be made available in the execution context.  Partition date would equal logical date always.  Data interval would behave the same.

No partitioning

To Then, to disable partitioning for a dag, we could user can set partition=None.

DAG(dag_id="blah", partition=None)

With partition=None, no partition records would be created in the dag_partition table.  No uniqueness would be enforced w.r.t. logical date.  

New style partitioning – a variant under consideration

We have explored adding a newer, preferred mode of partition-driven scheduling.  With this approach, (1) there would be no data intervals (2) logical date would always align with the "run at" date.

DAG(dag_id="blah", partition="new")  # change this name

4. Leave it sorta fuzzy and ambiguous

...