DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
With partition=None, no partition records would be created in the dag_partition table. No uniqueness would be enforced w.r.t. logical date.
...
Other modes
We have explored adding a newer, preferred mode of partition-driven scheduling. With this approach, (1) there would be no data intervals (2) logical date would always align with the "run at" date.
...
| language | py |
|---|---|
| theme | FadeToGrey |
...
other configurability to the partition paramater. E.g. something to allow us to not have data intervals, or to change when the partition is run – e.g. on the partition date and not on the start of next partition.
Aside: I think making dag partitioning an explicit thing in airflow would be an overall good thing, beyond the whole uniqueness issue. It clarifies what exactly is going on, i.e. why execution X does not run until the next partition – cus it would not be complete otherwise. It also makes it possible for the user to say, every day rerun the last N partitions. This is essentially a partition-driven workflow with a configurable lookback.
...