DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- Users are not protected against concurrent runs of the same logical date
- Suppose you backfill a range containing 10 runs with date X
- What if you ask for "rerun failed"? How do you determine the state of that logical date?
- Suppose they all failed. Should we rerun all of them? Only the one that started most recently?
- What if there is more than one "latest" one? I.e. with same start date?
- Suppose that their start dates differ by a few milliseconds, so there is no such ambiguity – do you really want your workflow to be driven by that randomness / race condition?
- In the old Airflow semantics, backfilling a range would never result in more than one run per logical date.
- If a task is marked "depends on past", and there are 3 runs for the prior logical date, which run should we look at?
- If you try to retrieve xcom of the task from the prior run, which run should it choose?
- We've lost our view of data completeness. Previously grid view functioned as data completeness view, because every partition's state was visible there. So if you saw a red bar you knew that partition did not load properly. But now if it was later re-run successfully, that red bar would not be telling the truth anymore.
Side problem: data intervals
...