DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
| Item / Description Describe the Airflow 3.0 Breaking Point that can not be achieved non-breaking in 2.x. Also try to sketch the "Value" it brings to user or product. | Describe the Pressure Please describe what the impact would be if we would not go with this, e.g. competitive/comparable products that carry this and Airflow has a gap because of existing 2.x concepts | Stakeholder backing Please add your name if you agree - so we see how many people share this demand | Open Questions to discuss Please post questions that need to be discussed for understanding of the raised point or clarifications (preventing too many comments, please directly here as text) |
|---|---|---|---|
DAG / Workflow Support for non-linear complexity. DAGs are only one-way. And findemantally this is because loops would call a task multiple times and you would need to keep the context and execution history all separate (see also AIP-64: Keep TaskInstance try history). But there is real demand to have support for "experimental approaches" calling for loops, e.g. attempt to train a network until the desired state is reached. For such cases a DAG is not the right thing, a workaround would be to have a long running task that calls a second DAG in a loop. | Support non-linear workflows which have experimental character, unable to model such as DAG. | ||
Inconsistencies in DAG run conf / params and persistence. Some values are overlapping, are re-interpreted from code changes during a DAG run. Idempotency for a DAG run not given, risk for inconsistencies. | DAGs might be running with a set of parameters which are defined on DAG and Task level. These parameters are over-ridden by a DAG run Therefore also as follow-up to bug #39904 and as a previous PR to clean-up |
What we must Keep from 2.x Approach
...