Status
Motivation
For long-running tasks (e.g. jobs on cloud providers), operators and sensors often poll for task status and/or task outputs to determine the success or failure of a task. These task monitoring processes are often blocking operations that can incur various problems, including:
- blocking wait operations that needlessly occupy a worker
- limited concurrency on local executor
- wasted resources on distributed executors
- db-sync operations for rescheduling
- passing XCom task-ID data
To enable the use of various non-blocking async options for hooks, sensors and operators, an async ecosystem is required and especially an async event loop (executor), task scheduler, and associated asyncio libraries for db-connections etc. Along with that, various ways to enhance existing blocking code with async options is required. One possibility to explore is to first add an option for an AsyncExecutor that can be used like the LocalExecutor.
Considerations
Is there anything special to consider about this AIP? Downsides? Difficultly in implementation or rollout etc?
What change do you propose to make?
- TBD - AIP WIP
What problem does it solve?
- TBD - AIP WIP
Why is it needed?
- TBD - AIP WIP
Are there any downsides to this change?
- TBD - AIP WIP
Which users are affected by the change?
- TBD - AIP WIP
How are users affected by the change? (e.g. DB upgrade required?)
- TBD - AIP WIP
Other considerations?
- TBD - AIP WIP
What defines this AIP as "done"?
- TBD - AIP WIP