THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
In favor of other scaling architectures we have decided to abandon the KnativeExecutor
In favor of other scaling architectures we have decided to abandon the KnativeExecutor
6 Comments
Jarek Potiuk
Daniel Imberman → I have one question. I think KNative executor is great however there are indeed limits on the very long running tasks. This is particularly important for managed KNative implementations (such s KNative at Google Cloud Platform). My question here is - will that be possible to somehow join both Kubernetes and Knative executor and have a possibility of running some tasks via Kubernetes executor and some via KNative one. Maybe that could be the best solution to make Airflow Kubernetes-Native.
Daniel Imberman
Jarek Potiuk That is actually something we've been actively discussing. One of the queues will be essentially a "KubernetesExecutor" queue. This queue will allow the tasks to be launched as isolated pods rather than via knative for this exact reason.
Sarah Johnson
May I ask what other scaling architectures the KnativeExecutor was abandoned in favor of?
Tomasz Urbaszek
Sarah Johnson CeleryExecutor + KEDA, more robust and easier to set up than Knative. Check Daniel Imberman blog post about it: https://www.astronomer.io/blog/the-keda-autoscaler/ AFAIK KEDA is going to be part of official Airflow Helm chart.
Ash Berlin-Taylor
Yes, KEDA support is in https://github.com/apache/airflow/pull/8777
Sarah Johnson
Thank you