| Item | Notes |
|---|
| Heading towards Kube as default deployment mechanism? | - Not everyone has a k8s cluster - so maybe it’s worth checking if celery executor can support autoscaling.
|
Need to find more scalable ways to run events in kubernetes | - looking into horizontal scaling of Celery worker pods?
- Astronomer tried Horiz. Pod Auto-scaling for celery but it does not work very well - no memory why
- May be worth spending a day or two to try again
|
| Proposed as AIP-25 | - Hacky proof-of-concept PR running locally (no Knative needed yet)
- Showed a 20x speed up execution time for dags/tasks
- Ash also has a less disruptive change to existing executors that shows a 8x speed up
- (Numbers not directly comparable as they were from different machines)
|
| Knative + k8s mixed approach | - Knative doesn't allow "long running tasks" (defined per cluster, default for Google Cloud Functions is 30mins)
- user should know she’s scheduling long running task and then use k8s queue.
- Can potentially merge into KubernetesExecutor, or just have a “branching executor” that can send to queues for either knative or kubernetes executor
- We could possible re-use the existing
queue attribute on Tasks to choose this behaviour - This should be well documented and it’s a good idea to write down best practises / how to choose knative / k8s queue. Polidea might help with that
|
| Side point about scaling compose | |
| Performance testing of Airflow | - Polidea will be working on automating performance tests for Airflow and can handle various executors - including Knative executor
|