...
Ash Berlin-Taylor @ Astronomer
- Kaxil Naik @Astronomer
- Daniel Imberman @Astronomer
- Greg Neiheisel @Astronomer
- Tomasz Kaminski @Polidea
- Kamil Bregula @Polidea
- Jarek Potiuk @Polidea
Goals
Discussion items
Who | Notes |
---|
Heading towards Kube as default deployment mechanism? | - Not everyone has a k8s cluster - so maybe it’s worth checking if celery executor can support autoscaling.
|
Auto-scaling celery worker as a "stop-gap"? | Jarek | |
Need to find more scalable ways to run events in kubernetes | - looking into horizontal scaling of Celery worker pods?
- Astronomer tried Horiz. Pod Auto-
|
Astronomer tried HA - scaling for celery but it does not work very well - no memory why
- May be worth spending a day or two to try again
|
Proposed as AIP-25 | - Hacky proof-of-concept PR running locally (no Knative needed yet)
- Showed a 20x speed up execution time for dags/tasks
- Ash also has a less disruptive change to existing executors that shows a 8x speed up
- (Numbers not directly comparable as they were from different machines)
|
KNative Knative + k8s mixed approach | - Knative doesn't allow "long running tasks" (defined per cluster, default for Google Cloud Functions is 30mins)
- user should know she’s scheduling long running task and then use k8s queue.
- Can potentially merge into KubernetesExecutor, or just have a “branching executor” that can send to queues for either knative or kubernetes executor
- We could possible re-use the existing
queue attribute on Tasks to choose this behaviour - This should be well documented and it’s a good idea to write down best practises / how to choose knative / k8s queue. Polidea might help with that
|
Side point about scaling compose | |
Performance testing of Airflow | - Polidea will be working on automating performance tests for Airflow and can handle various executors - including Knative executor
|
Action items
- Take a look at Daniel’s PR @Polidea
- Schedule sync on Knative executorDaniel Imberman
- Take a look at Celery autoscaling options @Polidea