You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
Version 1
Current »
Date
Attendees
Goals
Discussion items
| Item | Who | Notes |
|---|
| Heading towards Kube as default deployment mechanism? | | - Not everyone has a k8s cluster - so maybe it’s worth checking if celery executor can support autoscaling.
|
| Auto-scaling celery worker as a "stop-gap"? | Jarek | - Astronomer tried HA scaling for celery but it does not work very well - no memory why
- May be worth spending a day or two to try again
|
| Proposed as AIP-25 |
| - Hacky proof-of-concept PR running locally (no Knative needed yet)
- Showed a 20x speed up execution time for dags/tasks
- Ash also has a less disruptive change to existing executors that shows a 8x speed up
- (Numbers not directly comparable as they were from different machines)
|
| KNative + k8s mixed approach |
| - Knative doesn't allow "long running tasks" (defined per cluster, default for Google Cloud Functions is 30mins)
- user should know she’s scheduling long running task and then use k8s queue.
- We could possible re-use the existing
queue attribute on Tasks to choose this behaviour - This should be well documented and it’s a good idea to write down best practises / how to choose knative / k8s queue. Polidea might help with that
|
| Side point about scaling compose |
| |
| Performance testing of Airflow |
| - Polidea will be working on automating performance tests for Airflow and can handle various executors - including Knative executor
|
Action items
- Take a look at Daniel’s PR @Polidea
- Schedule sync on Knative executorDaniel Imberman
- Take a look at Celery autoscaling options @Polidea