Page History

...

Summary

• ...
• Discussion topics - Feedback from Airflow 2.11→3.1.7 Migration at Bosch (Marco Küttelwesch / Unknown User (jscheffl))

  • Issues we saw prior migration

    • Previous integration we had with direct DB access - workaround is to have a parallel DB connection until better solution(s)

      • Context does not provide status of other tasks in the Dag for Error handling/summary

      • Triggering and monitoring other dags not possible w/o DB connection (needed for scaling)

      • Setting of Dag and Task Notes not possible  --> Contribution to Task SDK was not (yet) accepted

      • Automation which adjust Pools and Queues for dedicated machine routing w/o DB access

      • Custom Dag level Archiving w/o DB Access

    • Suprises by breaking APIs that required some migration helpers:

      • XCom for SkipMixIn different semantic and key name changed in BranchOperator

      • XCom does not accept pathlib.Path objects anymore (==PosixPath())

      • Some other dataclass serialization problem in XCom

    • Python code in VenvOperator does not see code from Dag folder anymore --> Fix by adding PYTHONPATH manually

    • Triggerer did not see custom code from deferred sensor anymore, needed to tweak via adding to PYTHONPATH (Kafka Message Filter)

    • Airflow Python API breaking changes, needed to create a tool that can support AF2+3 backends to allow changing backend w/o integrated app change in parallel

    • Issue with XCom retrieval of Mapped task with mapping==1, not returning a list (Fix PR is open in parallel)

    • Our patched and beloved "custom footer" which showed Git sync status was not possible to be ported, alternative still in work via Plugin

    • Markdown in Trigger form UI was interpreted differently compared to AF2, UI glitches needed correction

    • Needed to change a lot of Dags interpreting "execution date" -> "logical date"

    • Trigger form failed with None as default in an enum with optional field (fix in AF3.1.8 coming)

    • Fab provider Oauth integration issues, mixing user contexts (Fixed in current provider)

    • Cookie of session across application, mixing different user contexts on multi site deployments (Fixed in 3.1.8)

    • Needed to re-write a MS EventGrid plugin receiving push events (Flask->FastAPI App)

    • Task log URL generation needed to change

    • Link to listing of "All failed tasks of all runs of a Dag with a Run-ID prefix" not existing anymore

  • Issues we faced during or post migration

    • Dag processor failed because of "unstable rendered Dags" --> Version increase --> Full table scan in TI --> Flaky Dags --> Fixed by DB index on dag_version_id (Upstream contribution pending)

    • Scheduler were not healthy because of unstable Dag (after fixing Dag parser) required consistency reconciliation when running in large queue and rotating with every Dag parse, all Schedulers locked DB for re-conciliation and workers and API server were locked out of access, heartbeat failed (Upstream contribution pending)

      • Hard to see which background task and including which Dag caused the problem until debug code injected, no metric on scheduler details!

    • We see some Postgres DB Locks on task_instances (~5min,  root unclear until now) - tasks fail as heartbeat every 5s, retry after 5s, API server running out of DB connections, kills Pods in liveness as DB connection pool exhausted by DDOS from workers

    • JWT token timeout of 10min too short if tasks are staying in Celery queue, way too short, such failures produce empty logs, hard to find

    • Previously limited Dags via "max_active_tasks" were running almost uncontrolled, lot of complaints by affected groups suffering from capacity

    • Happily running tasks were killed by re-assignment of Tasks in Celery after 1h (w/o any config change in Celery!), need to disable LATE_ACK

    • Bad Unicode chars killed jobs on DockerOperator --> Fix contributed

    • Bad Unicode chars kill KPO jobs on triggerer --> PR in review in parallel (was closed because of AI Slop 2 times…)

    • Celery worker needing 50% more RAM for same concurrency (8GB->12GB for 16 tasks), generated a couple of OOMs

    • Locally patch to count deferred task instances into running (PR in Airflow open in discussion in parallel)

    • Redirect after login broken if user not authenticated (Fix in 3.1.8)

    • (new) Validation that run id prefix matches run-type

    •  We still see DB Locks lasting for 5+ minutes in production and seek for root cause

  • Open items we see as gaps post migration

    • UI: Ability to batch clear/mark as failed Dag runs (Target: Workaround via script)

    • UI: Presentation of Task Notes is hard, proposal was rejected... planning to pitch an AIP soon as a new feature for "Dag/Task Summaries for Humans"

    • UI: Scrolling in large logs is as bad as in Github - initial loading faster but scrolling through in total is worse than in AF2. Also as partially loaded no browser search possible --> Search option in panel?

    • UI (requested but coming in 3.2.0): : Grid filter left panel by run status (running, failed, success at least)

    • UI (requested but coming in 3.2.0): Better filtering in Dag runs / Task Instances

    • UI (requested but coming in 3.2.0): Re-run Dag with previous config

    • UI nit: Can not click mapped task that is scheduled as link is missing

    • UI nit bug: Failure to load right panel when clicking on a task group left in grid

    • UI nit: Missing to color code test/development instances - hope this is coming with 3.2.0 theming...

    • UI nit: Admin / Pool view is bad if 20+ Pools listed, bad overview

    • UI nit: complaints that failed and skipped task color tooo similar --> will adjust locally with custom theming in 3.2.0?

    • UI nit: a lot of user bookmarks broken/404 (e.g. http://host/airflow/dags/<my_dag>/grid)

  • Compensated by very positive feedback in regards to

    • Now it is directly possible to see who triggered a Dag

    • URLs to logs finally working (was a pain with Grid in AF2)

    • UI: Much more modern than AF2!

    • UI: Translations are welcome!

    • HITL opens new options for approving runs which has a actual demand ATM

    • All users enjoy dark mode!