2024 Dev calls

4 June 2024

Agenda

Attendees

Summary

The following principles were agreed upon to drive Airflow 3 development:

• For the features that require breaking changes, ship Airflow 3 with the foundational code to allow for iterative development, optimizing for speed and a quicker feedback cycle.
  • Additional features that do not require breaking changes can be included in minor releases such as 3.1, 3.2, and beyond since we follow SemVer.
  • Discussion points:
    • Examples:
      • Add the hook points with a few references (fetcher for GCS, S3, Git) for DAG Versioning's Execution AIP (AIP-66)
      • Removing dependency on FAB in Core: The new plugin framework might support only a few functionalities, which are then built upon later.
      • For the multi-language AIP, start with only Python + one more language in AF 3.0 and then 3.1 and later minor versions can have more support for language like Typescript
    • Identify users who would be willing to give feedback during dev and beta snapshots. The Astronomer, MWAA, and GCC teams can help identify these.
• Ensure a smoother migration path between Airflow 2 and 3, particularly for DAG authors using the existing official Airflow providers.
  
  • Directionally the time required to update DAGs should be measured in hours, not days or months. 
  • Action Items:
    • Update the AIP template to ask for the level of effort (manual and automated) needed for the users to adapt to the breaking changes. This should include high-level details on what could go in the upgrade utilities. This will help the AIP authors consciously think through the migration efforts. (Unknown User (kaxilnaik) )
• For AIPs, be explicit about what's for AF 3.0 and what's for the next minor releases (3.1, 3.2, ..)
  • Action Items:
    • Complete the housekeeping of the AIPs (Unknown User (kaxilnaik) ):
      • Move the AIPs that we don't plan to work on in Abandoned state
      • Add labels if an AIP is for Airflow 2, 3.0 or >=3.1. These labels will be used via Macros to auto-populate the tables in Airflow Improvement Proposals, making it a good page for Roadmap items
• Build features that solidify Airflow as the modern Orchestrator that has state-of-the-art support for Data, AI & ML workloads.
  • This includes enhancing scalability, performance, and enterprise-level security, adhering to the principle of least privilege.
  • Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability
• Set up the codebase for the next 3-5 years.
  • Reducing matrix of supported combinations for reducing complexity in testing & development. E.g Remove MySQL support to reduce the test matrix
  • Simplifying codebase & standardize architecture (e.g consolidating serialization methods)
  • Remove deprecations
  • Consider optimizing development workflows (core airflow vs. provider, chart development)
• Simplify the Learning Curve for new Airflow users
  • Decrease time from running the install command to first DAG
  • Decrease the boilerplate code needed to run first dag/task
  • Action Items:
    • Write a first draft of a doc on the different personas of Airflow users & current state. This will help tailor the learning curve via docs & tutorials as well as tailor features towards that persona. (Unknown User (eladkal) )
• Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10. This should continue for a longer period of time after AF 3 release
  • The provider release will continue to happen independently of the core Airflow.
  • Discussion points:
    • After the AF 2.10 release (~Aug), the "main" branch will become Airflow 3, and the release manager will cherry-pick anything targeting Airflow 2 into the Airflow 2 release branch. 
    • The primary focus for AF 2 will shift to reliability. If certain features need to go in AF 2, they will be done by cherry-picking or opening PRs targeting the AF 2 branch.
• Target a shorter cycle to release Airflow 3
  • so that Airflow 2 branches for features don't diverge
  • users have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations
  • Discussion point:
    • This means the realistic target is March-April 2025. This is to allow users enough time to migrate to AF 3 and use it so they can submit CFPs around May-July 2025.
    • The focus for AF Summit 2025 would be on Migration stories & features of Airflow 3

The following Guidelines were agreed upon that helps decide if a feature should be in Airflow 3 or not.

• Alignment with Core Principles (mentioned above)
• Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found
• Community Demand and Feedback
• Impact on Scalability, Performance & Security 
• Backward Compatibility and Migration Effort
• Implementation Complexity and Maintenance
• For big features, discussion on AIPs & a successful vote on the dev mailing list 

Next steps

• Create AIPs for the features targeted for AF 3 in the next few days to start technical discussions. 
• Finalize the agenda for the next call

13 June 2024

Agenda

Attendees

Summary

• Drafting proposals & getting feedback
  • The team discussed the challenges of Apache Confluence access and considered using Google Docs for initial feedback before moving to Confluence.
  • The general consensus was that developers could utilize the tool of their choice to get initial feedback (on their high-level draft) as long as the final AIP is published on Airflow's Confluence space for AIPs.
  • If someone does not have an account on Apache Confluence to comment on existing proposals or create a new proposal, reach out on #airflow-3-dev Slack channel.
• Personas for the Airflow User
  • Elad prepared a document on Airflow Actors that the group discussed to get initial feedback. Several good suggestions were made, so it was decided to take the discussion offline/async and continue the discussion on that page. 
• Task Context draft AIP
  • Ash presented the initial draft of the Task Execution InterfaceContext AIP (formerly known as Task Context SDK). 
  • The draft AIP is now published at AIP-72 Task Execution Interface and ready for review, although some technical details still need to be worked out.
• Frequency of the Next Dev calls
  
  • The general consensus was to make the dev calls weekly for the next 2-3 weeks
• Timeline to release Airflow 3
  • Based on the discussions on the first dev call, we also agreed that we should target March-April 2025 to release Airflow 3.
  • We will announce the dates and discuss some of the feature sets with the community at the Airflow Summit in September this year.
• Action items
  • Docs needing reviews:
    • AIP-72 Task Execution Interface
    • Enhanced Data Awareness proposal
    • Airflow Actors
  • Updating AIP template & Housekeeping of AIPs & label additions (Unknown User (kaxilnaik) )

20 June 2024

Agenda

Attendees

Summary

• The team reviewed the existing status of AIPs and marked them for AF 3.0, 3.1+ and AF 2 as needed. Here is the summary:

  AIP	Decision
  AIP-1	Superseded by AIP-44 & AIP-67. Close it
  AIP-5	Superseded by AIP-66
  AIP-14	Mark as completed by AIP-58
  AIP-20	Superseded by AIP-66
  AIP-22	Superseded by AIP-72 due to Task Interface. If someone has time, it can be PRs, doesn't need AIP
  AIP-30	Airflow >= 3.1
  AIP-35	Abandoned
  AIP-38	Keep it for 3.0. Brent will update this. There was an Ask about expanding the Plugin migration strategy.
  AIP-44	Replaced by AIP-72 for AF 3, and we can ship what we have in main for 2.10 as "experimental" and stated to change for AF 3
  AIP-46	Replaced by AIP-72 and some parts of AIP-67
  AIP-49	Phase 1 is in 2.10 and Phase 2 will be in 3.1 and above unless Howard Yoo  wants to lead and implement it for AF 3.0
  AIP-51	Marking it for Airflow 2.10. Unknown User (onikolas) finishing up the last issue.
  AIP-55	Abandoned
  AIP-57	Airflow 3.0, Shubham owns it. There is a PR already with this comment contesting the proposed solution. We will review this again in few weeks
  AIP-59	Bartosz is working on it. Is OOO right now but will resume once back. Will be used in AF 2 & AF 3 to compare performance. Will target it for AF 2
  AIP-61	Marking it for Airflow 2.10. Unknown User (onikolas) to get it done in the next few weeks
  AIP-62	Marking it for Airflow 3.0 for now. But we will review it again in a couple of weeks; Unknown User (mobuchowski) owns it.
  AIP-64	Airflow 2.10
  AIP-65	Airflow 3.0
  AIP-66	Airflow 3.0
  AIP-67	Airflow 3.0
  AIP-68	Marking it for Airflow 3.0 for now, but it will be re-evaluated when Unknown User (jscheffl) creates a VOTE thread after discussing with Brent. Concern for keeping this for AF 2.10 was around adding a React Plugin interface that will need changing creating a migration pain.
  AIP-69	Tentatively marking it for 3.0. Unknown User (jscheffl) has a PR that needs review. The concern for keeping this for AF 2.10 even if most changes are in providers is that it builds on top of AIP-44 which will be experimental and replaced in AF 3. Will be reviewed in a few weeks
  AIP-70	Airflow 3.1+
  AIP-71	The general consensus was against this AIP and favouring the DAG bundles concept that is part of AIP-66.
  AIP-72	Airflow 3.0

• The team also decided to review the Airflow 3 Workstreams doc asynchronously. If the reviewers are interested in leading any workstream without an owner, they should add their names in the Owner column.
  • Other Action items:
    • Review SLA PR
    • Unknown User (jscheffl) to talk to Brent for AIP-68
    • Review PR from Jens for AIP-69 (Remote Executor)
    • Complete AIP-51 & AIP-61 for Airflow 2.10
    • Workstream owner should Start AIP or draft AIPs for items in Other Candidates items (under AF 3.0 section) in Airflow 3 Workstreams

27 June 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Unknown User (kaxilnaik) updated the AIP template to include the "Migration effort" section. If the template is used to create a new AIP doc (click "Create" → "Airflow Improvement Proposal" on the wiki) it should pre-populate sections. The access to editing the template itself is limited to PMC members & ASF Confluence Admin group. Need to retroactively add that section in existing AIPs planned for 3.0
    Screenshot of How to create a page using template
  • SLA PR: This is still pending review.
  • Complete AIP-51 & AIP-61 for Airflow 2.10
    • Marked AIP-51 as completed, and the pending UI issue will be covered in the UI refactor.
    • AIP-61: Unknown User (onikolas) is still targeting 2.10 for completion. PRs are raised & pending reviews: #40472 (Backfill part) & #40017 (Scheduler part).
• Discuss: Workstreams & workstream owners (Airflow 3 Workstreams)
  • Updates on the existing AIPs for the 3.0 list
    • AIP-66 (DAG Versioning—Execution side): Unknown User (jedcunningham) will have the draft by the next dev call.
    • AIP-72 (Task Execution SDK) : No updates this week. Unknown User (ash) aims to have the VOTE open for it by the next dev call.
    • AIP-38 (UI Modernization): Unknown User (bbovenzi) is focussing on Airflow 2.10 deliverables; post that, he will focus on replacing this AIP with a new Umbrella AIP. There was a consensus on the call that we should split this AIP into multiple AIPs (with an Umbrella AIP) as below:
      • React'ifying the UI & changing the UX required for a modern Webserver
      • FAB removal: The removal of Flask Appbuilder will have backend implications such as changes needed to Authentication (currently, it is done via FAB Auth backends.), changes to the Plugin interface, custom API endpoints etc. 
      • New Features like DAG Folders/DAG Groups
    • AIP-57 Refactor SLA Feature: The initial AIP was created with 2.x in mind, so it should be revised. Some things might be easier with 3.0, as we can make breaking changes. One of the contention points last time was around whether the SLA duration time should start from DAG/task start or should be based on absolute time, as reflected in this PR comment. Shubham Mehta and Unknown User (kaxilnaik) will reach out to Sungwon Yun to see if he is interested in being part of this effort.
    • AIP-65 (DAG Versioning—UI side): Before the next dev call, Unknown User (jedcunningham) will decide whether to keep this AIP or make it part of any existing DAG Versioning or UI Modernization AIP.
    • AIP-67 (Multi-team): Unknown User (potiuk) will revise this AIP based on all the discussions and new AIPs to present the overview in the next dev call. Shubham Mehta and his team are planning to co-own/contribute to this AIP with Jarek.
    • AIP-68 (Extending plugin interface): Unknown User (jscheffl) & Unknown User (bbovenzi) will create a POC PR to determine the direction to take. Jens will do it before the next dev call to then get the final feedback on 2.10.x vs 3.0. 
    • AIP-69 (Remote Executor): Unknown User (jscheffl) organized a call with interested contributors a day after the dev call to get feedback. A summary of it is posted here.
  • Updates on the "other candidates for 3.0" list:
    • Enhanced Data Awareness: Unknown User (uranusjr) and Constance Martineau created draft AIPs (AIP-74, AIP-75, AIP-76, AIP-77) for this workstream (under Umbrella AIP: AIP-73). These AIPs are marked for Airflow 3.0 but AIP-77 might move to Airflow 3.1, TBD for now. 
    • Scheduler-Managed Backills (aka Backfills at scale): Unknown User (dstandish) has an internal draft for it that will be published around next week.
    • Poll external Datasets to have event-based DAG scheduling: Vincent BECK volunteered to own this epic for 3.0. He and Shubham Mehta plan to publish a draft AIP in the next 2 weeks.
    • Unknown User (ferruzzi) has volunteered to own the "Inspect & Simplify Airflow Configurations" & "Inspect & Revamp the cardinality of Metrics" streams.
    • Synchronous DAG Execution: No update this week but targeting a proposal after the next dev call
    • Make Execution Date non-unique for a DAG: No update for now; TP is focussing on the Data Awareness AIPs.
    • Scheduler Performance Improvements: Some areas in the scheduler have been identified for improvements. If anyone has any specific areas of the Scheduler loop that they would like to own, please add it to the list on Airflow 3 Workstreams. The performance goals would be tied to the specific area of optimization rather than the scheduler as a whole.
    • Consider Developing Airflow and Providers separately. In the coming weeks, Unknown User (kaxilnaik) will draft a pro-cons list in a document to send to the mailing list around this epic.
    • Respect permissions in CLI: Buğra Öztürk has volunteered to own this stream and, ideally, will start working on a proposal next.
    • Improve security of Airflow Supply Chain: Once things are more defined, Unknown User (potiuk) will send a document about this stream of work describing its impact on dependencies in the coming month. 
    • Observability of Callbacks on UI: Some of this might be covered in AIP-72 & AIP-69 but we will keep this item in the table so we don't forget about it.
    • Remove StatsD and replace it with Prometheus as a first-class citizen: We agreed that we should replace StatsD with OpenTelemetry and not Prometheus. It needs an owner who could figure out the impact of breaking changes since StatsD metrics are utilized by all users, including Airflow service providers, to monitor and alert on the health of Airflow deployments.
    • Overhaul Operator Templating behaviour: Unknown User (uranusjr) mentioned he would be happy to write the design document & asked for someone to work on the implementation. Shahar Epstein expressed his interest in working on this effort. 
    • Shubham Mehta expressed interest in working on a native DAG Factory that is currently on the 3.1 list. We would keep this for 3.1 for now and reevaluate based on the progress of other epics.
    • There have been no noteworthy updates on any other streams. Unknown User (kaxilnaik) will create GitHub issues for items in that table with no owners and post them on the mailing list to see if anyone is interested in leading it. If there are no takers by the end of August, they will be moved to the 3.1+ list.
• The next dev call will be on 11 Jul 2024.

11 July 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • AIP-61 (Hybrid Executor): On track for Airflow 2.10.
  • Unknown User (kaxilnaik)  deferred Creating GitHub issues for items in the Workstream table with no owners to next week.
• Check-in: Workstreams & workstream owners (Airflow 3 Workstreams)
  • AIP-68 (Extending plugin interface): This is finalized for 3.0 instead of 2.10
  • AIP-38 (UI Modernization): Unknown User (bbovenzi) presented the progress of AIP-38 in the Airflow 2.3, 2.4, 2.7 & 2.8. An update on what's pending has been added to the AIP-38 page.
    • A new AIP was split from AIP-38 to replace FAB: AIP-79.
    • The new Plugin interface for UI will be defined by AIP-68
    • Unknown User (bbovenzi)  & Unknown User (jscheffl) : Could either of you add what you presented for AIP-68 in the AIP doc, around breaking changes to existing plugins & CLI to initialize a starter project?
    • Risk / Action Item: There were discussions around the impact on Auth due to the removal of FAB. KeyCloak Auth Manager was mentioned as a potential replacement for Enterprise installation and using FAB Airflow provider for local development. Fab auth manager via FAB provider might be a fallback until we get the Keycloak implementation. We should test this to make sure that it doesn’t become a blocker and that FAB provider is a reliable fallback. This should be noted in AIP-79.
  • AIP-67 (Multi-team): Unknown User (potiuk) presented the revised AIP. The AIP has been updated, and Jarek has answered the questions that were asked in the chat in this mailing list thread.
    • As a follow-up to the call, Jarek also created a mailing list thread about whether to finish AIP-44 for AF 2 or not.
  • AIP-62 (Lineage from Hooks): Unknown User (mobuchowski) presented current progress and is targeting Airflow 2.10 for its completion from the Airflow core side, the rest will be in Providers.
  • AIP-57 Refactor SLA Feature: Kaxil & Shubham both heard back from Sungwon Yun. Unfortunately, he doesn't have the capacity to work on it, but he is going to meet Shubham Mehta and the team for knowledge transfer.
  • AIP-65 (DAG Versioning—UI side): Unknown User (jedcunningham) decided to keep this AIP as is instead of making it part of the UI Modernization AIP.
  • AIP-66 (DAG Versioning—Execution side): Unknown User (jedcunningham) marked it as ready for review. The email thread is here.
  • AIP-69 (Remote Executor): Unknown User (jscheffl) wants to start voting, but it is dependent on AIP-44, which is discussed separately here.
  • AIP-72 (Task Execution SDK): VOTE has succeeded as of July 16.
  • AIP-73 (Data Awareness):  Unknown User (uranusjr) & Constance Martineau plan to remove AIP-74 (Introducing Data Assets) & AIP-75 (New Asset-Centric Syntax) from the draft in a week and timelines for the rest to follow.
  • AIP-78 (Scheduler-managed backfill) was published by Unknown User (dstandish) and is under-going discussion.
  • "other candidates for 3.0" list:
    • Poll external Datasets to have event-based DAG scheduling:  – This will become an AIP, Vincent BECK / Shubham Mehta working on the draft AIP.
    • Respect permissions in CLI: Buğra Öztürk is working on the first draft, timelines TBD.
• The group collectively agreed that all the big AIPs targeting Airflow 3.0 should at least have the vote started before August 1. Anything after that does not have a realistic chance of completion. This is to not rush the AIPs but to prioritize the key items, especially the ones that contain breaking changes, get them vetted out properly, and make them better for the initial MVP launch.

25 July 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Unknown User (kaxilnaik) created GitHub issues for workstream items with no owners; they have also been added as sub-items in this meta GitHub issue that tracks the 3.0 release. We had already had some interest from community members on some of these items, check individual issues for it.
• Discussions:
  • Branching Strategy:
    • As soon as 2.10.0rc1 is created, the Airflow main branch will become the Airflow 3 branch on 09 Aug 2024 .
    • Providers will continue to be developed on the main branch since we have the framework to run provider tests against older Airflow versions.
    • Bugfixes: If there is a PR with a bugfix to main, the committers should ask the PR author to raise a PR against v2-10-test  branch too. The team agreed to create a mailing list thread to discuss different approaches offline. The thread has been initiated here.
  • Which Python version should we support in Airflow 3?
    • The consensus was to keep things as-is i.e following the current Python-support policy where we would remove the support for a Python version when it is EOL.
  • Airflow 2 support policy including scope (feature vs bug fixes + security only) & support period
    • We might release 2.11 and mark it as a bridge release to add compatibility shims or deprecation warnings similar to 1.10.15. But we will decide on it as and when such a need arises.
    • For providers, we will need to increase the support window for the minimum Airflow version from the current 6 months. We will continue Airflow 2 provider support for a year or longer, exact period TBD. We will revisit this again in the coming months. If we cut 2.11 (or whatever the final 2.x version is), then after 6-12 months we could increase the minimum Airflow to 2.11. There were ideas around using a "compat" provider that might help with AF 2 and AF 3 stuff. This will be another thing we will discuss async at a later time to keep current focus on scope & AIPs.
• Workstream statuses for ones that does not have a vote started:
  
  • AIP-57 Refactor SLA Feature (Shubham Mehta):
    • It was agreed that for now, we will mark it for removal in Airflow 3.0 and adding it back in 3.1. We will assess again in the coming dev calls if something changes. Elad has also offered to help if needed.
  • AIP-67 Multi-team deployment of Airflow components (Jarek Potiuk ):
    • Jarek is waiting to hear back on his answers to some of the doubts raised in the discussion thread and AIP. This might also be moved to Airflow 3.1 since it depends on AIP-72 and AIP-66 and the implementation hasn't been started on those yet.
  • AIP-68 Extended Plugin Interface for React Views (Unknown User (jscheffl) / Unknown User (bbovenzi) ):
    • As of today, the vote has been started on the mailing list (link). During the meeting, a few things were discussed, including potentially renaming UI plugins to UI Extensions as the "Plugin" word has been overloaded. The other suggestion was to rename the AIP title to "New Interface" instead of "Extending the interface."
  • AIP-76 Asset Partitions (Unknown User (uranusjr)):
    • TP will send this out for VOTE in the next couple of days.
  • AIP-79: Remove Flask AppBuilder as Core dependency (Unknown User (bbovenzi) ):
    • Two big questions pending:
      1. Replacement for FAB Auth manager
      2. How does adding a FAB as Provider work? Can it allow users to make older plugins work?
    • Jed has been working on a POC on it. He is on vacation this week, but based on initial investigations, he was positive about the progress. We will check it back in the next dev call when he is back.
    • The pending item is also to update the AIP to add the decoupling of Providers's Connections metadata & form from FAB.
  • AIP-80 Explicit Template Fields in Operator Arguments (Unknown User (uranusjr) ):
    • VOTE has been kicked off here.
  • Poll external Datasets to have event-based DAG scheduling (Vincent BECK / Shubham Mehta): Draft AIP has been created as mentioned in this mailing list thread.
  • Respect permissions in CLI: Buğra Öztürk 
    • It now has an AIP-81 Enhanced Security in CLI via Integration of API and is awaiting feedback.
  • Returning DAG Results / Synchronous DAG Execution: Unknown User (vikramkoka) 
    • The first proposal (Make Execution DAG non-unique) from the original doc is separated out into a different workstream led by TP.
    • Language support has been excluded from the scope for Airflow 3.0. For now, the only change considered would be to add the support to the existing Trigger API to return an async result which doesn't need an AIP.
    • The other changes for GenAI will now be implemented as a new provider.
  • Make Execution Date to be non-unique for a DAG: Unknown User (uranusjr) 
    • In the next couple of days, a discussion thread with a draft AIP will be created by Unknown User (uranusjr) , followed by a vote.
• Splitting providers from the mono-repo (Unknown User (kaxilnaik))
  • Unknown User (kaxilnaik) is not going to propose this anymore as he agrees that doing this now will increase the work and split the focus of folks. So we are going to NOT split providers from mono-repo to laser focus on Airflow 3 feature work.
  • We might think about it at a later date after Airflow 3.0 is released.
• AIP-49 OpenTelemetry Support for Apache Airflow:
  • The PR for the 2nd part got merged for 2.10 and the AIP is marked complete.
  • Howard Yoo 's comment: Initially we thought to collect logs in standard way (scraping from log files), but found a much better way to collect logs, by utilizing log api to get the task logs, and then attaching them as 'span events' which would be much more useful. The extraction of Task Logs and attaching them as span events is thus included in the AIP-49, thus it is not 'excluded', albeit not as a separate 'log' data as originally thought.
• StatsD vs Open Telemetry
  • There was a discussion on whether we should mark StatsD as removed in Airflow 2.10 itself in the docs here. The conclusion was that we should discuss this more as this has a wider impact including but not limited to the official Helm Chart which uses StatsD, the services providers (Astronomer, Composer, MWAA) and other long-time Airflow users. OTel also had limitation around character length limit and that you have to use tags with OTel but it didn't allow aggregating metrics with different tags etc. Shubham Mehta is going to check this out with his team and see if there are other concerns.
• Triage Help needed: There are 360 open feature requests. Some we might want to consider and seek help for in 3.0, and others we should close as not planned. Elad, Jens, Jarek, Vikram & Kaxil have volunteered to look at those issues. Elad is also going to check with Dosu team to see if we can re-label these old issues so they are more accurate. Categorizing these issues into "areas" label was one of the next steps too so that the volunteers can look at a particular area or some segment to divide & conquer. Unknown User (eladkal) could you take the one on for issue-triage?

8 August 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • AIP-79: Remove Flask AppBuilder as Core dependency: 
    • Unknown User (jedcunningham)  does not have a working POC yet. He will provide an update on the next call.
    • Unknown User (jscheffl) will draft a high-level plan for Decoupling Providers's Connections metadata from FAB after he is back from vacation in the next 2 weeks.
  • Airflow 2.10rc1 & 3.0 branching check:
    • The main branch became the Airflow 3 branch on 09 Aug 2024 as planned. 
    • Guidelines around it have been documented in this PR and here.
  • StatsD vs Open Telemetry:  Shubham Mehta mentioned that the AWS team has not encountered any issues so far, but the POC is yet to be completed. He will update about it on the next dev call.
• Discussions
  • Addressing Active/Paused DAG terminology for Airflow 3.0: The Consensus was to change "Active" to a better alternative. This GitHub issue should now cover it and is open for anyone to grab it.
  • Impact on Airflow 3 development while new UI is being built (Unknown User (bbovenzi))
    • There will be two UIs: the existing UI and the new React UI. Unknown User (bbovenzi)  will write a "UI dev practices" in the Contributing guide for someone wanting to try out a new UI. The guide will also include details like ensuring that all new features go to the new React UI.
  • Should we split Webserver, SDK, CLI, Scheduler, and Triggerer into separately installable packages for the users, while the dev code is in same mono-repo? (Jarek Potiuk )
    • The consensus was to wait until there is decent progress on AIP-72 implementation, but there was a general agreement on splitting the packages out while keeping apache-airflow as a meta-package that brings it all together. The intention is not to have a separate version scheme or release these separately however unlike Providers.
  • Library choice for different Airflow APIs (API used internally by Webserver, AIP-72 and external API) (Unknown User (potiuk))
    • The consensus was to use Fast API and generate spec from the code. This will mean we remove Gunicorn from the Webserver, which will be a breaking change, but most folks agree that it is worth it.
    • This will be tracked as part of AIP-84 UI REST API. Buğra Öztürk has also volunteered to help
  • AIP-80: Operator Templating 
    • Some of the attendees strongly opposed breaking Airflow 2's behaviour because they feared it would cause migration issues and that migration tooling wouldn't be sufficient. So, for Airflow 3, we will continue with the existing templating behaviour and add the new (and explicit) feature proposed in the AIP-80. After implementation, if TP finds a good 1-command migration for this, he will propose it on the mailing list with a demo to showcase the migration effort (with 100s of DAGs) to convince that we can break this behaviour but until then the default answer here is, for now, the decision is to keep the Airflow 2 templating behavior in Airflow 3.

22 August 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • AIP-79: Remove Flask AppBuilder as Core dependency: 
    • Jed made good progress on a working POC – he was able to pull out FAB from the core easily; he is going to verify if he didn't miss anything. He will provide an update on the next call.
    • Jed , Vincent  & Jarek  have also been discussing what Auth manager we should have for Core. Vincent's plan is to have a simple auth manager for local installs, and this will be driven via an Airflow config. He will build one for production using Casbin or Keycloak after the Airflow Summit. Currently, his team is doing research on which would be better.
      • Action Item: Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. This can also wait for an in-person chat during the Airflow Summit.
    • Elad raised a concern about this PR that separates FAB migrations from Core. His concern was that this was the first time we would be doing a migration from a provider, so we should think about the messaging to the users via docs (both Airflow & FAB provider) because up until now, we tell users to just upgrade provider dependency without thinking of DB migrations. Ash had an idea to maybe call the FAB thing as a Plugin instead of a provider, we decided to discuss this async.
      • Action Item: Ephraim  will send a mailing list thread about how we plan to handle messaging around DB migrations from FAB and if we need to discuss Ash's idea about calling FAB a plugin rather than a provider. 
  • Update on development practices/guide while the new UI is being built (Brent )
    • Brent proposed 3 guidelines to help committers accept changes to legacy UI until the new UI is ready to be merged.
      1. Bugfixes that can be cherry-picked for 2.10.x, 2.11.
         1. It is fine to accept these changes
      2. Airflow 3.0 features
         1. Add minimum changes necessary to unblock dev work without adding a ton of FAB views which will be deleted anyway once new UI is ready
      3. Fixes or Updates to existing react views 
         1. It is fine to accept those changes. The changes will be ported over to the new UI.
    • Brent proposed that the same rules should apply to non-public API endpoints for UI.
    • The PR to add this new airflow/ui  project has been created.
    • Unknown User (pierrejeambrun) has joined forces with Brent to work on the UI pieces.
    • Action Item: Brent  will write up the guidelines in the docs and send an email with the link to PR on the dev mailing list.
  • Plan for Decoupling Providers's Connections metadata from FAB (Unknown User (jscheffl))
    • Jens created this draft PR with the POC for it and presented it on the call.
    • Jarek proposed the idea of dumping the JSON/YAML with connection fields in the Database or loading it via package metadata so we don't load all the dependencies on the webserver.
    • We will need some plan for external providers on how they can define connections or register them.
    • The POC successfully proved that we can separate the connection metadata from FAB
    • Action Item: Jens to create a GitHub issue for decoupling the Connection metadata from FAB
  • Open Telemetry as a StatsD replacement POC Update (Shubham  / Dennis ) 
    • POC is working fine and the AWS team were able to use Open Telemetry with Cloud Watch. There were some duplicate metrics (based on name, length & limitation) that they will clean up as they decide to move.
    • Howard Yoo gave a presentation on OTel support in Airflow. Recording details below:
      • Zoom Recording Link | Passcode: OXzbxW+9
    • Action Item: This discussion will be continued on the dev mailing list. Kaxil to create a thread on the mailing list about OTel vs StatsD to discuss if we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. 
• Discussions
  
  • Airflow 3 development guideline review / Airflow 2.11: Bridge release (Unknown User (eladkal))
    • We discussed two open issues:
      1. How are we going to manage milestones? There will be two PRs: one against main and one against v2-10-test. In some cases, which milestone should we use on the original PR and the backport PR?
         1. We agreed that for now, we will only add milestones to the original PR (targeting the main); for example, if the original PR targets the main but is intended for 2.10.1, it should have a 2.10.1 milestone.
      2. How are we backporting/cherry-picking?
    • We need to remove features from 2.11 GitHub milestone 
    • Action Item: Kaxil to create a lazy consensus email as a follow-up to this discussion about Airflow 2.11 and delete all features from 2.11 GitHub milestone 
    • For PRs that just have deprecations changes, we should just merge them to release in the next 2.10 patch release so the users are made aware of it asap so they can start planning. When we have a change that is targeted for the 2.11 bridge release, we will discuss how to handle it since it won't be merged to v2-10-test. Possibly, we would create a v2-11-test  branch from v2-10-stable later this year when we are closer to have most of the AIPs on the finishing lines.
  • More help needed on our CI / infrastructure (Unknown User (potiuk) )
    • Unknown User (husseinawala)  presented the Architecture of the current CI
      
    • Shahar Epstein & Buğra Öztürk volunteered to work on CI issues.
  • New code layout for core/providers/task-SDK (Unknown User (ash) )
    • We ran out of time on this one but Ash has started a thread here for discussion.
• Consolidated Action items list:
  • Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. This can also wait for an in-person chat during the Airflow Summit.
  • Ephraim  will send a mailing list thread about how we plan to handle messaging around DB migrations from FAB and if we need to discuss Ash's idea about calling FAB a plugin rather than a provider. 
  • Brent  will write up the guidelines in the docs and send an email with the link to PR on the dev mailing list.
  • Jens to create a GitHub issue for decoupling the Connection metadata from FAB
  • Kaxil to do some homework on OpenTelemetry support so we can confirm an opinion to keep or drop StatsD support when we make OpenTelemetry the default
  • Kaxil to create a thread on the mailing list about OTel vs StatsD to discuss if we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. 
  • Kaxil to create a lazy consensus email as a follow-up to this discussion about Airflow 2.11 and delete all features from 2.11 GitHub milestone 

5 September 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. They will discuss it during the Airflow Summit.
  • Ephraim  created a thread here to discuss the messaging around DB migrations from FAB
  • Brentwrote up the guidelines in the docs and sent an email here to the dev mailing list.
  • Jens created a GitHub issue for decoupling the Connection metadata from FAB and potentially from provider dependencies
  • OTel & StatsD
    • Dennis has started a thread here about StatsD/OTel metric naming.
    • After the summit, Kaxil will need to create a thread on the mailing list about OTel vs StatsD to discuss whether we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. 
  • Kaxil created a VOTE thread here about Airflow 2.11 being a bridge release and deleting all features from the 2.11 GitHub milestone 
• Presentations
  • Jarek  gave us a sneak peek of his Airflow Summit keynote talk on Security: "Airflow Beach Cleaning" project – Security work
  • Pierre presented the new UI Rest API to explain how it works, how to develop it, and how to migrate from Connexion to Fast API
  • Kaxilpresented the Survey around Debugging Improvements created as part of this GitHub Issue by Iliya Romm, Omkar & Amogh.
• Discussion about deprecation/removal of Core_operator functions if a provider package should be cut  (Elad Jens  Jarek Ash  )
  • We covered Airflow Standard Provider & deprecations (Unknown User (rom) ) as part of it.
  • The standard provider will contain "directly usable in DAGs" Operators, Hooks, Sensors that are currently in Airflow "core". Some of the base classes ( BaseOperator , Base Hooks, SkipMixins will remain in the airflow.sdk  as the "foundation" for all other operators)
  • Standard provider 1.0.0 will contain everything we want to extract there - which means that the provider will be in "not-ready" state until we migrate all the "standard" components - and we release it then as 1.0.0. Before the migration we remove all deprecations that were in those operators - so that the "standard" provider contains "clean Airflow 3 versions" of those operators.
  • We discussed a few paths of upgrade for users:
    1. The user is on Airflow 2; they fixed the deprecation warning and then installed the new standard provider. Then, switch the DAGs to use operators from the Standard provider instead of AF 2. They are now ready to upgrade to Airflow 3.
    2. The user on Airflow 2 fixes all the deprecations and upgrades to Airflow 3; the import paths ( airflow.operators.python ) work due to dynamic imports that Ash suggested on the mailing list. This will now throw warnings that a user needs to import PythonOperator (as an example) from the standard provider ( airflow.providers.standard ) instead of Airflow, but it will work as the "standard" provider will be preinstalled with Airflow. A user will now switch the import paths to using one from the standard provider.
  • The standard provider 1.0.0 will contain changelogs of breaking changes from the Airflow core. Where applicable, the relevant news fragment in the current core will be moved to the Airflow standard provider. The newsfragments (Updating guide) for Airflow 3 docs will say that the core operators have been moved to the standard provider and it will have a link to that Standard provider docs for detailed breaking changes. 
  • The Airflow 2 to 3 migration process will be documented in the Airflow docs (similar to this but better), with a recommended approach from each of the two approaches listed above. The news fragments are right now just used for tracking breaking changes, they will be collected & organized to create a detailed upgrade process and a list of breaking changes.
  • Jarekhas also started a lazy consensus email thread here about the above.
• The recording of this call was added to YouTube as an unlisted video as there were a lot of discussions & useful presentations on this call:
  

19 September 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Unknown User (kaxilnaik) showed the GitHub project boards for some of the existing AIPs. Kaxil will follow up with all other AIP-leads to have the project board created.
  • Unknown User (kaxilnaik) discussed a proposal for how the intermediate milestones would be created to track progress. Kaxil will follow up with a draft with Vikram & Rajesh and then follow up with all the AIP leads to have the milestones defined. The general agreement was to have 3 intermediate milestones: Week of 24 Oct, 18 Nov and 16 Dec
  • Unknown User (vikramkoka) confirmed that Astronomer did not end up using Casbin itself but just the interfaces. Shubham Mehta mentioned Unknown User (onikolas) is going to explore Casbin and Keycloak and start implementation in about a week. The Dex project was suggested as yet another option by Mathew Wicks at the Airflow Summit. This was used by them in the Kubeflow project. His suggestion was to utilize it for Authentication only and continue doing Authorization on the Airflow side. Shubham will reach out to him.
• Discussions:
  • Follow up of [LAZY CONSENSUS] "Edge" as the name for AIP-69 in devlist: Shall providers be split into Operators / Executors / Plugins for the purpose of deployment where needed? At the moment all features that can be contributed from the side are in the same structure (Unknown User (jscheffl) )
    • Diagram to show what functionalities we have in provider packages that does into which part of an Airflow deployment:

• • • The goal of the discussion was to split the providers into Executors, plugins, and what's needed for DAG Authors (i.e User Code) like Operators and hooks. This way, we can limit the Python dependencies needed for Webserver, Scheduler, Worker, etc. An example is an AWS provider that has an Executor as well as an Operator. If we split them, then Scheduler can use the Provider that has an Executor, and Worker (including Triggerer & DAG Parser) can use the AWS provider that has an Operator but not an Executor. Airflow 3 is the right time to split this out if we decide to go that way.
    • re External Operator Links: Some ideas were discussed, like the parser storing the links in the DB, which is then retrieved by the Webserver via an API, so the Webserver does not need to install a provider just for Operator Links.
    • Unknown User (jscheffl)  will redraw the above diagram to separate out the Scheduler and the DAG Parser as the Scheduler does not run User Code, but the Parser does. As a follow-up to it, Jens will have a proposal around what the separation of providers would like to discuss on the next call and/or the dev mailing list. Unknown User (vikramkoka) and Unknown User (kaxilnaik) have volunteered to work with Jens on it offline before the next call.
  • Airflow Standard Provider: Bash & Python Operator (Pavan Kumar)
    • Pavan brought up the concern that System tests will break if we change the example DAGs for Bash & Python from core to using Standard provider until it is released. This was because some of providers like Google use the Bash or Python Operators.
    • Ideally, it should work with Airflow 3 since the provider will be pre-installed (via required_deps) with Airflow core/SDK.
    • The team decided to take it offline. Reviews would be appreciated.
  • Potentially moving dev calls on Oct 3 to Oct 9 or Oct 10. And one on Oct 31 [Diwali] to Nov 1 (Unknown User (kaxilnaik) )
    • Decided to move 03 Oct 2024  call to 10 Oct 2024  and move the 01 Nov 2024  to 07 Nov 2024 

10 October 2024

Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Unknown User (jscheffl) walked through the draft diagram for splitting up the providers into Operators / Executors / Plugins, based on the long term (Airflow 3) view of where they would be used. Jens stated how a follow-up proposal could be to move the “Extra Links” to be stored in the Airflow meta-database, since they are shared across components. A similar issue is  for the Extra Fields for Connections since they are also shared across components such as DAG Parser and Webserver. Unknown User (vikramkoka) added that this separation of providers was primarily driven from the packaging and distribution strategy as well as the release strategy to be used for the providers going forward because of the Core vs. Task SDK split coming as part of Airflow 3. The consensus from the group was a proposal needed to be written as a follow-up and shared with the dev list. Unknown User (jscheffl) suggested that Unknown User (vikramkoka) take a stab at the draft of the proposal and Unknown User (vikramkoka) agreed to take it as an action item.
  • Unknown User (vikramkoka) walked through the monthly Development milestones for Airflow 3, across all AIPs. The feedback from the group was positive and it was felt that it was helpful not only for the contributors, but the larger community. Unknown User (kaxilnaik) suggested that this should be a page within the top-level Airflow 3.0 page on the wiki. Unknown User (jscheffl) mentioned that he needed feedback on this PR on AIP-69 based on the AIP-72 work and Unknown User (ash) responded he would review that async. Unknown User (vikramkoka) asked the AIP owners was to update the milestones with details and status as appropriate, but also to notify him of changes for potential ripple effects.
• Presentations:
  
  • Rahul Vats walked through the high level Airflow 3 test plan which included all the AIPs targeted for the release as well as some initial end to end scenarios. Rahul created these based on the AIP documents and asked for the AIP authors to review these and add additional end to end scenarios as appropriate, so that any breaking issues could be identified as part of the development process itself. Rahul recommended that these be used as the source of truth and be a foundation for the system tests. He also referenced the use of the testing slack channel #sig-airflow-3-qa-testing and asked for help from the community in testing this major release. Rahul took the action-item to articulate “what should be covered in the AIP unit tests vs. here in the E2E tests” with examples. Rajesh Bishundeo asked about performance testing AIP-59 in the context of Airflow 3. Michal Modras took the action-item to follow up with the original author of AIP-59 and see if the author or an alternate Google developer could give an updated proposal in time for Airflow 3 testing. 
  • Unknown User (bbovenzi) shared how the team has been using the UI update opportunity (AIP-38) for a UI Design refresh and presented mockups including the new home and DAGs list page. This had also been shared as part of the Airflow Summit. Pierre Jeambrun presented a progress update on the FastAPI-based approach, including a quick demo (AIPs 84). Unknown User (bbovenzi) also presented a progress update on AIP-68, including a quick demo of the approach for custom react components (plug-ins) based on FastAPI. 
• Discussions:

  • Unknown User (jscheffl) brought up the topic of continued support for Python 3.8 in the upcoming Airflow 2.11 release. I.e. if it was required or if it could be dropped. The consensus was that there were no objections from the attendees, but the next step was for Unknown User (jscheffl) to send an email out to the dev list about this asking for a “lazy consensus”.

24 October 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Providers follow-up: Unknown User (vikramkoka) commented that he, Jens Schubert , Jarek Potiuk , and others have been collaborating on a proposal document and that it would be shared well before the next dev call. Vikram shared a split version of that document to dev list on Friday as a pre-read for the providers repackaging document.
  • Performance test plan: Michal Modras shared that one of his team members would be working on it. He said that the current situation is that they have an “elastic DAG” which can be used for benchmarking various scenarios. He said that they would be working on the benchmark scenarios and sharing that document by the end of Nov. After that, the initial data collection based on these scenarios would be done and shared in Dec.
• Development updates and presentations:
  
  • Unknown User (uranusjr) shared a recorded video detailing the progress of AIP-74 on AIP-74 Introducing Data Assets. This showed the current progress on the AIP which was close to completion from a back-end perspective. This included the renaming of the existing Datasets and adding a name attribute to the asset class, to enable the specification of a more user-friendly name for the asset. He also shared that this name attribute is not currently shown in the UI, but that they would be working with their UI team to reflect these changes in the Airflow UI. 
  • Ankit Chaurasia shared an update on AIP-83 Remove Execution Date Unique Constraint from DAG run. He mentioned how this follows up on the work initially started in Airflow 2.2 with the shift towards run_id as a unique identifier and the logical_date as a human usable non-unique reference. He walked through the massive (around 280 files touched) draft PR focused on removing the execution_date completely including the CLI and the API. This PR has already had several comments from reviewers. The pervasiveness of the “execution date” was reinforced in the discussion. He mentioned how one of the challenges was also with the backfill PR and was waiting for the backfill PR to be completed and merged. He showcased the test plan for testing this AIP and requested the group to review and provide feedback on the test plan. He concluded that the PR is in good shape logic-wise, but fixing tests. The discussion after that raised the point that we need to upgrade the API version from v1 to v2 or v3 as part of Airflow 3, possible name to be “stable v3 API”.
  • Unknown User (ephraimanierobi) shared an update on AIP-79 Remove Flask AppBuilder as Core dependency. It has a lot of work streams including the auth manager. Ephraim mentioned how the part he was working on was the separation of the DB migration from Fab. He mentioned how apps could now use an alternative mechanism called external DB managers to run DB migrations as part of the upgrade. He mentioned how this AIP included other work such as Auth manager and UI. Rajesh added (on behalf of Vincent) that the Auth manager was close to being done. Ash had also done some work on this AIP regarding 5.0 RC, but was not yet complete with respect to testing. Vikram expressed concern that clearly multiple “necessary parts” were done as part of this AIP, but not clear if all the “sufficient parts” were done. Ephraim mentioned that “what is left is for the UI to switch”. This needs to be followed up in the next dev call. Jens suggested that we may need to migrate or reimplement the “Connection form UI”.
  • Unknown User (ephraimanierobi) also shared an update on AIP-65 Improve DAG history in UI - backend changes. He said how he is working on DAG Versioning around the serialized DAG, so that older versions of the DAGruns could be visible through the UI. Ephraim mentioned that the serialized DAG hash is not always consistent today because of inconsistent sorting, which is the first thing he just fixed. He also added a user-specified version for the DAG and talked about how he is close to completion on the DAG history database changes. He said that the next steps were the API and UI work.
  • Unknown User (kaxilnaik) shared an update on AIP-72 Task Execution Interface aka Task SDK, specifically focussing on the API server. He showed the addition of the scaffolding for the API server, so that in the future, the Core API and the execution API servers could be run independently if needed for different scaling requirements. He mentioned how he and Ash are working on porting code from Core to TaskSDK. Unknown User (ash) then followed up with sharing the main PR for the Task SDK, which covers the DAG object, the TaskGroup, the Base Operator and so on, which would be the heart of airflow.sdk, which is the core Task SDK. To make upgrades easier, DAG import would stay as “airflow.dag”, so people don’t even get any deprecation warnings. Ash mentioned how he was currently thinking of including this “airflow.sdk” in the Scheduler to avoid reimplementation, but was still thinking through the implications. He mentioned how a lot of the classes are now using “attrs” to save on boilerplate code. Ash mentioned how the next step is to start adding endpoints for the API server to start doing simple client-based execution.
  • Unknown User (bbovenzi) finished up the call with an update on designs for AIP-38 Modern Web Application by walking through mockups of the redesigned DAG details page, including a list of actions / filters to show only failed tasks, etc. Brent also walked through the mockups for DAGruns view for Gantt chart view vs. Graph view, etc. Brent showed how the modal view enables a quick view of the details of a DA run and to switch between different areas which a user may want to drill into. Brent mentioned how the motivation was to enable proper use of space and to zoom between a DAG and a DAGrun and TaskInstance, without having to show a giant Christmas tree of the grid view.
• Action items for next call:

  • Unknown User (vikramkoka) to update the development milestones doc based on the performance benchmark plan from Michal Modras  - DONE

  • Unknown User (vikramkoka) to create a subpage under Airflow 3.0 to track critical changes important for upgrades, including API changes such as those brought up in the Execution date removal discussion. API owners to update this page as they make user facing changes during development. 
  • Unknown User (ephraimanierobi) , Vincent, and others interested to follow-up on "What's left for AIP-79" and present at the next dev call. 

7 November 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on AIP-79 with respect to what's left Unknown User (jedcunningham), Vincent BECK, Unknown User (jscheffl): Jed shared that it was unclear at this time if AIP-79 was blocked on AIP-38 (new UI).
    • Jed to report back by the next dev call if he could magic back the older Fab provider, so that the old Fab provider could exist independent of the new Fab provider, so that we are not blocked by the new UI work
    • The Connection form for the new UI is another item which needs to be done and is not blocked by anything else. Jens's intention is to wait  and build the Connection form based on the Trigger form in the new UI. Though to be clear, no one actively owns the Connection form work at this time, but Brent and Jens believe that this will naturally follow as part of the AIP-38 work.
    • Clarity from Jed and Vincent that the new Fab provider is needed for the Auth manager and more and is not in conflict with the work that Jed is proposing above. 
  • Breaking changes update Unknown User (vikramkoka): Vikram walked the group through his thoughts around capturing all changes that users need to worry about as part of upgrading to Airflow 3. Specifically to structure the newsfragments with a template. Vikram proposed an initial template with the following elements: 

    • DAG code changes

      • Import changes

      • Other DAG code changes

      • e.g. SubDAGs - please note if it was already marked as “deprecated” or “experimental”

    • Stable API changes: both invocation and result

    • CLI changes:

    • Airflow configuration changes

      • Airflow.cfg changes

      • Integration changes

    • Behavior changes: Example Backfill or Execution date changes

    • Plugin changes

  • There was immediate agreement within the group for the above and the desire for a concrete template. Unknown User (vikramkoka) to publish the above and then it could be added as a pre-commit. Vikram to also create an initial Upgrade / Change overview page, which would have example results based on some of the initial newsfragments (around 10).
• Development updates and presentations:
  • Airflow 3 workstreams update Unknown User (vikramkoka): Vikram walked the group through the updates to the Airflow 3 workstreams and Airflow 3 dev milestones pages to reflect the high level status of the Airflow 3 project as well as finer level monthly details. Positive feedback from Rajesh and the rest of the team on the overall summarization in the Workstreams page, with the more granular detail in the monthly dev milestones page. Vikram reiterated the ask for more help on the AIP-38 UI work. Brent said that we are now ready for others to pickup work now, since the back-end APIs are ready to use. Brent to create a Slack sub-channel for UI development specifically for AIP-38 work. 
• Discussion topics:
  
  • Update other candidates from the Workstreams list:
    • Remove SLAs and AIP-57 Refactor SLA Feature - obsoleted by AIP-86 Deadline alerts?  Unknown User (ferruzzi)  confirmed that this entry was now obsoleted by AIP-86 Deadline Alerts. 
    • Remove poke, reschedule mode from Sensors? Unknown User (kaxilnaik) said that there was a discussion in one of the early dev calls and that he felt that this should be removed to really simplify the code and choices for the users. Unknown User (dstandish) felt that there was a difference between support in Operators vs. Airflow Core. He felt that the support in Operators was very confusing.  Shubham Mehta felt that this capability was useful for very long running tasks (greater than 12 hours) in the past, but wasn't sure if it was still necessary after Airflow 2.8, and would get back to the team about the same. Unknown User (jedcunningham) felt that it would be much simpler to take it out. There was a good discussion with multiple members of the team and both Igor Kholopov and Shubham Mehta confirmed that they didn't have a strict "no" at this time, but would need to confirm. After the discussion, the action item was for Unknown User (kaxilnaik) to start an email discussion around this in the dev list. 
    • Inspect & Simplify Airflow Configurations: Shubham Mehta and Unknown User (ferruzzi) confirmed that we should just defer this to post Airflow 3.0. 
  • OTEL integration into Airflow Scheduler: Unknown User (dstandish) 
    • There was a long discussion about the task log reading introduction into the very performance intensive Scheduler loop and the challenges therein. 
    • Unknown User (kaxilnaik) to start an async discussion around what changes need to be done to either refactor or revert these changes for OTEL traces. 
• Action items for next dev call:
  • Unknown User (jedcunningham) to confirm back regarding the older Fab provider changes for compatibility. 
  • Unknown User (vikramkoka) to create a template for newsfragments for flagging upgrade issues. And to create an initial Airflow 3 upgrade considerations page based on some of the existing PRs. 
  • Unknown User (bbovenzi) to create an Airflow 3 UI slack channel to facilitate collaboration on Airflow 3 UI dev ("airflow-3-ui"). 
  • Unknown User (kaxilnaik) to start email discussion regarding removal of Poke and reschedule mode from sensors. 
  • Unknown User (kaxilnaik)  to send an email about the issues with OTEL integration for Traces into the Scheduler. 

21 November 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on the POC of Flask/FAB app into the Fast API Unknown User (jedcunningham) :
    • The POC worked and that means we will not need a separate webserver to maintain backward compatibility with existing plugins. 
    • In addition, this means that we don't need the new UI (AIP-38) to be far enough along to remove the old UI. The old UI can stay as-is, and support for plugins can be added in the new FastAPI webserver.
    • Taking this from POC to production level reality will take some work. This will be planned by Jed and Vincent. Anticipate seeing a Github project board with the items to be done in time for the next dev call. 
  • Breaking changes update Unknown User (vikramkoka) : 
    • Vikram walked the group through the draft of the Airflow 3 breaking changes document. 
    • Vikram mentioned that this is about 50% populated right now, with 50% of current known changes still to be populated. 
    • To reduce the manual effort going forward, we will have a newsfragment template to be filled out as part of the PR process
• Development updates and presentations:
  • AIP-72 update (Unknown User (ash)  and Unknown User (kaxilnaik) ): 
    • Ash demonstrated the working state of AIP-72 with respect to the separation of the Task Execution code into the TaskSDK vs. Airflow Core. This was in the form of tasks created with the new Task SDK and being run on the Local Executor, using a supervisory process.
    • The demo included the code showing how the supervisory process launched the task process and the socket pair / dup mechanism for logging. Ash also mentioned that the logging providers may need changes to accommodate the new system.
    • The team also discussed how this would make it a lot easier to intelligently display logs in the UI, without the UI needing to reparse the logs. It was very encouraging to see this coming together. 
  • AIP-66 update (Jed):
    • Jed shared a quick update regarding the progress being made on AIP-66 including the focus on the refreshing bundles from an external source. Jed also mentioned that he would be ready to open a PR for this early next week and that the demo could happen in the next dev call. He also expressed hope that the connection form work could be leveraged for the UI support here.
  • AIP-38 update (Brent):
    • Brent presented the UI progress including the new react-based DAGs list page and the dashboard page with basic statistics all leveraging the new FastAPI work. Brent mentioned that they were working on the Dag Run and task instance pages.
  • AIP-84 update (Pierre):
    • Pierre and Brent expressed gratitude for the many contributions from the community on AIP-84. Pierre confirmed that they were almost done with the migration of all public endpoints and that there were only a few remaining private endpoints to be done. 
• Discussion topics:
  • Meta milestones (Vikram):
    • Vikram highlighted that from an overall standpoint, the project was generally on track with the published schedule. 
    • He emphasized that the backend features should ideally be completed by the end of December and must be integrated by the week of the 18th of January in order to be on schedule. 
    • He said that this was critical for the packaging and upgrade work to be done on a stable base and be completed by the end of March. He mentioned that the UI could continue to change till early March, but the Jan backend feature freeze was critical. He asked all the AIP owners to keep this in mind and be pragmatic about their AIP scope
• Action items for next dev call:
  • Update on Fab provider backward compatability project board Unknown User (jedcunningham)  and Vincent BECK 
  • Update on Performance benchmark scenarios Michal Modras (based on the call on 24th Oct)

5 December 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on the FAB provider for backwards compatibility project Unknown User (jedcunningham)  and Vincent BECK 
    • Jed and Vincent shared the progress to date including a PR that already implements plug-in backwards compatibility. 
    • Unknown User (vikramkoka)  expressed appreciation for the progress and asked about the expected timing of the remainder of the items to be done and their response was outside of the New UI completeness blocker, the other items could be done by mid-Jan. 
    • Unknown User (jscheffl) suggested that the PR to validate dependencies without the new UI be created as a draft and validated with the existing functionality of the new UI rather than waiting for the new UI to be completed.  
  • Update on Performance benchmark scenarios Michal Modras :
    • Augusto shared the thinking around performance benchmark scenarios and metrics, with a focus on DAG performance and resource consumption. 
    • Augusto shared that this was a follow up on the work already done on AIP-59 and would be based on the existing performance framework. 
    • There was a significant discussion around the task timings and if those were relevant for realistic performance benchmarks. 
    • Jens asked if this would cover different executors and Augusto responded that this would be Celery first and possibly Kubernetes executor later.
    • Jens and Vikram brought up comparing the performance of Airflow 2.10 vs. Airflow 3 to identify performance differences. Augusto confirmed that all the tests would be run on both Airflow 2 and 3 to confirm performance changes. 
• Development updates and presentations:
  • Update on AIP-75 New Asset-Centric Syntax Unknown User (uranusjr) 
    • TP shared a recording of the new syntax for asset creation.
    • TP also showed the demo of a new Airflow CLI command to list all the Data Assets and to show the details of a specified Data Asset. 
    • Finally, TP also introduced the "materialized" command for a data asset which ensures that the asset is created by running the DAG which outputs that asset. 
  • Update on AIP-66: DAG Bundles & Parsing Unknown User (jedcunningham) 
    • Jed demonstrated the process of defining DAG bundles and how DAG bundles would be parsed by the DAG processor
    • He mentioned how some of the changes are happening in conjunction with the changes being done in AIP-72. 
    • He also showed bundleIDs and bundle Versions. He then showed how a new version is parsed and reprocessed. 
    • He mentioned that there is much more work to be done, but the core of bundle definitions and DAGs being processed from those bundles is now in place.
    • In response to questions, he clarified how DAG Bundles currently pull down the entire Git clone into a temporary folder, so that all DAGs and their friends/dependencies could be processed. And that, more optimization is very feasible.
  • Update on AIP-78 Scheduler-managed backfill Unknown User (dstandish) 
    • Daniel said that all the back-end server work for this AIP as scoped has been completed for a bit. He added that the front-end UI work will be done as part of AIP-38. 
    • He however added that there is a Data completeness conversation to be had as a result, which led to the discussion below.
• Discussion topics:
  • Data completeness discussion  Unknown User (dstandish) 
    • Daniel brought up the concept of implicit data partitioning already in Airflow with the concept of execution date, when catchup is defined to be True. 
    • Daniel advocated making this implicit data partitioning an explicit concept in Airflow, arguing that the existing grid view is already an incarnation of the same. 
    • At a high level, users could declare that a DAG is partition-driven, based on the timetable. Going forward, Backfills or catchup would only be supported for partition-driven DAGs. 
    • For backwards compatibility, old DAGs would be assumed to be partition driven. 
    • The immediate reaction from the team is that this is a big change and there was significant discussion if this is absolutely required.
    • Daniel said that the trigger for this was AIP-78 Scheduler-managed backfill and AIP-83 Remove Execution Date Unique Constraint from DAG run, which left a bit of a vacuum between them.
    • The follow-up action item after the discussion was for Daniel to share thoughts async and everyone to think about the need for this. 
  • Milestone and scope update Unknown User (vikramkoka) 
    • Vikram shared that at a high level development was on track towards the plan shared earlier.
    • However, there would be one scope change with AIP-80 Explicit Template Fields in Operator Arguments being deferred from 3.0 to a future 3.x release. 
• Action items on/before next dev call:
  • Unknown User (dstandish) to post a document regarding explicit vs. implicit partitioning and its need as a result of the removal of execution date, especially with an eye towards backwards compatibility. Team to consider the introduction of a partition concept in Airflow. 

19 December 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on the Data completeness discussion Unknown User (dstandish) 
    • Daniel presented the problem around ambiguous behavior for backfills, when logical dates are no longer unique. 
    • He spoke about how the traditional Airflow behaviour using execution date based on the DAG schedule implicitly established a date-based partitioning scheme for DAGs.  
    • He then advocated making this behaviour explicit, by defining this to be a partition scheme of "implicit" and also supporting a partition scheme of "none" for non-date-based partitioned DAGs which could be run multiple times for the exact same logical date, which is a common use case both for model training as well as for inference execution. 
    • There was a significant level of debate around this topic, with two counter-points. The first counter-point was that the "partition" term was confusing, especially with "Data Asset Partitions" upcoming in AIP-76, and if this was the same, different, or related. The second counter-point was around whether a new DAG-level parameter should be added for this, as opposed to either one or more combinations of DAG API/UI invocation parameters and existing DAG parameters such as "catchup". 
    • Since a conclusion could not be reached in the call, the action item for Daniel, was to write up a document as an amendment to the existing AIP-83 Remove logical date uniqueness and share it with the mailing list. Daniel created and shared this doc on Friday.
• Development updates and presentations
  • Update on AIP-38 Modern Web Application Unknown User (bbovenzi) 
    • Brent demonstrated the progress on the UI specifically including the DagsList and DagDetails pages. 
    • This included the ability to trigger DAGs and view DagRuns.
    • Brent then drilled into viewing DAG Run details such as Task logs and XCom data. 
    • It was very encouraging to see the core DAG development and test workflows in action as part of Airflow 3. 
  • Update on AIP-72 Task Execution Interface aka Task SDK Unknown User (amoghdesai) , Unknown User (kaxilnaik) , Unknown User (ash) 
    • Amogh started the update on AIP-72 by walking through how deferred tasks were being executed based on the Task SDK. 
    • The demo was based on a DAG including a date time sensor and the interaction between the supervisor process and the task runner for different task states.
    • Amogh also explained the task execution states which were currently implemented and the ones which were still to be done. 
    • Kaxil then walked through the implementation of the TaskContext, a key concept within the Task Execution Interface.
    • This included the interface and the implementation of Connections, and how that was fetched from the API server. 
    • Ash briefly shared a PR that removed the database dependency for DAG parsing and how that leveraged the supervisor-task runner communication interface. 
    • Because we were running out of time in the dev call, he said that would subsequently post a video of the demo. Ash posted this demo video to the dev list later. 
• Discussion topics:
  • PR strategy for 3.1+ feature work - Rajesh
    • Rajesh wanted to discuss how to handle PRs for post-3.0 feature work. Since the dev call was already running over, this was to be discussed async. 
  • Scheduling for next dev call
    • Since a lot of committers are out of holidays, the next dev call will be on the 9th of January, 2025 instead of the 2nd of January.