• Created by Unknown User (vikramkoka), last updated on Nov 21, 2024 9 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

This is an early draft of the changes coming in Airflow 3 which can have backwards compatibility impact. 

The data below is captured from the "newsfragments" incorporated into the Pull Requests and is intended to give a quick summary overview of the changes. 

At this point, the attached GoogleSheets link is the source of truth (for easier and more broad editing) and the table below is a view of that sheet. 

AIP/PR

(Newsfragment)

Title

DAG

changes

Config

changes

API

changes

CLI

changes

Behaviour

changes

Plugin

changes

Dependency

changes

Notes

Doc link

(if any)

24842Default DAG schedule changed to NoneNNNNYN

N

When a schedule parameter is not passed to the DAG constructor, Airflow now defaults to never automatically schedule the DAG at all. The created DAG can still be manually triggered, either by the user directly, or from another DAG with TriggerDagRunOperator.
40029Deprecated allow_raw_html_descriptionsNNNNNYNRemove deprecated allow_raw_html_descriptions from UI Trigger forms
40931Deprecated store_serialized_dagsNNNN?NNRemoved dagbag deprecated store_serialized_dags parameter. Please use read_dags_from_db parameter.
40936on_success_callback no longer called if task skippedNNNNYNNon_success_callback will no longer execute if a task is skipped. Previously, this callback was triggered even when the task was skipped, which could lead to unintended behavior or inconsistencies in downstream processes. This is a breaking change because workflows that rely on on_success_callback running for skipped tasks will need to be updated. Consider updating your DAGs to handle cases where the callback is not invoked due to task skipping.
41096Deprecated process_poll_intervalNYNNNNNRemoved deprecated processor_poll_interval configuration parameter from scheduler section. Please use scheduler_idle_sleep_time configuration parameter.
41348AIP-74 (Rename Dataset to Asset)??????Y??N?N?NBig set of changes to rename Dataset to Asset. Touches many core Airflow packages. Unclear, if backward compatibility shims were added?
41366Deprecated airflow.contribNNNNNNNairflow.contrib modules have been removed.
All modules from airflow.contrib which were deprecated in Airflow 2, have been removed.
41367Deprecated ImportError YNNNNNNDeprecated ImportError removed from airflow.models
The deprecated ImportError class can no longer be imported from airflow.models. It has been moved to airflow.models.errors.ParseImportError.
41368Deprecated core imports YNNNNNNSupport for deprecated core imports removed
Support for importing classes etc from the following locations was deprecated at various times during Airflow 2s lifecycle, and has been removed:
airflow.executors
airflow.hooks
airflow.macros
airflow.operators
airflow.sensors
Instead, import from the right provider or more specific module instead. For example, instead of from airflow.sensors import TimeDeltaSensor, use from airflow.sensors.time_delta import TimeDeltaSensor.
41390Deprecated SubDAGs YNYYNNNSubdags have been removed from the following locations:
CLI
API
SubDagOperator
This removal marks the end of Subdag support across all interfaces. Users should transition to using TaskGroups as a more efficient and maintainable alternative. Please ensure your DAGs are updated to remove any usage of Subdags to maintain compatibility with future Airflow releases.
41391Deprecated airflow.sensors.external_task.ExternalTaskSensorLink YNNNNNNThe airflow.sensors.external_task.ExternalTaskSensorLink class has been removed. This class was deprecated and is no longer available. Users should now use the airflow.sensors.external_task.ExternalDagLink class directly.
41393Deprecated use_task_execution_day parameter in DayOfWeekSensor YNNNNNNThe use_task_execution_day parameter has been removed from the DayOfWeekSensor class. This parameter was previously deprecated in favor of use_task_logical_date.
If your code still uses use_task_execution_day, you should update it to use use_task_logical_date instead to ensure compatibility with future Airflow versions.
Example update:
sensor = DayOfWeekSensor(
task_id="example",
week_day="Tuesday",
use_task_logical_date=True,
dag=dag,
)
41394Deprecated airflow.models.taskMixin.TaskMixinYNNNNNNThe airflow.models.taskMixin.TaskMixin class has been removed. It was previously deprecated in favor of the airflow.models.taskMixin.DependencyMixin class. If your code relies on TaskMixin, please update it to use DependencyMixin instead to ensure compatibility with Airflow 3.0 and beyond.
41395Deprecated airflow.utils functions, constants, classesYNNNNNNThe following deprecated functions, constants, and classes have been removed as part of Airflow 3.0:
airflow.executors.executor_loader.UNPICKLEABLE_EXECUTORS: No direct replacement; this constant is no longer needed.

airflow.utils.dag_cycle_tester.test_cycle function: Use airflow.utils.dag_cycle_tester.check_cycle instead.
airflow.utils.file.TemporaryDirectory function: Use tempfile.TemporaryDirectory instead.
airflow.utils.file.mkdirs function: Use pathlib.Path.mkdir instead.
airflow.utils.state.SHUTDOWN state: No action needed; this state is no longer used.
airflow.utils.state.terminating_states constant: No action needed; this constant is no longer used
41420Replaced Python's list with MutableSet for the property DAG.tags.N?NNNNNNReplaced Python's list with MutableSet for the property DAG.tags.

At the constractur you still can use list, you actually can use any data structure that implements the Collection interface.

The tags property of the DAG model would be of type MutableSet instead of list, as there are no actual duplicates at the tags.
41434Experimental API removedNNYNNNNDeprecated Experimental API is no longer available in Airflow. Users should transition to using Rest API as an alternative.
41440Deprecated and unused methods / properties on DAGYNNNNNNRemoved unused methods / properties in models/dag.py

Methods removed:
date_range
is_fixed_time_schedule
next_dagrun_after_date
get_run_dates
normalize_schedule
full_filepath
concurrency
filepath
concurrency_reached
normalized_schedule_interval
latest_execution_date
set_dag_runs_state
bulk_sync_to_db
41453Legacy DAG Schedule args removedY?NYNNNNRemoved legacy scheduling arguments on DAG
The schedule_interval and timetable arguments are removed from DAG.
The schedule_interval _attribute_ has also been removed. In the API, a new timetable_summary field has been added to replace schedule_interval for presentation purposes.

Since the DAG object no longer has the schedule_interval attribute, OpenLineage facets that contain the dag key produced on Airflow 3.0 or later will also no longer contain the field.
41496Deprecated methods in date utilsYNNNNNNRemoved deprecated methods in airflow/utils/dates.py

Methods removed:
date_range
days_ago (Use pendulum.today('UTC').add(days=-N, ...))
41520Deprecated methods in Helper utilsYNNNNNNRemoved deprecated methods in airflow/utils/helpers.py

Methods removed:
chain (Use airflow.models.baseoperator.chain)
cross_downstream (Use airflow.models.baseoperator.cross_downstream)
41533Deprecated load_connections paramYNNNNNNThe load_connections parameter has been removed from the local_file_system. This parameter was previously deprecated in favor of load_connections_dict.

If your code still uses load_connections, you should update it to use load_connections_dict instead to ensure compatibility with future Airflow versions.
Example update:
connection_by_conn_id = local_filesystem.load_connections_dict(file_path="a.json")
The get_connections parameter has been removed from the LocalFilesystemBackend class. This parameter was previously deprecated in favor of get_connection.

If your code still uses get_connections, you should update it to use get_connection instead to ensure compatibility with future Airflow versions.

Example update:
connection_by_conn_id = LocalFilesystemBackend().get_connection(conn_id="conn_id")
41539Deprecated smtp config paramsNYNNNNNRemoved deprecated smtp_user and smtp_password configuration parameters from smtp section. Please use smtp connection (smtp_default).
41550Deprecated session configsNYNNNNNRemoved deprecated session_lifetime_days and force_log_out_after configuration parameters from webserver section. Please use session_lifetime_minutes.

Removed deprecated policy parameter from airflow_local_settings. Please use task_policy.
41552Deprecated log handler argYYNNNNNRemoved deprecated filename_template argument from airflow.utils.log.file_task_handler.FileTaskHandler.
41579Deprecated apply_defaults in decoratorN?NNNNNNRemoved deprecated apply_defaults function from airflow/utils/decorators.py.
41690Deprecated dependency_detectorN?NNNNNNRemoved deprecated dependency_detector parameter from scheduler
41635Deprecated opton ignore-depends-on-pastNNNYNNNRemoved deprecated --ignore-depends-on-past cli option from task command. Please use --depends-on-past ignore.
41642Deprecated secrets backend methods YNNNNNNRemoved deprecated secrets backend methods get_conn_uri and get_connections.

Please use get_conn_value and get_connection instead
41663Deprecated auth (basic_auth)NYNNNNNRemoved deprecated auth airflow.api.auth.backend.basic_auth from auth_backends. Please use airflow.providers.fab.auth_manager.api.auth.backend.basic_auth instead
41693Deprecated auth (kerberos)NYNNNNNRemoved deprecated auth airflow.api.auth.backend.kerberos_auth and airflow.auth.managers.fab.api.auth.backend.kerberos_auth from auth_backends. Please use airflow.providers.fab.auth_manager.api.auth.backend.kerberos_auth instead
41708Deprecated fab_auth_managerNYNNNNNRemoved deprecated auth manager airflow.auth.managers.fab.fab_auth_manager and airflow.auth.managers.fab.security_manager.override. Please use airflow.providers.fab.auth_manager.security_manager.override instead
41733Deprecated get_connections from BaseHookYNNNNNNRemoved deprecated function get_connections() function in airflow.hooks.base.BaseHook
41735Deprecated package airflow.kubernetesN?NNNNNNRemoved deprecated module airflow.kubernetes
41736Deprecated parameters from core-operatorsYNNNNNNRemoved deprecated parameters from core-operators.

Parameters removed:
airflow.operators.datetime.BranchDateTimeOperator: use_task_execution_date
airflow.operators.trigger_dagrun.TriggerDagRunOperator: execution_date
airflow.operators.weekday.BranchDayOfWeekOperator: use_task_execution_day
41737Deprecated TaskStateTrigger from airflow.triggers.externalYNNNNNNRemoved deprecated TaskStateTrigger from airflow.triggers.external_task module
41739Deprecated Backfill optionsNNNYNNNRemoved backfill job command cli option ignore-first-depends-on-past. Its value always set to True. No replcaement cli option.

Removed backfill job command cli option treat-dag-as-regex. Please use treat-dag-id-as-regex instead
41748Deprecated module airflow.hooks.dbapiYNNNNNNDeprecated module airflow.hooks.dbapi removed. Please use airflow.providers.common.sql.hooks.sql instead
41758Deprecated functions, modules from airflow.wwwNNN?NNY?NRemoved deprecated functions and modules from airflow.www module.
- Config flag default warning cookie_samesite option in section [webserver] removed.
- Legacy decorator @has_access in airflow.www.auth: Please use one of the decorator has_access_* defined in airflow/www/auth.py instead.
- Removed legacy modules airflow.www.security: Should be inherited from airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride instead. The constant value EXISTING_ROLES should be used from airflow.www.security_manager module.
- Removed the method get_sensitive_variables_fields() from airflow.www.utils: Please use airflow.utils.log.secrets_masker.get_sensitive_variables_fields instead.
- Removed the method should_hide_value_for_key() from airflow.www.utils: Please use airflow.utils.log.secrets_masker.should_hide_value_for_key instead.
41761Deprecations in base operatorYNNNNNNRemoved a set of deprecations in BaseOperator.

Parameter task_concurrency removed, please use max_active_tis_per_dag.
Support for additional (not defined) arguments removed.
Support for trigger rule dummy removed. Please use always.
Support for trigger rule none_failed_or_skipped removed. Please use none_failed_min_one_success.
Support to load BaseOperatorLink via airflow.models.baseoperator module removed.
41762Deprecated Connection elements in airflow.modelsYNNNNNNRemoved a set of deprecations in Connection from airflow.models.

Validation of extra fields is now enforcing that JSON values are provided. If a non-JSON value is provided a ValueError will be raised.
Removed utility method parse_netloc_to_hostname()
Removed utility method parse_from_uri().
Removed utility method log_info() and debug_info()
41744Deprecations DAG in airflow.modelsYNNNNNNRemoved a set of deprecations in DAG from airflow.models.

Removed deprecated parameters full_filepath and concurrency (Replaced by max_active_tasks) from DAG and @dag decorator.
Removed fallback support for legacy default_view == "tree".
Removed legacy support for permissions named can_dag_read and can_dag_edit. The permissions need to be named can_read and can_edit.
Removed legacy deprecated functions following_schedule() and previous_schedule.
Removed deprecated support for datetime in next_dagrun_info(). Use DataInterval.
Removed legacy DAG property is_paused. Please use get_is_paused instead.
Removed legacy parameters get_tis, recursion_depth and max_recursion_depth from DAG.clear().
Removed implicit support to call create_dagrun() without data interval.
Removed support for deprecated parameter concurrency in DagModel.
Removed support for datetime in DagModel.calculate_dagrun_date_fields. Use DataInterval
41766Deprecations in airflow.models.paramYNNNNNNRemoved a set of deprecations in from airflow.models.param.

Removed deprecated direct access to DagParam as module. Please import from airflow.models.param.
Ensure all param values are JSON serialiazable and raise a ParamValidationError if not.
Ensure parsed date and time values are RFC3339 compliant
41778Deprecations in airflow.models.dagrunYNNNNNNRemoved a set of deprecations in from airflow.models.dagrun.

Removed deprecated method DagRun.get_run(). Instead you should use standard Sqlalchemy DagRun model retrieval.
Removed deprecated method DagRun.get_log_filename_template(). Please use get_log_template() instead
41779Deprecations in airflow.models.errorsYNNNNNNRemove deprecated support for airflow.models.errors.ImportError which has been renamed to ParseImportError.
41780Deprecations in airflow.models.skipmixinYNNNNNNRemove deprecated support for passing execution_date to airflow.models.skipmixin.SkipMixin.skip().
41784Deprecations in airflow.models.taskinstanceYNNNNNNRemoved deprecated arg activate_dag_runs from TaskInstance.clear_task_instances(). Please use dag_run_state instead.
Removed deprecated arg execution_date from TaskInstance.__init__(). Please use run_id instead.
Removed deprecated property _try_number from TaskInstance. Please use try_number instead.
Removed deprecated property prev_attempted_tries from TaskInstance. Please use try_number instead.
Removed deprecated property next_try_number from TaskInstance. Please use try_number + 1 instead.
Removed deprecated property previous_ti from TaskInstance. Please use get_previous_ti instead.
Removed deprecated property previous_ti_success from TaskInstance. Please use get_previous_ti instead.
Removed deprecated property previous_start_date_success from TaskInstance. Please use get_previous_start_date instead.
Removed deprecated function as_dict from SimpleTaskInstance. Please use BaseSerialization.serialize instead.
Removed deprecated function from_dict from SimpleTaskInstance. Please use BaseSerialization.deserialize instead
41808Deprecations in airflow.models.taskrescheduleYNNNNNNRemoved deprecations in airflow.models.taskreschedule.

Removed methods:
query_for_task_instance()
find_for_task_instance()
Note: there are no replacements, if data is needed, you need to query via sqlalchemy
41857Min version of Pydantic changedNNNNNNYAirflow core now depends on Pydantic v2. If you have Pydantic v1 installed, please upgrade.
Update min version of Pydantic to 2.6.4
41910Deprecated method in api_connexion_securityYNNNNNNRemoved deprecated method requires_access from module airflow.api_connexion.security. Please use requires_access_* instead.
41964Command --tree flag from airflow tasks list NNNYNNN--tree flag for airflow tasks list command removed

The format of the output with that flag can be expensive to generate and extremely large, depending on the DAG. airflow dag show is a better way to visualize the relationship of tasks in a DAG.
DAG.tree_view and DAG.get_tree_view have also been removed
41975Deprecated metrics removedNYNNYNNMetrics basic deprecated validators (AllowListValidator and BlockListValidator) were removed in favor of pattern matching. Pattern matching validators (PatternAllowListValidator and PatternBlockListValidator) are enabled by default. Configuration parameter metrics_use_pattern_match``was removed from the ``metrics section
  • No labels