Page History

...

• When needing to change your start_date and schedule interval, change the name of the dag (a.k.a. dag_id) - I follow the convention : my_dag_v1, my_dag_v2, my_dag_v3, my_dag_v4, etc...
  • Changing schedule interval always requires changing the dag_id, because previously run TaskInstances will not align with the new schedule interval
  • Changing start_date without changing schedule_interval is safe, with some caveats.Changing but changing to an earlier start_date will not create any new DagRuns for the time between the new start_date and the old one, so tasks will not automatically backfill to the new dates.The scheduler must be restarted to pick up the new   If you manually create DagRuns, tasks will be scheduled, as long as the DagRun date is after both the task start_date and the dag start_date.
• When talking to MySQL, please use mysqlclient, not pymysql
• Using a start_date of datetime.now() can lead to unpredictable behavior, and your DAG never starting. See this post for details. It's recommended to subtract a timespan to force the scheduler to recognize the start_date. ***
• Solving MySQL Server has gone away
  • you may have an issue with your connection pool keeping connections open too long and checking out an old connection for you. To get a fresh connection, you can set the sql_alchemy_pool_recycle option in your configuration. Set this sql_alchemy_pool_recycle to the number of seconds of time in the connection pool between checkouts that you want a new connection to be created instead of an existing connection to be returned.
• If you intend to use extended ASCII or Unicode characters in Airflow, you have to provide a proper connection string to Airflow MySQL database, which defines charset explicitly, ex.:sql_alchemy_conn = mysql://airflow@localhost:3306/airflow?charset=utf8 Otherwise you will experience exceptions thrown by WTForms templating and other Airflow modules:UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 506: ordinal not in range(128)

...