...

Video of current problem: https://www.youtube.com/watch?v=sNrBruPS3r4

...

• Webserver Gunicorn processes read state from the DB instead of each webserver process managing its own state by reading the DAGs itself. This currently results in the weird disappearing and reappearing behaviour for several minutes after you make a change (e.g. add a new DAG). The DB will be the single source of truth.

• Once the scheduler processed the DAGit DAG it should be visible for all Gunicorn workers. The webserver will not scan the DAG files anymore, only use it for specific actions:

• Trigger; This requires the DAG to be processed once. This does not need to live in memory, and can be done on the fly.

• Code; Just to view the code, execution of the DAG is not required here.

Required changes

• The webserver should not use the DagBag but the metastore DB as the single source of truth. This will prevent DagBags in different Gunicorn workers to go out of sync with each other. Also the webserver does not need to process the DAG files anymore.
  Related JIRA issue: https://jira.apache.org/jira/browse/AIRFLOW-3562
  In order to achieve this all DAG components should be stored in the database. This is currently not the case so the following components should be persisted in the database:

• Biggest database change: DagEdges: https://jira.apache.org/jira/browse/AIRFLOW-3585

• Other information will be detected during working on this.

Edges in the database

Adding edges to the database is needed to be able to visualize the graph of a DAG.

...