Between 2017-06-10 and 2017-06-24, Gerard Toonstra ran a survey on the Apache Airflow developer mailing list to gain a better understanding how organizations are deploying Apache Airflow, how they are using it, the scale they are using it and some qualitative questions on problems they encounter. There were 29 respondents in total (1 duplicate) and this page documents the results of that survey.

The raw results of the survey (CSV) are available at the bottom of this page.

Question 1: What is the scale of your airflow deployment in number of worker cores?

Half of the respondents only use 1-5 worker cores, so very small setups. There are no respondents with a setup that has more than 500 cores, although there are 3 out of 29 that are between 50-500 worker nodes. This is not terribly surprising, because many organizations would choose Airflow to "orchestrate" the work by starting jobs on other services like EMR, Spark, etc. and not process all of that data on the airflow workers themselves. What we see is that from this sample of respondents, 75% are (probably) running this on a single beefy server with up to 50 cores or running it in a cluster of cooperating smaller VM's.

Question 2: How do you deploy (and synchronize) dags to all workers?

This was an interesting question and something I was personally very interested in. The answers are quite diverse in nature, we see deployments made from CI tools like jenkins, teamcity or travis, git deployments with webhooks, driven by cron (the irony! ), NFS shares, instances of S3 synchronization and deployment tools like Ansible. One response was more of a question "how to do this right", because it involved a lot of manual steps to stop workers and restart everything.

The majority of respondents have some way of interfacing with git for this. There were some responses that just state "docker", others looking at kubernetes to make the whole process more predictable through better control of container start/stop times. 

It's clear that there's no definite communicated "best practice" when it comes to this and it may be interesting for someone to dig deep into this topic and write an extended article about this some time how to do this in a way that minimizes scheduler + processes downtime, but allows for testing of new dags during the day. There are definitely some trade-offs in every method that was mentioned and it's good to understand each of them and figure out the optimal method.

Question 3: How do you enter and back up 'secrets' (connection strings, passwords, API keys, etc.) in airflow?

There are three main patterns here:

• Just using the airflow database connection table, so manually adding the secrets there
• Storing them elsewhere and populating the connections table using a separate script or DAG
• Environment variables, set up through ansible
  
• Storing them encrypted in git, then getting decrypted on deployment

Question 4: What is the profile of people in your organization who develop the DAGS?

The majority are data engineers and working in cooperation with data scientists. There are interesting cases where business analysts are contributing to this mix by writing the SQL views for the data with collaboration from data engineers.

Question 5: Do you use metadata/lineage tools that integrate with airflow? Which tools are they and how do you achieve this?

Almost no one uses data lineage / metadata tools with airflow. There are people looking into tools and know that this is a topic that at some point in the future will come up. For European organizations, this topic will become incredibly important in the next couple of months due to GDPR legislation.

The wikipedia article about data lineage: https://en.wikipedia.org/wiki/Data_lineage

Products mentioned are:

• Waterline data : https://www.waterlinedata.com/product-overview/
• Apache Atlas : http://atlas.incubator.apache.org/
• IBM Infosphere : https://www.ibm.com/analytics/us/en/technology/agile/
• Athena IT : http://athena-solutions.com/
• DataPortal : https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770

Question 6: What is the hardest thing you had to solve for airflow?

The responses here are very diverse. There are three topics that seem to bubble up from here:

• bugs
• understanding how the scheduler works
• approach to testing

Question 7: How do you get different teams working on airflow and only seeing their own DAGs?

The majority of organizations do not use multi-tenancy and are just using airflow within a single team. 7 out of 29 respondents use "n" deployments and 3 out of 29 use multi-tenancy options.

Question 8: Do you have a training program/bootcamp for new "DAG developers"?

Question 9: What did you solve with airflow that you're most proud of?

Here are some cool responses that demonstrate the value of airflow:

• "Robust ELT and workflows in general"
• "Getting off of our two bespoke, systems that did the rudiments of what Airflow does, but poorly and requiring more maintenance."
• "Made it much easier to create a cultural change in data science teams, toward traceable, repeatable jobs via Airflow, Docker, etc. Showing people Airflow makes it more obvious what the benefits of more disciplined practices are."
• "an entire company data pipeline"
• Having airflow run docker instances to allow for different execution environments

Question 10: What is the most irritable thing about airflow that you have to face on a daily basis?

 This is a grasp out of things that bug users the most:

• "Mailing list - not conducive for searching for answers"
  
• "There are still quite a few rough patches that we still run into, some of which are either actual bugs, configuration issues, or rough patches with running in containers on ECS"
• "debilitating number of bugs and regressions."
• "noticed that scheduler something dies or stops scheduling and needs to be restarted, so big concerns about reliability"
• "Bugs / Deadlocks / Scheduling"
• " Lack of security"

Raw survey responses can be downloaded here: