Page History

...

Our team of mentors are looking forward to hearing from you <3

Project Ideas

1. Apache Airflow architecture

Jira issue: AIRFLOW-4368

Project description

The Apache Airflow website does not have an architectural overview section. An overview would enable new contributors and users to develop a mental model of Apache Airflow and to start contributing sooner.

...

• What are the different components of Apache Airflow[1]?
• Which components are stateful or stateless?
• How does Apache Airflow distribute tasks to workers[2]?
• How can Apache Airflow run on different deployment architectures and databases?

Expected deliverables

• A page that describes the architecture
• The page should have detailed descriptions of each of the following components:
• Scheduler
• Web Server
• Worker
• Metadata DB
• The page should contain a diagram of the Apache Airflow architecture (e.g. [1])
• Description of how Apache Airflow schedules tasks[2]
• Detailed examples with diagrams and text [3]

Related resources

[1] https://imgur.com/a/YGpg5Wa

...

2. Deployment

Jira issue: AIRFLOW-4369

Project description

Apache Airflow automates and orchestrates complex workflows. It hides the complexity of managing dependencies between operators and scheduling tasks, enabling users to focus on the logic of their workflows.

...

• Cloud (AWS / GCP / Azure)
• On-premises
• Special attention can be given to the Kubernetes executor, as Kubernetes is a very popular technology to manage workloads.

Expected deliverables

• A page that introduces and describes deployment techniques
• A page that describes the deployment models and helps users choose the best one (full management, GKE-like service, PaaS - Astronomer/Google Composer)
• A page that helps users to choose the best executor
• A section that describes how to deploy Apache Airflow with Kubernetes. The section should include snippets for Kubernetes files, scripts, a PlantUML diagram for clarity.
• A section on running Apache Airflow on different cloud providers (AWS / Azure / GCP)
• A table comparing different executors

Related resources

[1] https://github.com/jghoman/awesome-apache-airflow

...

[3] https://gtoonstra.github.io/etl-with-airflow/

3. Testing

Jira issue: AIRFLOW-4370

Project description

Apache Airflow enables people to perform complex workflows that might affect many components in their infrastructure. It is important to be able to test an Apache Airflow workflow and ensure that it works as intended when run in a production environment.

...

Users who know best practices for creating Apache Airflow DAGs and using operators will be able to adopt Apache Airflow more easily and with fewer mishaps.

Expected deliverables

• A page or section that introduces testing workflows. The page should include information about the following testing stages:
• Unit tests: applies to one class
• DAG integrity tests: checks DAG code for missing variables, imports, etc.
• System tests
• Data tests: checks if the DAG performs its purpose
• A page for designing and testing DAGs and includes the following information:
• Tips and working examples on good practices for designing DAGs
• Descriptions on how to perform DAG dry-runs
• Descriptions on how to write unit tests for DAGs
• Snippets with working examples of DAGs and tests for them, including diagrams where possible
• A section on how to develop operators that are testable

Related resources

[1] https://github.com/jghoman/awesome-apache-airflow

...

4. How to create a workflow

Jira issue: AIRFLOW-4371

Project description

In Apache Airflow, workflows are saved as a code.  DAGs use operators to build complex workflow. A DAG is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies. A developer can describe the relationships in several ways. Task logic is saved in operators. Apache Airflow has operators that integrate with many services, but often developers need to write their own operators. Tasks can use the xcom metabase to communicate.

Expected deliverables

• A page for how to create a DAG that also includes:
• Revamping the page related to scheduling a DAG
• Adding tips for specific DAG conditions, such as rerunning a failed task
• A page for developing custom operators that includes:
• Describing mechanisms that are important when creating an operator, such as template fields, UI color, hooks, connection, etc.
• Describing the responsibility between the operator and the hook
• Considerations for dealing with shared resources (such as connections and hooks)
• A page that describes how to define the relationships between tasks. The page should include information about:
• >> <<
• set_upstream/set_downstream
• helpers method ex. chain
• A page that describes the communication between tasks that also includes:
• Revamping the page related to macros and XCOM

5. Documenting using local development environments

Jira issue: AIRFLOW-4372

Project description

Currently, people who want to join Apache Airflow Community and start contributing to Apache Airflow might find it very difficult to on-board. Setting up a local development environment is difficult. Depending on the level of testing needed, Apache Airflow might require manual setup of a combination of environment variables, external dependencies (Postgres, MySQL databases, Kerberos, and others), proper configuration and database initialization. Additionally, they have to know how to run the tests. There are scripts that run in CI environment to help, but the scripts are typically used for running a full set of tests not individual tests.

...

There are two relevant documents: CONTRIBUTING.md and BREEZE.rst. But we can think about different structure.

Expected deliverables

• A chapter or page of onboarding documentation that will be easy to find for new developers joining Apache Airflow community or someone who wants to start working on Apache Airflow development on a new PC. Ideally, the documentation could be a step-by-step guide, interactive tutorial, or a video guide—generally something easy to follow. Specifically, it should be clear that there are different local development environments from a local virtualenv through Docker image to full-blown replica of CI integration testing environment and that choosing one depends on your needs and experience level.

...

6. System maintenance

Jira issue: AIRFLOW-4373

Project description

Users rely on Apache Airflow to provide a reliable scheduler to orchestrate and run tasks. This means that an Airflow deployment should be resilient and low maintenance. This project involves documenting how to ensure a reliable deployment and maintain a healthy Apache Airflow Instance.

...

• Good practices on how to ensure continuous, trouble-free system operation
• Methods and mechanisms for system monitoring
• Description of the SLA mechanism, such as:
• Monitoring a running Apache Airflow instance and doing health checks, etc.
• Setting up Prometheus and Grafana, the two most common monitoring tools, to monitor Apache Airflow metrics

Expected deliverables

• Instructions and a step-by-step guide on how to set up monitoring for Apache Airflow, including Prometheus and Grafana (two most common monitoring tools)

...