Status: Draft
Authors: Bolke de Bruin
Created: 2025-10-08
Target Airflow Version: 3.2.0
Related Issues: 

Abstract

This AIP proposes implementing a privacy-first, transparent, and community-governed telemetry system for Apache Airflow using the Apache Software Foundation's Matomo instance as the default collection endpoint. This addresses the complete absence of telemetry data since the removal of Scarf, while learning from past mistakes to ensure community trust.

Motivation

Currently, Airflow has no telemetry collection capability following the removal of Scarf. While this removal was necessary due to community concerns about privacy and transparency, it has created significant challenges:

Current Problems:

  1. No visibility into feature usage - Maintainers cannot determine which features are actually being used
  2. Difficult deprecation decisions - No data on which operators, providers, or Python versions are in active use
  3. Wasted development effort - Resources spent on features that may have minimal adoption
  4. Inability to prioritize - Cannot make data-driven decisions about which providers or features to maintain
  5. Performance optimization challenges - No understanding of real-world deployment patterns

Why telemetry is crucial for Airflow's development:

  1. Feature prioritization - Understanding how features are used helps prioritize development efforts
  2. Deprecation decisions - Knowing which Python versions, databases, or providers are in use prevents breaking production systems
  3. API design decisions - Usage patterns inform API design and breaking change decisions
  4. Performance optimization - Deployment patterns guide optimization efforts
  5. Provider maintenance - Understanding which providers are actively used informs maintenance priorities
  6. Community resource allocation - Data helps direct volunteer and sponsored development time effectively

Historical Context: The Airflow 2.10.0 release included opt-out telemetry via Scarf, which created significant community backlash due to:

This led to Scarf's removal. However, the need for usage insights remains critical for the project's long-term health and sustainability.

Goals

  1. Privacy by design - Collect only minimal, non-personal data with explicit user consent
  2. Transparency - Users can see exactly what data is collected and sent
  3. User control - Easy opt-in/opt-out with clear visibility into telemetry status
  4. Community governance - Changes to collected data require community approval
  5. Trust rebuilding - Demonstrate Airflow's commitment to user privacy and autonomy

Non-Goals

  1. Collecting personally identifiable information (PII)
  2. Collecting DAG names, task names, or other deployment-specific identifiers
  3. Collecting connection details, credentials, or sensitive configuration
  4. Real-time monitoring or performance profiling of individual deployments
  5. Commercial use of telemetry data

Proposal

Data Collection Endpoint

Default: Apache Software Foundation's Matomo instance (https://analytics.apache.org/)

Data Collected

The following minimal data will be collected when telemetry is enabled:

Installation Metrics

Usage Metrics

Aggregated Statistics (collected weekly)

Technical Metadata

Data NOT Collected

The following data will explicitly not be collected:

Consent Mechanism

First-Time Installation

When Airflow is started for the first time (or after upgrade to a version with this AIP):

  1. CLI Installation: A 10-second interactive prompt appears:

    ╔═══════════════════════════════════════════════════════════════╗
║                  Apache Airflow Telemetry                     ║
╠═══════════════════════════════════════════════════════════════╣
║                                                               ║
║  Airflow would like to collect anonymous usage data to help   ║
║  improve the project. This is entirely optional and can be    ║
║  disabled at any time.                                        ║
║                                                               ║
║  Data collected:                                              ║
║  • Airflow version and Python version                         ║
║  • Installed providers and operators used                     ║
║  • Deployment type and database backend                       ║
║  • Aggregate usage counts (no personal data)                  ║
║                                                               ║
║  Full details: https://airflow.apache.org/docs/telemetry      ║
║                                                               ║
║  Enable telemetry? [Y/n] (auto-decline in 10 seconds)         ║
╚═══════════════════════════════════════════════════════════════╝

  2. User Options:

  3. Non-interactive Mode: If stdin is not a TTY or --non-interactive flag is present, no prompt appears and telemetry defaults to disabled

Web UI Admin Screen

If no explicit choice has been recorded, the Admin UI displays a banner:

╭─────────────────────────────────────────────────────────────────╮
│ ⓘ  Airflow Telemetry Not Configured                            │
│                                                                 │
│ Help improve Airflow by sharing anonymous usage data.           │
│ [Learn More] [Enable Telemetry] [Disable Telemetry]             │
╰─────────────────────────────────────────────────────────────────╯

Telemetry Settings Page (Admin > Telemetry Settings):

Configuration

Database Storage

Telemetry preference is stored in the airflow_settings table:

INSERT INTO airflow_settings (key, value) VALUES 
  ('telemetry.enabled', 'true'),
  ('telemetry.installation_uuid', 'a1b2c3d4-e5f6-7890-1234-567890abcdef'),
  ('telemetry.consent_timestamp', '2025-10-08T12:34:56Z'),
  ('telemetry.last_sent_timestamp', '2025-10-08T13:00:00Z');

Configuration File

Users can override database settings via airflow.cfg:

[telemetry]
# Options: enabled, disabled, unset
# unset = defer to database setting (default)
enabled = unset

# Optional: Override collection endpoint
# Default: https://matomo.apache.org/
endpoint = https://analytics.apache.org/

# Optional: Custom installation UUID (for testing)
# installation_uuid = custom-uuid-here

# Collection interval in seconds (default: 86400 = daily)
collection_interval = 86400

Configuration Precedence:

  1. Environment variable AIRFLOW__TELEMETRY__ENABLED
  2. Configuration file [telemetry].enabled
  3. Database setting
  4. Default (disabled)

Data Transmission

Transmission Schedule

Transmission Method

Sample Payload

{
  "telemetry_version": "1.0",
  "timestamp": "2025-10-08T12:00:00Z",
  "airflow_version": "3.2.0",
  "python_version": "3.11",
  "deployment_type": "kubernetes",
  "os": "linux",
  "architecture": "x86_64",
  "database_backend": "postgres",
  "executor": "KubernetesExecutor",
  "providers": [
    {"name": "apache-airflow-providers-amazon", "version": "8.0.0"},
    {"name": "apache-airflow-providers-google", "version": "10.0.0"}
  ],
  "enabled_features": {
    "webserver_auth": true,
    "dag_serialization": true
  },
  "usage_stats": {
    "dag_count": 47,
    "task_count": 312,
    "dag_runs_7d": 1840,
    "task_instances_7d": 24576
  },
  "operator_usage": {
    "PythonOperator": 145,
    "BashOperator": 67,
    "S3ToRedshiftOperator": 23
  }
}

Governance and Changes

Process for Modifying Collected Data

Any changes to the data collected (additions or removals) require:

  1. AIP or GitHub Discussion: Proposal explaining the change and justification
  2. Dev List Vote: Lazy consensus vote on dev@airflow.apache.org (72-hour voting period)
  3. Documentation Update: Update telemetry documentation with exact fields
  4. Release Notes: Prominent mention in release notes under "Telemetry Changes" section
  5. In-App Notification: Users with telemetry enabled see a one-time notification in Web UI about data collection changes with option to review and opt-out

Telemetry Schema Versioning

Transparency and Data Access

Public Dashboard

A public dashboard will be created showing:

Dashboard URL: https://telemetry.airflow.apache.org/ (to be created)

Raw Data Access

Security Considerations

  1. No Authentication Required: Telemetry endpoint is unauthenticated (prevents tracking via auth tokens)
  2. Rate Limiting: Backend implements rate limiting per installation UUID to prevent abuse
  3. Schema Validation: All payloads are validated against JSON schema before processing
  4. Data Retention: Raw telemetry data retained for 2 years, then deleted; aggregates retained indefinitely
  5. ASF Infrastructure: Hosted on ASF infrastructure with ASF security policies
  6. HTTPS Only: All transmissions over TLS 1.2+
  7. No External Dependencies: Telemetry collection uses only Python standard library (except HTTP client)

Implementation Plan

Phase 1: Core Infrastructure (Airflow 3.2.0-alpha)

Phase 2: Documentation and Transparency (Airflow 3.2.0-beta)

Phase 3: Testing and Refinement (Airflow 3.2.0-rc)

Phase 4: Launch (Airflow 3.2.0)

Backward Compatibility

Removal of Scarf

Upgrade Experience

Users upgrading from Airflow 3.1.x or earlier to 3.2.0:

  1. On first 3.2.0 startup, see the telemetry consent prompt
  2. Release notes prominently explain new telemetry system and lessons learned from Scarf incident
  3. Admin UI shows telemetry banner until explicit choice is made
  4. Clear communication that this is opt-in and fundamentally different from the Scarf approach

Documentation

New documentation will be added:

  1. Main Documentation Page: /airflow-core/docs/telemetry.rst

  2. Admin Guide: /airflow-core/docs/administration-and-deployment/telemetry.rst

  3. Release Notes: Dedicated section in every release with telemetry changes

Alternatives Considered

Alternative 1: No Telemetry

Rationale for rejection: Development prioritization would rely solely on GitHub issues and surveys, which don't represent actual usage patterns.

Alternative 2: Opt-Out by Default

Rationale for rejection: Violates privacy-first principles and would erode community trust further given the history with Airflow 2.10.

Alternative 3: Self-Hosted Telemetry Endpoint

Rationale for rejection: Users could configure this, but default should be ASF-hosted for trust and convenience.

Alternative 4: Third-Party Service (e.g., PostHog, Segment)

Rationale for rejection: Using ASF infrastructure keeps data under project control and avoids third-party dependencies.

References

Open Questions

  1. Should telemetry be enabled for dev/test environments by default, or only production?
  2. What is the process for enterprises to share aggregate data without raw telemetry?
  3. Should we offer a "telemetry lite" mode with even less data for privacy-sensitive users?
  4. How do we handle telemetry in CI/CD environments where Airflow is started hundreds of times?

Risks and Mitigations

RiskMitigation
Low adoption rateProvide clear value proposition; show public dashboard early
Performance impactThorough testing; async transmission; graceful degradation
Privacy concernsPrivacy-first design; ASF hosting; full transparency
Community backlashEarly community engagement; clear communication; opt-in default
Data not usefulStart with minimal set; iterate based on actual needs
ASF infrastructure unavailableConfiguration allows alternative endpoints

Conclusion

This AIP proposes a privacy-first, transparent, and community-governed telemetry system that rebuilds trust while providing the data Airflow maintainers need to make informed development decisions. By defaulting to opt-in, using ASF infrastructure, and maintaining full transparency, we can achieve the right balance between user privacy and project sustainability.

Discussion: https://github.com/apache/airflow/discussions/XXXXX
Vote Thread: TBD after discussion period