Status: Draft
Authors: [Your Name]
Created: 2025-10-08
Target Airflow Version: 3.2.0
Related Issues: [Link to GitHub issue]

Abstract

This AIP proposes replacing Airflow's current telemetry implementation (Scarf) with a privacy-first, transparent, and community-governed telemetry system that uses Apache Software Foundation's Matomo instance as the default collection endpoint.

Motivation

Currently, Airflow uses Scarf for telemetry collection, which has raised several concerns:

  1. Privacy and GDPR compliance issues, particularly around data collection consent
  2. Limited transparency about what data is collected and how it's used
  3. Lack of user control over what information is shared
  4. Corporate adoption barriers - enterprise users face challenges enabling telemetry due to internal policies
  5. Trust erosion - the opt-out nature of previous implementations damaged community trust

However, telemetry data is crucial for Airflow's development:

  1. Feature prioritization - Understanding how features are used helps prioritize development efforts
  2. Error identification - Error reports help identify and fix issues more quickly
  3. API design decisions - Usage patterns inform API design and deprecation decisions
  4. Performance optimization - Performance metrics guide optimization efforts
  5. Provider maintenance - Understanding which providers are actively used informs maintenance priorities

The current situation has led to lower telemetry adoption, reducing the insights available to maintainers while also potentially creating privacy risks for users who do enable telemetry.

Goals

  1. Privacy by design - Collect only minimal, non-personal data with explicit user consent
  2. Transparency - Users can see exactly what data is collected and sent
  3. User control - Easy opt-in/opt-out with clear visibility into telemetry status
  4. Community governance - Changes to collected data require community approval
  5. Trust rebuilding - Demonstrate Airflow's commitment to user privacy and autonomy

Non-Goals

  1. Collecting personally identifiable information (PII)
  2. Collecting DAG names, task names, or other deployment-specific identifiers
  3. Collecting connection details, credentials, or sensitive configuration
  4. Real-time monitoring or performance profiling of individual deployments
  5. Commercial use of telemetry data

Proposal

Data Collection Endpoint

Default: Apache Software Foundation's Matomo instance (https://xxx)

Data Collected

The following minimal data will be collected when telemetry is enabled:

Installation Metrics

Usage Metrics

Aggregated Statistics (collected weekly)

Technical Metadata

Data NOT Collected

The following data will explicitly not be collected:

Consent Mechanism

First-Time Installation

When Airflow is started for the first time (or after upgrade to a version with this AIP):

  1. CLI Installation: A 10-second interactive prompt appears:

    ╔═══════════════════════════════════════════════════════════════╗
║                  Apache Airflow Telemetry                     ║
╠═══════════════════════════════════════════════════════════════╣
║                                                               ║
║  Airflow would like to collect anonymous usage data to help  ║
║  improve the project. This is entirely optional and can be   ║
║  disabled at any time.                                       ║
║                                                               ║
║  Data collected:                                             ║
║  • Airflow version and Python version                        ║
║  • Installed providers and operators used                    ║
║  • Deployment type and database backend                      ║
║  • Aggregate usage counts (no personal data)                 ║
║                                                               ║
║  Full details: https://airflow.apache.org/docs/telemetry     ║
║                                                               ║
║  Enable telemetry? [Y/n] (auto-decline in 10 seconds)       ║
╚═══════════════════════════════════════════════════════════════╝

  2. User Options:

  3. Non-interactive Mode: If stdin is not a TTY or --non-interactive flag is present, no prompt appears and telemetry defaults to disabled

Web UI Admin Screen

If no explicit choice has been recorded, the Admin UI displays a banner:

╭─────────────────────────────────────────────────────────────────╮
│ ⓘ  Airflow Telemetry Not Configured                            │
│                                                                 │
│ Help improve Airflow by sharing anonymous usage data.          │
│ [Learn More] [Enable Telemetry] [Disable Telemetry]            │
╰─────────────────────────────────────────────────────────────────╯

Telemetry Settings Page (Admin > Telemetry Settings):

Configuration

Database Storage

Telemetry preference is stored in the airflow_settings table:

INSERT INTO airflow_settings (key, value) VALUES 
  ('telemetry.enabled', 'true'),
  ('telemetry.installation_uuid', 'a1b2c3d4-e5f6-7890-1234-567890abcdef'),
  ('telemetry.consent_timestamp', '2025-10-08T12:34:56Z'),
  ('telemetry.last_sent_timestamp', '2025-10-08T13:00:00Z');

Configuration File

Users can override database settings via airflow.cfg:

[telemetry]
# Options: enabled, disabled, unset
# unset = defer to database setting (default)
enabled = unset

# Optional: Override collection endpoint
# Default: https://matomo.apache.org/
endpoint = https://matomo.apache.org/

# Optional: Custom installation UUID (for testing)
# installation_uuid = custom-uuid-here

# Collection interval in seconds (default: 86400 = daily)
collection_interval = 86400

Configuration Precedence:

  1. Environment variable AIRFLOW__TELEMETRY__ENABLED
  2. Configuration file [telemetry].enabled
  3. Database setting
  4. Default (disabled)

Data Transmission

Transmission Schedule

Transmission Method

Sample Payload

{
  "telemetry_version": "1.0",
  "timestamp": "2025-10-08T12:00:00Z",
  "installation_uuid": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "airflow_version": "3.2.0",
  "python_version": "3.11",
  "deployment_type": "kubernetes",
  "os": "linux",
  "architecture": "x86_64",
  "database_backend": "postgres",
  "executor": "KubernetesExecutor",
  "providers": [
    {"name": "apache-airflow-providers-amazon", "version": "8.0.0"},
    {"name": "apache-airflow-providers-google", "version": "10.0.0"}
  ],
  "enabled_features": {
    "webserver_auth": true,
    "dag_serialization": true
  },
  "usage_stats": {
    "dag_count": 47,
    "task_count": 312,
    "dag_runs_7d": 1840,
    "task_instances_7d": 24576
  },
  "operator_usage": {
    "PythonOperator": 145,
    "BashOperator": 67,
    "S3ToRedshiftOperator": 23
  }
}

Governance and Changes

Process for Modifying Collected Data

Any changes to the data collected (additions or removals) require:

  1. AIP or GitHub Discussion: Proposal explaining the change and justification
  2. Dev List Vote: Lazy consensus vote on dev@airflow.apache.org (72-hour voting period)
  3. Documentation Update: Update telemetry documentation with exact fields
  4. Release Notes: Prominent mention in release notes under "Telemetry Changes" section
  5. In-App Notification: Users with telemetry enabled see a one-time notification in Web UI about data collection changes with option to review and opt-out

Telemetry Schema Versioning

Transparency and Data Access

Public Dashboard

A public dashboard will be created showing:

Dashboard URL: https://telemetry.airflow.apache.org/ (to be created)

Raw Data Access

Security Considerations

  1. No Authentication Required: Telemetry endpoint is unauthenticated (prevents tracking via auth tokens)
  2. Rate Limiting: Backend implements rate limiting per installation UUID to prevent abuse
  3. Schema Validation: All payloads are validated against JSON schema before processing
  4. Data Retention: Raw telemetry data retained for 2 years, then deleted; aggregates retained indefinitely
  5. ASF Infrastructure: Hosted on ASF infrastructure with ASF security policies
  6. HTTPS Only: All transmissions over TLS 1.2+
  7. No External Dependencies: Telemetry collection uses only Python standard library (except HTTP client)

Implementation Plan

Phase 1: Core Infrastructure (Airflow 3.2.0-alpha)

Phase 2: Documentation and Transparency (Airflow 3.2.0-beta)

Phase 3: Testing and Refinement (Airflow 3.2.0-rc)

Phase 4: Launch (Airflow 3.2.0 GA)

Backward Compatibility

Removal of Scarf

Upgrade Experience

Users upgrading from Airflow 3.1.x or earlier to 3.2.0:

  1. On first 3.2.0 startup, see the telemetry consent prompt
  2. Release notes prominently explain new telemetry system
  3. Admin UI shows telemetry banner until explicit choice is made

Testing Strategy

  1. Unit Tests: All telemetry collection, transmission, and configuration logic
  2. Integration Tests: End-to-end telemetry flow with mock Matomo endpoint
  3. Privacy Tests: Verify no PII is collected or transmitted
  4. Security Tests: Attempt to inject malicious data or cause DoS
  5. Performance Tests: Ensure telemetry adds < 1ms to startup time and no runtime impact
  6. User Acceptance Testing: Community beta testing period

Documentation

New documentation will be added:

  1. Main Documentation Page: /docs/telemetry.rst

  2. Admin Guide: /docs/administration-and-deployment/telemetry.rst

  3. Contributing Guide: /docs/contributing/telemetry-changes.rst

  4. Release Notes: Dedicated section in every release with telemetry changes

Metrics for Success

Success of this AIP will be measured by:

  1. Adoption Rate: Target 15-20% opt-in rate within 6 months (up from essentially 0% post-2.10 removal)
  2. Community Trust: Reduction in GitHub issues/discussions complaining about telemetry
  3. Data Quality: Sufficient data to answer development prioritization questions
  4. Transparency: Public dashboard launch within 3 months of 3.2.0 release
  5. Enterprise Adoption: At least 2 major cloud providers participate (even if via aggregate data sharing)

Alternatives Considered

Alternative 1: No Telemetry

Rationale for rejection: Development prioritization would rely solely on GitHub issues and surveys, which don't represent actual usage patterns.

Alternative 2: Opt-Out by Default

Rationale for rejection: Violates privacy-first principles and would erode community trust further given the history with Airflow 2.10.

Alternative 3: Self-Hosted Telemetry Endpoint

Rationale for rejection: Users could configure this, but default should be ASF-hosted for trust and convenience.

Alternative 4: Third-Party Service (e.g., PostHog, Segment)

Rationale for rejection: Using ASF infrastructure keeps data under project control and avoids third-party dependencies.

References

Open Questions

  1. Should telemetry be enabled for dev/test environments by default, or only production?
  2. What is the process for enterprises to share aggregate data without raw telemetry?
  3. Should we offer a "telemetry lite" mode with even less data for privacy-sensitive users?
  4. How do we handle telemetry in CI/CD environments where Airflow is started hundreds of times?

Risks and Mitigations

RiskMitigation
Low adoption rateProvide clear value proposition; show public dashboard early
Performance impactThorough testing; async transmission; graceful degradation
Privacy concernsPrivacy-first design; ASF hosting; full transparency
Community backlashEarly community engagement; clear communication; opt-in default
Data not usefulStart with minimal set; iterate based on actual needs
ASF infrastructure unavailableConfiguration allows alternative endpoints

Conclusion

This AIP proposes a privacy-first, transparent, and community-governed telemetry system that rebuilds trust while providing the data Airflow maintainers need to make informed development decisions. By defaulting to opt-in, using ASF infrastructure, and maintaining full transparency, we can achieve the right balance between user privacy and project sustainability.

Discussion:
Vote Thread: TBD after discussion period