Versions Compared

Old Version 3

changes.mady.by.user Unknown User (bolke)

Saved on

compared with

New Version 4

changes.mady.by.user Unknown User (bolke)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Status: Draft
Authors: [Your Name]Bolke de Bruin
Created: 2025-10-08
Target Airflow Version: 3.2.0
Related Issues: [Link to GitHub issue] 

Abstract

This AIP proposes replacing Airflow's current telemetry implementation (Scarf) with implementing a privacy-first, transparent, and community-governed telemetry system that uses for Apache Airflow using the Apache Software Foundation's Matomo instance as the default collection endpoint. This addresses the complete absence of telemetry data since the removal of Scarf, while learning from past mistakes to ensure community trust.

Motivation

Currently, Airflow uses Scarf for telemetry collection, which has raised several concerns:

  1. Privacy and GDPR compliance issues, particularly around data collection consent
  2. Limited transparency about what data is collected and how it's used
  3. Lack of user control over what information is shared
  4. Corporate adoption barriers - enterprise users face challenges enabling telemetry due to internal policies
  5. Trust erosion - the opt-out nature of previous implementations damaged community trust

However, telemetry data is crucial for Airflow's development:

  1. Feature prioritization - Understanding how features are used helps prioritize development efforts
  2. Error identification - Error reports help identify and fix issues more quickly
  3. API design decisions - Usage patterns inform API design and deprecation decisions
  4. Performance optimization - Performance metrics guide optimization efforts
  5. Provider maintenance - Understanding which providers are actively used informs maintenance priorities

The current situation has led to lower telemetry adoption, reducing the insights available to maintainers while also potentially creating privacy risks for users who do enable telemetry.

Goals

  1. Privacy by design - Collect only minimal, non-personal data with explicit user consent
  2. Transparency - Users can see exactly what data is collected and sent
  3. User control - Easy opt-in/opt-out with clear visibility into telemetry status
  4. Community governance - Changes to collected data require community approval
  5. Trust rebuilding - Demonstrate Airflow's commitment to user privacy and autonomy

Non-Goals

  1. Collecting personally identifiable information (PII)
  2. Collecting DAG names, task names, or other deployment-specific identifiers
  3. Collecting connection details, credentials, or sensitive configuration
  4. Real-time monitoring or performance profiling of individual deployments
  5. Commercial use of telemetry data

Proposal

Data Collection Endpoint

Default: Apache Software Foundation's Matomo instance (https://xxx)

has no telemetry collection capability following the removal of Scarf. While this removal was necessary due to community concerns about privacy and transparency, it has created significant challenges:

Current Problems:

  1. No visibility into feature usage - Maintainers cannot determine which features are actually being used
  2. Difficult deprecation decisions - No data on which operators, providers, or Python versions are in active use
  3. Wasted development effort - Resources spent on features that may have minimal adoption
  4. Inability to prioritize - Cannot make data-driven decisions about which providers or features to maintain
  5. Performance optimization challenges - No understanding of real-world deployment patterns

Why telemetry is crucial for Airflow's development:

  1. Feature prioritization - Understanding how features are used helps prioritize development efforts
  2. Deprecation decisions - Knowing which Python versions, databases, or providers are in use prevents breaking production systems
  3. API design decisions - Usage patterns inform API design and breaking change decisions
  4. Performance optimization - Deployment patterns guide optimization efforts
  5. Provider maintenance - Understanding which providers are actively used informs maintenance priorities
  6. Community resource allocation - Data helps direct volunteer and sponsored development time effectively

Historical Context: The Airflow 2.10.0 release included opt-out telemetry via Scarf, which created significant community backlash due to:

  • Lack of advance communication about the feature
  • Opt-out rather than opt-in approach
  • Limited transparency about data collection
  • Privacy concerns, even though no PII was collected
  • Corporate compliance challenges

This led to Scarf's removal. However, the need for usage insights remains critical for the project's long-term health and sustainability.

Goals

  1. Privacy by design - Collect only minimal, non-personal data with explicit user consent
  2. Transparency - Users can see exactly what data is collected and sent
  3. User control - Easy opt-in/opt-out with clear visibility into telemetry status
  4. Community governance - Changes to collected data require community approval
  5. Trust rebuilding - Demonstrate Airflow's commitment to user privacy and autonomy

Non-Goals

  1. Collecting personally identifiable information (PII)
  2. Collecting DAG names, task names, or other deployment-specific identifiers
  3. Collecting connection details, credentials, or sensitive configuration
  4. Real-time monitoring or performance profiling of individual deployments
  5. Commercial use of telemetry data

Proposal

Data Collection Endpoint

Default: Apache Software Foundation's Matomo instance (https://analytics.apache.org/)

  • Hosted and managed by the ASF
  • Complies with ASF privacy policies
  • Data is owned by
  • Hosted and managed by the ASF
  • Complies with ASF privacy policies
  • Data is owned by the Apache Airflow project
  • Users may configure alternative endpoints if required by organizational policies

...

The following minimal data will be collected when telemetry is enabled:

Installation Metrics

  • Installation UUID: A randomly generated identifier (UUID4) created at first startup
  • Airflow version: Full semantic Airflow version: Full semantic version (e.g., 23.101.0)
  • Python version: Major and minor version only (e.g., 3.11)
  • Deployment type: One of docker, kubernetes, systemd, standalone, unknown
  • Operating system: Generic OS type (e.g., linux, darwin, windows)
  • Architecture: System architecture (e.g., x86_64, arm64)

...

  • Active providers: List of installed provider packages and versions (e.g., apache-airflow-providers-amazon==8.0.0)
  • Executor type: Configured executor (e.g., LocalExecutor, CeleryExecutor, KubernetesExecutor)
  • Database backend: Database type only (e.g., postgres, mysql, sqlite)
  • Enabled features: Boolean flags for optional features (e.g., has_webserver_auth, has_dag_serialization)
  • Operator usage counts: Aggregated counts of operator types used (no task names or parameters)

...

  1. CLI Installation: A 10-second interactive prompt appears:

    ╔═══════════════════════════════════════════════════════════════╗
║                  Apache Airflow Telemetry                     ║
╠═══════════════════════════════════════════════════════════════╣
║                                                               ║
║  Airflow would like to collect anonymous usage data to help   ║
║  improve the project. This is entirely optional and can be    ║
║  disabled at any time.                                        ║
║                                                               ║
║  Data collected:                                              ║
║  • Airflow version and Python version                         ║
║  • Installed providers and operators used                     ║
║  • Deployment type and database backend                       ║
║  • Aggregate usage counts (no personal data)                  ║
║                                                               ║
║  Full details: https://airflow.apache.org/docs/telemetry      ║
║                                                               ║
║  Enable telemetry? [Y/n] (auto-decline in 10 seconds)         ║
╚═══════════════════════════════════════════════════════════════╝

  2. User Options:

    • Press Y or y: Enable telemetry
    • Press N or n: Disable telemetry
    • No input within 10 seconds: Default to disabled (not recorded as explicit choice)

  3. Non-interactive Mode: If stdin is not a TTY or --non-interactive flag is present, no prompt appears and telemetry defaults to disabled

...

╭─────────────────────────────────────────────────────────────────╮
│ ⓘ  Airflow Telemetry Not Configured                            │
│                                                                 │
│ Help improve Airflow by sharing anonymous usage data.           │
│ [Learn More] [Enable Telemetry] [Disable Telemetry]             │
╰─────────────────────────────────────────────────────────────────╯

...

[telemetry]
# Options: enabled, disabled, unset
# unset = defer to database setting (default)
enabled = unset

# Optional: Override collection endpoint
# Default: https://matomo.apache.org/
endpoint = https://matomoanalytics.apache.org/

# Optional: Custom installation UUID (for testing)
# installation_uuid = custom-uuid-here

# Collection interval in seconds (default: 86400 = daily)
collection_interval = 86400

...

{
  "telemetry_version": "1.0",
  "timestamp": "2025-10-08T12:00:00Z",
  "installation_uuid": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "airflow_version": "3.2.0",
  "python_version": "3.11",
  "deployment_type": "kubernetes",
  "os": "linux",
  "architecture": "x86_64",
  "database_backend": "postgres",
  "executor": "KubernetesExecutor",
  "providers": [
    {"name": "apache-airflow-providers-amazon", "version": "8.0.0"},
    {"name": "apache-airflow-providers-google", "version": "10.0.0"}
  ],
  "enabled_features": {
    "webserver_auth": true,
    "dag_serialization": true
  },
  "usage_stats": {
    "dag_count": 47,
    "task_count": 312,
    "dag_runs_7d": 1840,
    "task_instances_7d": 24576
  },
  "operator_usage": {
    "PythonOperator": 145,
    "BashOperator": 67,
    "S3ToRedshiftOperator": 23
  }
}

...

  • [ ] Community review period
  • [ ] Security audit of telemetry implementation
  • [ ] Performance testing (ensure no impact on Airflow operations)
  • [ ] Privacy review
  • [ ] Integration testing with Matomo

Phase 4: Launch (Airflow 3.2.0

...

)

  • [ ] Enable telemetry system in release
  • [ ] Launch public dashboard
  • [ ] Announcement blog post
  • [ ] Monitor adoption and feedback

...

  • Scarf telemetry was removed in Airflow 2.10.x 2 following community feedback about privacy and transparency concerns
  • Airflow currently has no telemetry capability since 2.10.2
  • This AIP proposes a completely new, privacy-first implementation
  • No migration of previous telemetry data to new systemUsers who had any previous telemetry enabled will need to explicitly opt-in to new telemetry

Upgrade Experience

Users upgrading from Airflow 3.1.x or earlier to 3.2.0:

  1. On first 3.2.0 startup, see the telemetry consent prompt
  2. Release notes prominently explain new telemetry system
  3. Admin UI shows telemetry banner until explicit choice is made

Testing Strategy

  1. Unit Tests: All telemetry collection, transmission, and configuration logic
  2. Integration Tests: End-to-end telemetry flow with mock Matomo endpoint
  3. Privacy Tests: Verify no PII is collected or transmitted
  4. Security Tests: Attempt to inject malicious data or cause DoS
  5. Performance Tests: Ensure telemetry adds < 1ms to startup time and no runtime impact
  6. User Acceptance Testing: Community beta testing period

Documentation

New documentation will be added:

  1. Main Documentation Page: /docs/telemetry.rst

    • What data is collected (comprehensive list)
    • How to enable/disable telemetry
    • Where data is sent
    • How data is used
    • Privacy policy

  2. Admin Guide: /docs/administration-and-deployment/telemetry.rst

    • Configuration options
    • Enterprise deployment considerations
    • Troubleshooting

  3. Contributing Guide: /docs/contributing/telemetry-changes.rst

    • Process for proposing data collection changes
    • Schema versioning guidelines

  4. Release Notes: Dedicated section in every release with telemetry changes

Metrics for Success

Success of this AIP will be measured by:

  1. new telemetry system and lessons learned from Scarf incident
  2. Admin UI shows telemetry banner until explicit choice is made
  3. Clear communication that this is opt-in and fundamentally different from the Scarf approach

Documentation

New documentation will be added:

  1. Main Documentation Page: /airflow-core/docs/telemetry.rst

    • What data is collected (comprehensive list)
    • How to enable/disable telemetry
    • Where data is sent
    • How data is used
    • Privacy policy

  2. Admin Guide: /airflow-core/docs/administration-and-deployment/telemetry.rst

    • Configuration options
    • Enterprise deployment considerations
    • Troubleshooting

  3. Release Notes: Dedicated section in every release with telemetry changes

  4. Adoption Rate: Target 15-20% opt-in rate within 6 months (up from essentially 0% post-2.10 removal)
  5. Community Trust: Reduction in GitHub issues/discussions complaining about telemetry
  6. Data Quality: Sufficient data to answer development prioritization questions
  7. Transparency: Public dashboard launch within 3 months of 3.2.0 release
  8. Enterprise Adoption: At least 2 major cloud providers participate (even if via aggregate data sharing)

Alternatives Considered

Alternative 1: No Telemetry

...

Rationale for rejection: Using ASF infrastructure keeps data under project control and avoids third-party dependencies.

References

...

Open Questions

  1. Should telemetry be enabled for dev/test environments by default, or only production?
  2. What is the process for enterprises to share aggregate data without raw telemetry?
  3. Should we offer a "telemetry lite" mode with even less data for privacy-sensitive users?
  4. How do we handle telemetry in CI/CD environments where Airflow is started hundreds of times?

...

This AIP proposes a privacy-first, transparent, and community-governed telemetry system that rebuilds trust while providing the data Airflow maintainers need to make informed development decisions. By defaulting to opt-in, using ASF infrastructure, and maintaining full transparency, we can achieve the right balance between user privacy and project sustainability.

...

Discussion: https://github.com/apache/airflow/discussions/XXXXX
Vote Thread: TBD after discussion period