Status: Draft
Authors: Bolke de Bruin
Created: 2025-10-08
Target Airflow Version: 3.2.0
Related Issues:

Abstract

This AIP proposes implementing a privacy-first, transparent, and community-governed telemetry system for Apache Airflow using the Apache Software Foundation's Matomo instance as the default collection endpoint. This addresses the complete absence of telemetry data since the removal of Scarf, while learning from past mistakes to ensure community trust.

Motivation

Currently, Airflow has no telemetry collection capability following the removal of Scarf. While this removal was necessary due to community concerns about privacy and transparency, it has created significant challenges:

Current Problems:

Why telemetry is crucial for Airflow's development:

Historical Context

The Airflow 2.10.0 release included opt-out telemetry via Scarf, which created significant community backlash due to:

This led to Scarf's removal. However, the need for usage insights remains critical for the project's long-term health and sustainability.

Goals

Non-Goals

Proposal

Data Collection Endpoint

Data Collected

The following minimal data will be collected when telemetry is enabled:

Installation Metrics (with noise)

Usage Metrics (aggregated and noised)

Aggregated Statistics (collected weekly with differential privacy)

Note: All counts below have Gaussian noise added (±10% standard deviation) to prevent exact fingerprinting while maintaining statistical usefulness.

Operator Usage (aggregated with noise)

Technical Metadata

Data NOT Collected

The following data will explicitly not be collected:

Consent Mechanism

Default Behavior: Opt-In

Following community feedback from the Scarf incident, telemetry is opt-in by default. While this typically yields lower participation rates (3-10%), it prioritizes user trust and explicit consent, which is critical for rebuilding community confidence after the previous telemetry issues.

First-Time Installation

When Airflow is started for the first time (or after upgrade to a version with this AIP):

CLI Installation: A 30-second interactive prompt appears when a TTY is detected:

╔══════════════════════════════════════════════════════════════════╗
║              Apache Airflow Anonymous Telemetry                  ║
╠══════════════════════════════════════════════════════════════════╣
║                                                                  ║
║  Help improve Airflow by sharing anonymous usage data.           ║
║                                                                  ║
║  We collect: Feature usage, performance metrics, errors          ║
║  We DON'T collect: Personal data, file contents, IPs             ║
║                                                                  ║
║  Privacy protections:                                            ║
║  • Differential privacy (noise added to all counts)              ║
║  • Daily rotating session IDs (no long-term tracking)            ║
║  • Open source implementation you can inspect                    ║
║                                                                  ║
║  View details: airflow.apache.org/docs/telemetry                 ║
║  Debug mode: airflow telemetry --debug                           ║
║                                                                  ║
║  Enable telemetry? [y/N] (auto-decline in 10 seconds)            ║
╚══════════════════════════════════════════════════════════════════╝

User Options:

Non-interactive Mode: If stdin is not a TTY or --non-interactive flag is present, telemetry defaults to disabled (unset)

Web UI Admin Screen

If no explicit choice has been recorded, the Admin UI displays a dismissible banner:

╭────────────────────────────────────────────────────────────────────╮
│ Help Improve Airflow                                               │
│                                                                    │
│ Share anonymous usage data to help us prioritize features.         │
│ [Learn More] [Enable Telemetry] [No Thanks]                        │
╰────────────────────────────────────────────────────────────────────╯

Telemetry Settings Page (Admin > Telemetry Settings):

Configuration

Database Storage

Telemetry preference is stored in the airflow_settings table:

INSERT INTO airflow_settings (key, value) VALUES 
  ('telemetry.enabled', 'true'),
  ('telemetry.daily_session_salt', 'random-daily-salt'),
  ('telemetry.consent_timestamp', '2025-10-08T12:34:56Z'),
  ('telemetry.last_sent_timestamp', '2025-10-08T13:00:00Z');

Note: No persistent installation UUID is stored to prevent long-term tracking.

Configuration File

Users can override database settings via airflow.cfg:

[telemetry]
# Options: true, false, unset
# unset = defer to database setting (default)
enabled = unset

# Optional: Override collection endpoint
# Default: https://analytics.apache.org/
endpoint = https://analytics.apache.org/

# Collection interval in seconds (default: 86400 = daily)
collection_interval = 86400

# Debug mode - print payloads to stdout instead of sending
debug = false

Environment Variables

# Disable telemetry
export AIRFLOW__TELEMETRY__ENABLED=false

# Enable debug mode to see payloads
export AIRFLOW__TELEMETRY__DEBUG=true

Configuration Precedence:

  1. Environment variable AIRFLOW__TELEMETRY__ENABLED
  2. Configuration file [telemetry].enabled
  3. Database setting
  4. Default (enabled with easy opt-out)

Five Ways to Control Telemetry

Following best practices, we provide multiple easy methods to enable or disable telemetry:

  1. Environment variable: export AIRFLOW__TELEMETRY__ENABLED=true or false
  2. Config file: Set enabled = true or false in [telemetry] section
  3. CLI command: airflow config set telemetry.enabled true or false
  4. Admin UI: Toggle in Telemetry Settings page
  5. Database: Update airflow_settings table directly

Data Transmission

Transmission Schedule

Transmission Method

Differential Privacy Implementation

All numeric counts have Gaussian noise added before transmission:

def add_noise(value: int, sensitivity: float = 0.1) -> int:
    """Add Gaussian noise to protect individual values while maintaining aggregate accuracy.
    
    sensitivity: Standard deviation as percentage of value (default 10%)
    """
    noise = random.gauss(0, value * sensitivity)
    return max(0, int(value + noise))

# Example: Report "approximately 47" not "exactly 47"
dag_count = add_noise(actual_dag_count, sensitivity=0.1)

Local Aggregation

Data is aggregated locally before transmission to minimize information leakage:

# Bad: Send every DAG run
for dag_run in dag_runs:
    send_telemetry(dag_run)

# Good: Aggregate to daily totals with noise
daily_stats = {
    'dag_runs_today': add_noise(len(dag_runs)),
    'success_rate': round(success_count / total_count, 2)
}
send_telemetry(daily_stats)

Sample Payload

{
  "telemetry_version": "1.0",
  "timestamp": "2025-10-08T12:00:00Z",
  "daily_session_id": "a7b9c2e4",
  "airflow_version": "3.2",
  "python_version": "3.11",
  "deployment_type": "kubernetes",
  "install_method": "helm",
  "os": "linux",
  "architecture": "x86_64",
  "database_backend": "postgres",
  "executor": "KubernetesExecutor",
  "providers": [
    {"name": "apache-airflow-providers-amazon", "version": "8.x"},
    {"name": "apache-airflow-providers-google", "version": "10.x"}
  ],
  "enabled_features": {
    "webserver_auth": true,
    "dag_serialization": true
  },
  "usage_stats": {
    "dag_count_tier": "51-100",
    "task_count_tier": "251-1000",
    "dag_runs_7d": "~2000",
    "task_instances_7d": "~25000",
    "active_hours_per_week": "100-168"
  },
  "operator_usage_top10": {
    "PythonOperator": "45%",
    "BashOperator": "22%",
    "S3ToRedshiftOperator": "8%"
  }
}

Transparency Features

Debug Mode

Users can inspect telemetry data before it's sent:

# See exactly what would be sent (without actually sending)
$ airflow telemetry --debug

Telemetry Debug Mode
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Telemetry Status: ENABLED
Endpoint: https://analytics.apache.org/
Next transmission: 2025-10-08 14:23:15 UTC

Payload Preview:
{
  "telemetry_version": "1.0",
  "airflow_version": "3.2",
  ...
}

This payload will be sent in 2 hours
Disable with: airflow config set telemetry.enabled false

Version Status

$ airflow version
Apache Airflow: 3.2.0
Python: 3.11.5
Telemetry: ENABLED (disable: airflow config set telemetry.enabled false)

$ airflow version --no-telemetry
Apache Airflow: 3.2.0
Python: 3.11.5
Running with telemetry disabled

Governance and Changes

Process for Modifying Collected Data

Any changes to the data collected (additions or removals) require:

  1. Discussion on Dev List: Proposal explaining the change and justification
  2. Dev List Vote: Lazy consensus vote on dev@airflow.apache.org (72-hour voting period)
  3. Documentation Update: Update telemetry documentation with exact fields
  4. Release Notes: Prominent mention in release notes under "Telemetry Changes" section
  5. In-App Notification: Users with telemetry enabled see a one-time notification in Web UI about data collection changes with option to review and opt-out
  6. Cli Notification: Users with telemetry enabled see a one-time cli notification with an option to review and opt-out

Telemetry Schema Versioning

Transparency and Data Access

Public Dashboard

A public dashboard will be created showing:

Dashboard URL: https://analytics.apache.org/ (to be created)

Raw Data Access

Open Source Implementation

All telemetry code will be:

Security Considerations

Our Telemetry Promise

We publicly commit to:

  1. We collect the minimum data necessary to improve Airflow
  2. We never collect personal information or correlate sessions across days
  3. We delete raw data after 90 days and only keep aggregates
  4. We will never sell or share individual-level data
  5. You can opt out anytime without feature degradation
  6. We add statistical noise to all counts to prevent fingerprinting
  7. We publish all telemetry code for community review

Backward Compatibility

Removal of Scarf

Upgrade Experience

Users upgrading from Airflow 3.1.x or earlier to 3.2.0:

Documentation

New documentation will be added:

Main Documentation Page: /docs/apache-airflow/telemetry.rst

Admin Guide: /docs/apache-airflow/administration-and-deployment/telemetry.rst

Release Notes

Dedicated section in every release with telemetry changes

Alternatives Considered

Alternative 1: No Telemetry

Rationale for rejection: Development prioritization would rely solely on GitHub issues and surveys, which don't represent actual usage patterns. This leads to poor prioritization decisions and wasted effort.

Alternative 2: Opt-Out by Default

Rationale for rejection: While opt-out would provide better data coverage (20-30% vs 3-10%), it violates the privacy-first principle that's critical for rebuilding trust after the Scarf incident. Given Airflow's history, explicit opt-in is the only approach that respects user autonomy and rebuilds community confidence. We accept lower participation rates as the cost of maintaining trust.

Alternative 3: Exact Counts Without Noise

Rationale for rejection: Exact counts could fingerprint individual installations and track them across sessions. Differential privacy with noise prevents this while maintaining statistical usefulness.

Alternative 4: Persistent Installation UUID

Rationale for rejection: A persistent UUID enables long-term tracking of individual installations. Daily rotating session IDs prevent correlation across days while still allowing daily aggregation.

Alternative 5: Self-Hosted Telemetry Endpoint Only

Rationale for rejection: Users can configure this, but default should be ASF-hosted for trust and convenience. Most users won't self-host.

Alternative 6: Third-Party Service (e.g., PostHog, Segment)

Rationale for rejection: Using ASF infrastructure keeps data under project control and avoids third-party dependencies. Builds more trust with community.

Addressing Common Concerns

"Why not make it opt-out?"

Given typical opt-in participation rates of 3-10%, opt-out would give us much better data coverage. However, after the Scarf incident, explicit user consent is critical for rebuilding trust. We prioritize community trust over data coverage and accept that we'll need creative approaches (surveys, optional extended telemetry programs) to supplement the lower opt-in rates.

"How do I verify what you're sending?"

Three ways:

  1. Run airflow telemetry --debug to see the exact payload before transmission
  2. Inspect network traffic (single HTTPS POST daily)
  3. Review our open source implementation at https://github.com/apache/airflow/tree/main/airflow/telemetry

"Will you sell this data?"

We legally commit in our telemetry promise to never selling individual-level data. We may share aggregate statistics like "40% of users use feature X" in our reports, but never individual installation data.

"Can you track me across reinstalls?"

No. We use daily rotating session IDs instead of persistent UUIDs. Each day gets a new random session ID that cannot be correlated with previous days.

"What about CI/CD environments?"

CI/CD environments should set AIRFLOW__TELEMETRY__ENABLED=false in their environment. The data would also naturally be filtered out on our end due to unusual patterns (hundreds of installations per day from same source).

Risks and Mitigations

RiskMitigation
Low adoption rateAccept 5-10% participation as cost of trust-building; supplement with targeted surveys and optional "extended telemetry" programs for willing users; focus on quality of insights over quantity
Community backlashExtensive pre-announcement community engagement; address Scarf learnings explicitly; privacy-first design with differential privacy; explicit opt-in approach prioritizes user trust over data coverage
Performance impactThorough testing; async transmission; 5s timeout; silent failures; zero impact on Airflow operations
Privacy concernsDifferential privacy; noise addition; daily rotating IDs; no persistent tracking; ASF hosting; open source implementation
Data not usefulStart minimal; noise level calibrated to maintain statistical significance while protecting privacy
ASF infrastructure unavailableConfiguration allows alternative endpoints; silent failure prevents impact
False positives in fingerprinting preventionNoise level calibrated through testing to prevent exact fingerprinting while maintaining aggregate accuracy

References

Open Questions

  1. Should we have different default behavior for development vs production environments (detected by database type or explicit flag)?
  2. What is the process for enterprises to share aggregate data without enabling raw telemetry?
  3. Should we offer a "telemetry lite" mode with even less data for highly privacy-sensitive users?
  4. What specific noise levels (sensitivity parameters) provide the best balance between privacy and utility?
  5. Should we implement a "trust score" showing telemetry participation rate to encourage adoption?

Conclusion

This AIP proposes a privacy-first, transparent, and community-governed telemetry system that rebuilds trust while providing the data Airflow maintainers need to make informed development decisions.

Key improvements over previous attempts:

By learning from the Scarf incident, implementing industry best practices for privacy protection, and prioritizing explicit user consent over data coverage, we can rebuild trust while still gathering valuable (though more limited) insights for project sustainability.

Discussion: https://github.com/apache/airflow/discussions/XXXXX
Vote Thread: TBD after discussion period