Page History

Status: Draft
Authors:

...

Bolke de Bruin
Created: 2025-10-08
Target Airflow Version: 3.2.0
Related Issues:

...

Abstract

This AIP proposes

...

implementing a privacy-first, transparent, and community-governed telemetry system

...

for Apache Airflow using the Apache Software Foundation's Matomo instance as the default collection endpoint. This addresses the complete absence of telemetry data since the removal of Scarf, while learning from past mistakes to ensure community trust.

Motivation

Currently, Airflow

...

Privacy and GDPR compliance issues, particularly around data collection consent
Limited transparency about what data is collected and how it's used
Lack of user control over what information is shared
Corporate adoption barriers - enterprise users face challenges enabling telemetry due to internal policies
Trust erosion - the opt-out nature of previous implementations damaged community trust

However, telemetry data is crucial for Airflow's development:

Feature prioritization - Understanding how features are used helps prioritize development efforts
Error identification - Error reports help identify and fix issues more quickly
API design decisions - Usage patterns inform API design and deprecation decisions
Performance optimization - Performance metrics guide optimization efforts
Provider maintenance - Understanding which providers are actively used informs maintenance priorities

The current situation has led to lower telemetry adoption, reducing the insights available to maintainers while also potentially creating privacy risks for users who do enable telemetry.

Goals

Privacy by design - Collect only minimal, non-personal data with explicit user consent
Transparency - Users can see exactly what data is collected and sent
User control - Easy opt-in/opt-out with clear visibility into telemetry status
Community governance - Changes to collected data require community approval
Trust rebuilding - Demonstrate Airflow's commitment to user privacy and autonomy

Non-Goals

Collecting personally identifiable information (PII)
Collecting DAG names, task names, or other deployment-specific identifiers
Collecting connection details, credentials, or sensitive configuration
Real-time monitoring or performance profiling of individual deployments
Commercial use of telemetry data

Proposal

Data Collection Endpoint

Default: Apache Software Foundation's Matomo instance (https://xxx)

Hosted and managed by the ASF
Complies with ASF privacy policies
Data is owned by the Apache Airflow project
Users may configure alternative endpoints if required by organizational policies

Data Collected

The following minimal data will be collected when telemetry is enabled:

Installation Metrics

Installation UUID: A randomly generated identifier (UUID4) created at first startup
Airflow version: Full semantic version (e.g., 2.10.0)
Python version: Major and minor version only (e.g., 3.11)
Deployment type: One of docker, kubernetes, systemd, standalone, unknown
Operating system: Generic OS type (e.g., linux, darwin, windows)
Architecture: System architecture (e.g., x86_64, arm64)

Usage Metrics

Active providers: List of installed provider packages and versions (e.g., apache-airflow-providers-amazon==8.0.0)
Executor type: Configured executor (e.g., LocalExecutor, CeleryExecutor, KubernetesExecutor)
Database backend: Database type only (e.g., postgres, mysql, sqlite)
Enabled features: Boolean flags for optional features (e.g., has_webserver_auth, has_dag_serialization)
Operator usage counts: Aggregated counts of operator types used (no task names or parameters)

Aggregated Statistics (collected weekly)

DAG count: Total number of DAGs (integer only)
Task count: Total number of tasks across all DAGs (integer only)
DAG run count: Total number of DAG runs in the past 7 days
Task instance count: Total number of task instances in the past 7 days

Technical Metadata

Timestamp: UTC timestamp of telemetry event
IP address: Used only for geolocation to country level, then immediately discarded (last octet zeroed before logging)

Data NOT Collected

The following data will explicitly not be collected:

DAG names, descriptions, or any DAG content
Task names, parameters, or configurations
Variable names or values
Connection names, URIs, or credentials
Log contents or error messages containing user data
User names, emails, or authentication information
Full IP addresses (only country-level geolocation)
Hostnames or deployment identifiers
Code or custom operator implementations
File paths or directory structures

Consent Mechanism

First-Time Installation

When Airflow is started for the first time (or after upgrade to a version with this AIP):

CLI Installation: A 10-second interactive prompt appears:

...

has no telemetry collection capability following the removal of Scarf. While this removal was necessary due to community concerns about privacy and transparency, it has created significant challenges:

Current Problems:

No visibility into feature usage - Maintainers cannot determine which features are actually being used
Difficult deprecation decisions - No data on which operators, providers, or Python versions are in active use
Wasted development effort - Resources spent on features that may have minimal adoption
Inability to prioritize - Cannot make data-driven decisions about which providers or features to maintain
Performance optimization challenges - No understanding of real-world deployment patterns

Why telemetry is crucial for Airflow's development:

Feature prioritization - Understanding how features are used helps prioritize development efforts
Deprecation decisions - Knowing which Python versions, databases, or providers are in use prevents breaking production systems
API design decisions - Usage patterns inform API design and breaking change decisions
Performance optimization - Deployment patterns guide optimization efforts
Provider maintenance - Understanding which providers are actively used informs maintenance priorities
Community resource allocation - Data helps direct volunteer and sponsored development time effectively

Historical Context

The Airflow 2.10.0 release included opt-out telemetry via Scarf, which created significant community backlash due to:

Lack of advance communication about the feature
Opt-out rather than opt-in approach
Limited transparency about data collection
Privacy concerns, even though no PII was collected
Corporate compliance challenges

This led to Scarf's removal. However, the need for usage insights remains critical for the project's long-term health and sustainability.

Goals

Privacy by design - Collect only minimal, non-personal data with differential privacy techniques
Transparency - Users can see exactly what data is collected and sent
User control - Easy opt-out with clear visibility into telemetry status
Community governance - Changes to collected data require community approval
Trust rebuilding - Demonstrate Airflow's commitment to user privacy and autonomy
Statistical usefulness - Collect enough data to make meaningful decisions (targeting >20% adoption)
No-impact on performance - Ensure that any data collection does not affect the performance of Airflow

Non-Goals

Collecting personally identifiable information (PII)
Collecting DAG names, task names, or other deployment-specific identifiers
Collecting connection details, credentials, or sensitive configuration
Real-time monitoring or performance profiling of individual deployments
Commercial use of telemetry data
Tracking individual users or installations over time

Proposal

Data Collection Endpoint

Default: Apache Software Foundation's Matomo instance (https://analytics.apache.org/)
- Hosted and managed by the ASF
- Complies with ASF privacy policies
- Data is owned by the Apache Airflow project
Alternative endpoints: Users may configure alternative endpoints if required by organizational policies

Data Collected

The following minimal data will be collected when telemetry is enabled:

Installation Metrics (with noise)

Airflow version: Major and minor version including patch version (e.g., 3.2.0)
Python version: Major and minor version only (e.g., 3.11)
Deployment type: One of docker, kubernetes, systemd, standalone, unknown
Operating system: Generic OS type (e.g., linux, darwin, windows)
Architecture: System architecture (e.g., x86_64, arm64)
Installation method: One of pip, docker, helm, managed_service, unknown

Usage Metrics (aggregated and noised)

Active providers: List of installed provider packages including version (e.g., apache-airflow-providers-amazon==8.x.y)
Executor type: Configured executor(s) (e.g., LocalExecutor, CeleryExecutor, KubernetesExecutor - only executors that are known to 'Official' Airflow otherwise "unspecified")
Database backend: Database type only (e.g., postgres, mysql, sqlite)
Enabled features: Boolean flags for optional features (e.g., has_webserver_auth, has_dag_serialization)

Aggregated Statistics (collected weekly with differential privacy)

Note: All counts below have Gaussian noise added (±10% standard deviation) to prevent exact fingerprinting while maintaining statistical usefulness.

DAG count tier: Bucketed as 1-10, 11-50, 51-100, 101-500, 501-1000, 1000+
Task count tier: Bucketed as 1-50, 51-250, 251-1000, 1001-5000, 5000+
DAG run count (7d): Rounded to nearest power of 10 (e.g., ~1000, ~10000)
Task instance count (7d): Rounded to nearest power of 10
Active hours per week: Bucketed as <10, 10-50, 50-100, 100-168 (helps understand CI vs production usage)

Operator Usage (aggregated with noise)

Top 10 operator only: Count of usage with ±5 noise added to each count
Percentages only: Instead of raw counts for privacy (e.g., "PythonOperator: 45% of tasks")
No parameters or task names

Technical Metadata

Timestamp: UTC timestamp rounded to the nearest hour
Session ID: Daily rotating hash (not persistent across days)
IP address: Used only for geolocation to country level, then immediately discarded (last two octets zeroed before logging)

Data NOT Collected

The following data will explicitly not be collected:

DAG names, descriptions, or any DAG content
Task names, parameters, or configurations
Variable names or values
Connection names, URIs, or credentials
Log contents or error messages containing user data
User names, emails, or authentication information
Full IP addresses (only country-level geolocation)
Hostnames or deployment identifiers
Installation UUIDs that persist across reinstalls
Code or custom operator implementations
File paths or directory structures
Exact counts that could fingerprint individual installations
Any correlation between daily sessions

Consent Mechanism

Default Behavior: Opt-In

Following community feedback from the Scarf incident, telemetry is opt-in by default. While this typically yields lower participation rates (3-10%), it prioritizes user trust and explicit consent, which is critical for rebuilding community confidence after the previous telemetry issues.

First-Time Installation

When Airflow is started for the first time (or after upgrade to a version with this AIP):

CLI Installation: A 30-second interactive prompt appears when a TTY is detected:

╔══════════════════════════════════════════════════════════════════╗
║              Apache Airflow Anonymous Telemetry

...

║
╠══════════════════════════════════════════════════════════════════╣
║

...

                 ║
║  Help improve Airflow by sharing anonymous usage data.           ║
║

...

              ║
║  We collect: Feature usage, performance metrics, errors          ║
║  We DON'T collect: Personal data, file contents, IPs             ║
║

...

                      ║
║  Privacy protections:                                            ║
║  • Differential privacy (noise added to all counts)              ║
║  • Daily rotating session IDs (no long-term tracking)            ║
║  • Open source implementation you can inspect                    ║
║                                                                  ║
║  View details: airflow.apache.org/docs/telemetry                 ║
║  Debug mode: airflow telemetry --debug                           ║
║                                                                  ║
║  Enable telemetry? [y/N] (auto-decline in 10 seconds)            ║
╚══════════════════════════════════════════════════════════════════╝

User Options:

Press Y or y: Enable telemetry
Press N or n or wait 10 seconds: Disable telemetry (default)
All choices are stored in database for persistence

Non-interactive Mode: If stdin is not a TTY or --non-interactive flag is present, telemetry defaults to disabled (unset)

Web UI Admin Screen

If no explicit choice has been recorded, the Admin UI displays a dismissible banner:

╭────────────────────────────────────────────────────────────────────╮
│ Help Improve Airflow                                               │
│                                                                    │
│ Share anonymous usage data to help us prioritize features.         │
│ [Learn More] [Enable Telemetry] [No Thanks]                        │
╰────────────────────────────────────────────────────────────────────╯

Telemetry Settings Page (Admin > Telemetry Settings):

Current status (Enabled/Disabled)
Last transmission timestamp
Preview of next data payload (viewable as JSON before sending)
Enable/Disable toggle with immediate effect
Link to full telemetry documentation
Export of all telemetry data sent in past 90 days (for GDPR compliance)
Telemetry test button (send test payload and show response)

Configuration

Database Storage

Telemetry preference is stored in the airflow_settings table:

INSERT INTO airflow_settings (key, value) VALUES 
  ('telemetry.enabled', 'true'),
  ('telemetry.daily_session_salt', 'random-daily-salt'),
  ('telemetry.consent_timestamp', '2025-10-08T12:34:56Z'),
  ('telemetry.last_sent_timestamp', '2025-10-08T13:00:00Z');

Note: No persistent installation UUID is stored to prevent long-term tracking.

Configuration File

Users can override database settings via airflow.cfg:

[telemetry]
# Options: true, false, unset
# unset = defer to database setting (default)
enabled = unset

# Optional: Override collection endpoint
# Default: https://analytics.apache.org/
endpoint = https://analytics.apache.org/

# Collection interval in seconds (default: 86400 = daily)
collection_interval = 86400

# Debug mode - print payloads to stdout instead of sending
debug = false

Environment Variables

# Disable telemetry
export AIRFLOW__TELEMETRY__ENABLED=false

# Enable debug mode to see payloads
export AIRFLOW__TELEMETRY__DEBUG=true

Configuration Precedence:

Environment variable AIRFLOW__TELEMETRY__ENABLED
Configuration file [telemetry].enabled
Database setting
Default (enabled with easy opt-out)

Five Ways to Control Telemetry

Following best practices, we provide multiple easy methods to enable or disable telemetry:

Environment variable: export AIRFLOW__TELEMETRY__ENABLED=true or false
Config file: Set enabled = true or false in [telemetry] section
CLI command: airflow config set telemetry.enabled true or false
Admin UI: Toggle in Telemetry Settings page
Database: Update airflow_settings table directly

Data Transmission

Transmission Schedule

Frequency: Once per day (configurable via collection_interval)
Time: Randomized within a 1-hour window to avoid thundering herd
Retry logic: Up to 3 retries with exponential backoff on failure
Timeout: 5-second timeout per request (fails silently if exceeded)
Graceful degradation: Failures are logged at DEBUG level only and never impact Airflow functionality

Transmission Method

Protocol: HTTPS POST to Matomo tracking API
Format: JSON payload
User-Agent: Apache-Airflow/{version} Telemetry/1.0
IP Anonymization: Last two octets zeroed before Matomo processing (dependent on ASF Infra)
No cookies: No tracking cookies or persistent identifiers
Daily session ID: Rotates daily using hash(date + random_salt) - cannot correlate across days

Differential Privacy Implementation

All numeric counts have Gaussian noise added before transmission:

def add_noise(value: int, sensitivity: float = 0.1) -> int:
    """Add Gaussian noise to protect individual values while maintaining aggregate accuracy.
    
    sensitivity: Standard deviation as percentage of value (default 10%)
    """
    noise = random.gauss(0, value * sensitivity)
    return max(0, int(value + noise))

# Example: Report "approximately 47" not "exactly 47"
dag_count = add_noise(actual_dag_count, sensitivity=0.1)

Local Aggregation

Data is aggregated locally before transmission to minimize information leakage:

# Bad: Send every DAG run
for dag_run in dag_runs:
    send_telemetry(dag_run)

# Good: Aggregate to daily totals with noise
daily_stats = {
    'dag_runs_today': add_noise(len(dag_runs)),
    'success_rate': round(success_count / total_count, 2)
}
send_telemetry(daily_stats)

Sample Payload

{
  "telemetry_version": "1.0",
  "timestamp": "2025-10-08T12:00:00Z",
  "daily_session_id": "a7b9c2e4",
  "airflow_version": "3.2",
  "python_version": "3.11",
  "deployment_type": "kubernetes",
  "install_method": "helm",
  "os": "linux",
  "architecture": "x86_64",
  "database_backend": "postgres",
  "executor": "KubernetesExecutor",
  "providers": [
    {"name": "apache-airflow-providers-amazon", "version": "8.x"},
    {"name": "apache-airflow-providers-google", "version": "10.x"}
  ],
  "enabled_features": {
    "webserver_auth": true,
    "dag_serialization": true
  },
  "usage_stats": {
    "dag_count_tier": "51-100",
    "task_count_tier": "251-1000",
    "dag_runs_7d": "~2000",
    "task_instances_7d": "~25000",
    "active_hours_per_week": "100-168"
  },
  "operator_usage_top10": {
    "PythonOperator": "45%",
    "BashOperator": "22%",
    "S3ToRedshiftOperator": "8%"
  }
}

Transparency Features

Debug Mode

Users can inspect telemetry data before it's sent:

# See exactly what would be sent (without actually sending)
$ airflow telemetry --debug

Telemetry Debug Mode
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Telemetry Status: ENABLED
Endpoint: https://analytics.apache.org/
Next transmission: 2025-10-08 14:23:15 UTC

Payload Preview:
{
  "telemetry_version": "1.0",
  "airflow_version": "3.2",
  ...
}

This payload will be sent in 2 hours
Disable with: airflow config set telemetry.enabled false

Version Status

$ airflow version
Apache Airflow: 3.2.0
Python: 3.11.5
Telemetry: ENABLED (disable: airflow config set telemetry.enabled false)

$ airflow version --no-telemetry
Apache Airflow: 3.2.0
Python: 3.11.5
Running with telemetry disabled

Governance and Changes

Process for Modifying Collected Data

Any changes to the data collected (additions or removals) require:

Discussion on Dev List: Proposal explaining the change and justification
Dev List Vote: Lazy consensus vote on dev@airflow.apache.org (72-hour voting period)
Documentation Update: Update telemetry documentation with exact fields
Release Notes: Prominent mention in release notes under "Telemetry Changes" section
In-App Notification: Users with telemetry enabled see a one-time notification in Web UI about data collection changes with option to review and opt-out
Cli Notification: Users with telemetry enabled see a one-time cli notification with an option to review and opt-out

Telemetry Schema Versioning

Telemetry payloads include a telemetry_version field
Breaking changes increment the major version
Additive changes increment the minor version
Older Airflow versions continue sending their schema version
Backend supports multiple schema versions simultaneously

Transparency and Data Access

Public Dashboard

A public dashboard will be created showing:

Aggregate statistics (total installations, version distribution)
Provider popularity
Executor type distribution
Database backend distribution
Deployment type breakdown
Geographic distribution (country level only)
Python version adoption over time

Dashboard URL: https://analytics.apache.org/ (to be created)

Raw Data Access

Aggregated, anonymized data will be made available as quarterly CSV exports
Individual installation data will never be published
Large deployment users (AWS, Google, Astronomer, etc.) may request access to aggregate insights for comparison purposes if they share telemetry data

Open Source Implementation

All telemetry code will be:

Located in a single module: airflow/telemetry/ for easy inspection
Thoroughly commented explaining privacy decisions
Linked prominently in documentation
Reviewable by privacy advocates before release

Security Considerations

No Authentication Required: Telemetry endpoint is unauthenticated (prevents tracking via auth tokens) - access is through greylisting if possible (check with ASF Infra)
Rate Limiting: Backend needs to implement rate limiting per daily session ID to prevent abuse (dependent on ASF)
Schema Validation: All payloads are validated against JSON schema before processing
Data Retention: Raw telemetry data retained for 90 days, then deleted; aggregates retained indefinitely
ASF Infrastructure: Hosted on ASF infrastructure with ASF security policies
HTTPS Only: All transmissions over TLS 1.2+
No External Dependencies: Telemetry collection uses only Python standard library (except HTTP client)
Silent Failures: Network errors, timeouts, and transmission failures never raise exceptions or log above DEBUG level

Our Telemetry Promise

We publicly commit to:

We collect the minimum data necessary to improve Airflow
We never collect personal information or correlate sessions across days
We delete raw data after 90 days and only keep aggregates
We will never sell or share individual-level data
You can opt out anytime without feature degradation
We add statistical noise to all counts to prevent fingerprinting
We publish all telemetry code for community review

Backward Compatibility

Removal of Scarf

Scarf telemetry was removed in Airflow 2.10.2 following community feedback
Airflow currently has no telemetry capability since 2.10.2
This AIP proposes a completely new, privacy-first implementation
No migration of previous telemetry data to new system

Upgrade Experience

Users upgrading from Airflow 3.1.x or earlier to 3.2.0:

On first 3.2.0 startup, see the telemetry consent prompt
Release notes prominently explain new telemetry system and lessons learned from Scarf
Admin UI shows telemetry status banner (dismissible for 30 days)
Clear communication about five easy ways to opt out
Emphasis on differential privacy and transparency improvements

Documentation

New documentation will be added:

Main Documentation Page: `/docs/apache-airflow/telemetry.rst`

What data is collected (comprehensive list with examples)
How we protect privacy (differential privacy, noise, bucketing)
Five ways to enable or disable telemetry
Where data is sent and who controls it
How data is used
Our telemetry promise
Debug mode instructions
FAQ addressing common concerns
Why opt-in and how we handle low participation rates

Admin Guide: `/docs/apache-airflow/administration-and-deployment/telemetry.rst`

Configuration options
Enterprise deployment considerations
Corporate compliance guidance
Troubleshooting
Alternative endpoint configuration

Release Notes

Dedicated section in every release with telemetry changes

Alternatives Considered

Alternative 1: No Telemetry

Rationale for rejection: Development prioritization would rely solely on GitHub issues and surveys, which don't represent actual usage patterns. This leads to poor prioritization decisions and wasted effort.

Alternative 2: Opt-Out by Default

Rationale for rejection: While opt-out would provide better data coverage (20-30% vs 3-10%), it violates the privacy-first principle that's critical for rebuilding trust after the Scarf incident. Given Airflow's history, explicit opt-in is the only approach that respects user autonomy and rebuilds community confidence. We accept lower participation rates as the cost of maintaining trust.

Alternative 3: Exact Counts Without Noise

Rationale for rejection: Exact counts could fingerprint individual installations and track them across sessions. Differential privacy with noise prevents this while maintaining statistical usefulness.

Alternative 4: Persistent Installation UUID

Rationale for rejection: A persistent UUID enables long-term tracking of individual installations. Daily rotating session IDs prevent correlation across days while still allowing daily aggregation.

Alternative 5: Self-Hosted Telemetry Endpoint Only

Rationale for rejection: Users can configure this, but default should be ASF-hosted for trust and convenience. Most users won't self-host.

Alternative 6: Third-Party Service (e.g., PostHog, Segment)

Rationale for rejection: Using ASF infrastructure keeps data under project control and avoids third-party dependencies. Builds more trust with community.

Addressing Common Concerns

"Why not make it opt-out?"

Given typical opt-in participation rates of 3-10%, opt-out would give us much better data coverage. However, after the Scarf incident, explicit user consent is critical for rebuilding trust. We prioritize community trust over data coverage and accept that we'll need creative approaches (surveys, optional extended telemetry programs) to supplement the lower opt-in rates.

"How do I verify what you're sending?"

Three ways:

Run airflow telemetry --debug to see the exact payload before transmission
Inspect network traffic (single HTTPS POST daily)
Review our open source implementation at https://github.com/apache/airflow/tree/main/airflow/telemetry

"Will you sell this data?"

We legally commit in our telemetry promise to never selling individual-level data. We may share aggregate statistics like "40% of users use feature X" in our reports, but never individual installation data.

"Can you track me across reinstalls?"

No. We use daily rotating session IDs instead of persistent UUIDs. Each day gets a new random session ID that cannot be correlated with previous days.

"What about CI/CD environments?"

CI/CD environments should set AIRFLOW__TELEMETRY__ENABLED=false in their environment. The data would also naturally be filtered out on our end due to unusual patterns (hundreds of installations per day from same source).

Risks and Mitigations

Risk	Mitigation
Low adoption rate	Accept 5-10% participation as cost of trust-building; supplement with targeted surveys and optional "extended telemetry" programs for willing users; focus on quality of insights over quantity
Community backlash	Extensive pre-announcement community engagement; address Scarf learnings explicitly; privacy-first design with differential privacy; explicit opt-in approach prioritizes user trust over data coverage
Performance impact	Thorough testing; async transmission; 5s timeout; silent failures; zero impact on Airflow operations
Privacy concerns	Differential privacy; noise addition; daily rotating IDs; no persistent tracking; ASF hosting; open source implementation
Data not useful	Start minimal; noise level calibrated to maintain statistical significance while protecting privacy
ASF infrastructure unavailable	Configuration allows alternative endpoints; silent failure prevents impact
False positives in fingerprinting prevention	Noise level calibrated through testing to prevent exact fingerprinting while maintaining aggregate accuracy

References

...

User Options:

Press Y or y: Enable telemetry
Press N or n: Disable telemetry
No input within 10 seconds: Default to disabled (not recorded as explicit choice)

...

Non-interactive Mode: If stdin is not a TTY or --non-interactive flag is present, no prompt appears and telemetry defaults to disabled

Web UI Admin Screen

If no explicit choice has been recorded, the Admin UI displays a banner:

╭─────────────────────────────────────────────────────────────────╮
│ ⓘ  Airflow Telemetry Not Configured                            │
│                                                                 │
│ Help improve Airflow by sharing anonymous usage data.          │
│ [Learn More] [Enable Telemetry] [Disable Telemetry]            │
╰─────────────────────────────────────────────────────────────────╯

Telemetry Settings Page (Admin > Telemetry Settings):

Current status (Enabled/Disabled/Not Configured)
Last transmission timestamp
Summary of last data sent (viewable as JSON)
Enable/Disable toggle
Link to full telemetry documentation
Export of all telemetry data sent (for GDPR compliance)

Configuration

Database Storage

Telemetry preference is stored in the airflow_settings table:

INSERT INTO airflow_settings (key, value) VALUES 
  ('telemetry.enabled', 'true'),
  ('telemetry.installation_uuid', 'a1b2c3d4-e5f6-7890-1234-567890abcdef'),
  ('telemetry.consent_timestamp', '2025-10-08T12:34:56Z'),
  ('telemetry.last_sent_timestamp', '2025-10-08T13:00:00Z');

Configuration File

Users can override database settings via airflow.cfg:

[telemetry]
# Options: enabled, disabled, unset
# unset = defer to database setting (default)
enabled = unset

# Optional: Override collection endpoint
# Default: https://matomo.apache.org/
endpoint = https://matomo.apache.org/

# Optional: Custom installation UUID (for testing)
# installation_uuid = custom-uuid-here

# Collection interval in seconds (default: 86400 = daily)
collection_interval = 86400

Configuration Precedence:

Environment variable AIRFLOW__TELEMETRY__ENABLED
Configuration file [telemetry].enabled
Database setting
Default (disabled)

Data Transmission

Transmission Schedule

Frequency: Once per day (configurable via collection_interval)
Time: Randomized within a 1-hour window to avoid thundering herd
Retry logic: Up to 3 retries with exponential backoff on failure
Timeout: 10-second timeout per request
Graceful degradation: Failures are logged but do not impact Airflow functionality

Transmission Method

Protocol: HTTPS POST to Matomo tracking API
Format: JSON payload
User-Agent: Apache-Airflow/{version} Telemetry/1.0
IP Anonymization: Last octet zeroed before Matomo processing
No cookies: No tracking cookies or persistent identifiers beyond installation UUID

Sample Payload

{
  "telemetry_version": "1.0",
  "timestamp": "2025-10-08T12:00:00Z",
  "installation_uuid": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "airflow_version": "3.2.0",
  "python_version": "3.11",
  "deployment_type": "kubernetes",
  "os": "linux",
  "architecture": "x86_64",
  "database_backend": "postgres",
  "executor": "KubernetesExecutor",
  "providers": [
    {"name": "apache-airflow-providers-amazon", "version": "8.0.0"},
    {"name": "apache-airflow-providers-google", "version": "10.0.0"}
  ],
  "enabled_features": {
    "webserver_auth": true,
    "dag_serialization": true
  },
  "usage_stats": {
    "dag_count": 47,
    "task_count": 312,
    "dag_runs_7d": 1840,
    "task_instances_7d": 24576
  },
  "operator_usage": {
    "PythonOperator": 145,
    "BashOperator": 67,
    "S3ToRedshiftOperator": 23
  }
}

Governance and Changes

Process for Modifying Collected Data

Any changes to the data collected (additions or removals) require:

AIP or GitHub Discussion: Proposal explaining the change and justification
Dev List Vote: Lazy consensus vote on dev@airflow.apache.org (72-hour voting period)
Documentation Update: Update telemetry documentation with exact fields
Release Notes: Prominent mention in release notes under "Telemetry Changes" section
In-App Notification: Users with telemetry enabled see a one-time notification in Web UI about data collection changes with option to review and opt-out

Telemetry Schema Versioning

Telemetry payloads include a telemetry_version field
Breaking changes increment the major version
Additive changes increment the minor version
Older Airflow versions continue sending their schema version
Backend supports multiple schema versions simultaneously

Transparency and Data Access

Public Dashboard

A public dashboard will be created showing:

Aggregate statistics (total installations, version distribution)
Provider popularity
Executor type distribution
Database backend distribution
Deployment type breakdown
Geographic distribution (country level only)

Dashboard URL: https://telemetry.airflow.apache.org/ (to be created)

Raw Data Access

Aggregated, anonymized data will be made available as quarterly CSV exports
Individual installation data will never be published
Large deployment users (AWS, Google, Astronomer, etc.) may request access to aggregate insights for comparison purposes

Security Considerations

No Authentication Required: Telemetry endpoint is unauthenticated (prevents tracking via auth tokens)
Rate Limiting: Backend implements rate limiting per installation UUID to prevent abuse
Schema Validation: All payloads are validated against JSON schema before processing
Data Retention: Raw telemetry data retained for 2 years, then deleted; aggregates retained indefinitely
ASF Infrastructure: Hosted on ASF infrastructure with ASF security policies
HTTPS Only: All transmissions over TLS 1.2+
No External Dependencies: Telemetry collection uses only Python standard library (except HTTP client)

Implementation Plan

Phase 1: Core Infrastructure (Airflow 3.2.0-alpha)

[ ] Implement telemetry data collection module
[ ] Add database schema for telemetry settings
[ ] Create CLI prompt for first-time setup
[ ] Implement configuration file parsing
[ ] Add basic transmission logic with Matomo integration
[ ] Create admin UI for telemetry management

Phase 2: Documentation and Transparency (Airflow 3.2.0-beta)

[ ] Complete telemetry documentation page
[ ] Set up public dashboard infrastructure
[ ] Create data export functionality for GDPR compliance
[ ] Add release notes and upgrade guide
[ ] Implement in-app changelog for data collection changes

Phase 3: Testing and Refinement (Airflow 3.2.0-rc)

[ ] Community review period
[ ] Security audit of telemetry implementation
[ ] Performance testing (ensure no impact on Airflow operations)
[ ] Privacy review
[ ] Integration testing with Matomo

Phase 4: Launch (Airflow 3.2.0 GA)

[ ] Enable telemetry system in release
[ ] Launch public dashboard
[ ] Announcement blog post
[ ] Monitor adoption and feedback

Backward Compatibility

Removal of Scarf

Scarf telemetry was removed in Airflow 2.10.x following community feedback
No migration of previous telemetry data to new system
Users who had any previous telemetry enabled will need to explicitly opt-in to new telemetry

Upgrade Experience

Users upgrading from Airflow 3.1.x or earlier to 3.2.0:

On first 3.2.0 startup, see the telemetry consent prompt
Release notes prominently explain new telemetry system
Admin UI shows telemetry banner until explicit choice is made

Testing Strategy

Unit Tests: All telemetry collection, transmission, and configuration logic
Integration Tests: End-to-end telemetry flow with mock Matomo endpoint
Privacy Tests: Verify no PII is collected or transmitted
Security Tests: Attempt to inject malicious data or cause DoS
Performance Tests: Ensure telemetry adds < 1ms to startup time and no runtime impact
User Acceptance Testing: Community beta testing period

Documentation

New documentation will be added:

Main Documentation Page: /docs/telemetry.rst
- What data is collected (comprehensive list)
- How to enable/disable telemetry
- Where data is sent
- How data is used
- Privacy policy
Admin Guide: /docs/administration-and-deployment/telemetry.rst
- Configuration options
- Enterprise deployment considerations
- Troubleshooting
Contributing Guide: /docs/contributing/telemetry-changes.rst
- Process for proposing data collection changes
- Schema versioning guidelines
Release Notes: Dedicated section in every release with telemetry changes

Metrics for Success

Success of this AIP will be measured by:

Adoption Rate: Target 15-20% opt-in rate within 6 months (up from essentially 0% post-2.10 removal)
Community Trust: Reduction in GitHub issues/discussions complaining about telemetry
Data Quality: Sufficient data to answer development prioritization questions
Transparency: Public dashboard launch within 3 months of 3.2.0 release
Enterprise Adoption: At least 2 major cloud providers participate (even if via aggregate data sharing)

Alternatives Considered

Alternative 1: No Telemetry

Rationale for rejection: Development prioritization would rely solely on GitHub issues and surveys, which don't represent actual usage patterns.

Alternative 2: Opt-Out by Default

Rationale for rejection: Violates privacy-first principles and would erode community trust further given the history with Airflow 2.10.

Alternative 3: Self-Hosted Telemetry Endpoint

Rationale for rejection: Users could configure this, but default should be ASF-hosted for trust and convenience.

Alternative 4: Third-Party Service (e.g., PostHog, Segment)

Rationale for rejection: Using ASF infrastructure keeps data under project control and avoids third-party dependencies.

References

...

GDPR Compliance Guidelines
ASF Privacy Policy
Telemetry Best Practices for Open Source
Airflow 2.10.0 Release Notes - Scarf
Previous discussion:

[GitHub Issue #XXXXX]

Open Questions

Should telemetry be enabled for dev/test environments by default, or only productionwe have different default behavior for development vs production environments (detected by database type or explicit flag)?
What is the process for enterprises to share aggregate data without enabling raw telemetry?
Should we offer a "telemetry lite" mode with even less data for highly privacy-sensitive users?
How do we handle telemetry in CI/CD environments where Airflow is started hundreds of times?

Risks and Mitigations

Risk	Mitigation
Low adoption rate	Provide clear value proposition; show public dashboard early
Performance impact	Thorough testing; async transmission; graceful degradation
Privacy concerns	Privacy-first design; ASF hosting; full transparency
Community backlash	Early community engagement; clear communication; opt-in default
Data not useful	Start with minimal set; iterate based on actual needs
ASF infrastructure unavailable	Configuration allows alternative endpoints

Conclusion

This AIP proposes a privacy-first, transparent, and community-governed telemetry system that rebuilds trust while providing the data Airflow maintainers need to make informed development decisions. By defaulting to opt-in, using ASF infrastructure, and maintaining full transparency, we can achieve the right balance between user privacy and project sustainability.

What specific noise levels (sensitivity parameters) provide the best balance between privacy and utility?
Should we implement a "trust score" showing telemetry participation rate to encourage adoption?

Conclusion

This AIP proposes a privacy-first, transparent, and community-governed telemetry system that rebuilds trust while providing the data Airflow maintainers need to make informed development decisions.

Key improvements over previous attempts:

Explicit opt-in: Users must actively consent, rebuilding trust after Scarf incident
Differential privacy: Noise added to all counts prevents fingerprinting
No long-term tracking: Daily rotating session IDs prevent correlation across days
Five control methods: Makes enabling or disabling telemetry easy
Debug mode: Complete transparency into what's being sent
Open source implementation: All code reviewable by privacy advocates
Public legal commitment: Our telemetry promise is binding

By learning from the Scarf incident, implementing industry best practices for privacy protection, and prioritizing explicit user consent over data coverage, we can rebuild trust while still gathering valuable (though more limited) insights for project sustainability.

...

Discussion: https://github.com/apache/airflow/discussions/XXXXX

Vote Thread: TBD after discussion period

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 3

New Version Current

Key

Abstract

Motivation

Goals

Non-Goals

Proposal

Data Collection Endpoint

Data Collected

Installation Metrics

Usage Metrics

Aggregated Statistics (collected weekly)

Technical Metadata

Data NOT Collected

Consent Mechanism

First-Time Installation

Current Problems:

Why telemetry is crucial for Airflow's development:

Historical Context

Goals

Non-Goals

Proposal

Data Collection Endpoint

Data Collected

Installation Metrics (with noise)

Usage Metrics (aggregated and noised)

Aggregated Statistics (collected weekly with differential privacy)

Operator Usage (aggregated with noise)

Technical Metadata

Data NOT Collected

Consent Mechanism

Default Behavior: Opt-In

First-Time Installation

Web UI Admin Screen

Configuration

Database Storage

Configuration File

Environment Variables

Configuration Precedence:

Five Ways to Control Telemetry

Data Transmission

Transmission Schedule

Transmission Method

Differential Privacy Implementation

Local Aggregation

Sample Payload

Transparency Features

Debug Mode

Version Status

Governance and Changes

Process for Modifying Collected Data

Telemetry Schema Versioning

Transparency and Data Access

Public Dashboard

Raw Data Access

Open Source Implementation

Security Considerations

Our Telemetry Promise

Backward Compatibility

Removal of Scarf

Upgrade Experience

Documentation

Main Documentation Page: /docs/apache-airflow/telemetry.rst

Admin Guide: /docs/apache-airflow/administration-and-deployment/telemetry.rst

Release Notes

Alternatives Considered

Alternative 1: No Telemetry

Alternative 2: Opt-Out by Default

Alternative 3: Exact Counts Without Noise

Alternative 4: Persistent Installation UUID

Alternative 5: Self-Hosted Telemetry Endpoint Only

Alternative 6: Third-Party Service (e.g., PostHog, Segment)

Addressing Common Concerns

"Why not make it opt-out?"

"How do I verify what you're sending?"

Main Documentation Page: `/docs/apache-airflow/telemetry.rst`

Admin Guide: `/docs/apache-airflow/administration-and-deployment/telemetry.rst`

GDPR Compliance Guidelines
ASF Privacy Policy
Telemetry Best Practices for Open Source
Airflow 2.10.0 Release Notes - Scarf
Previous discussion:

[GitHub Issue #XXXXX]