Versions Compared

Old Version 1

changes.mady.by.user Peter Corless

Saved on

compared with

New Version Current

changes.mady.by.user Peter Corless

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Kafka Improvement Proposal: Quality of Service (QoS) Framework

Status

Draft

Motivation

Apache Kafka has become the de facto standard for event streaming, with a growing ecosystem of Kafka-compliant services and implementations. While these services conform to the wire protocol, they differ drastically in their Quality of Service (QoS) characteristics—including latency, throughput, elasticity, storage architecture, and observability.

...

Any QoS implementation protocols and methods should be open standards, free of vendor bias as much as possible, while still allowing for customization and extensibility for advanced features that one vendor or implementation might support that others do not (or do not yet).

Proposed Changes

  1. QoS Declarations: Allow producers and consumers to declare desired QoS in their configurations.
  2. Cluster Capabilities Description: Brokers will expose supported QoS ranges, capabilities (e.g., self-balancing, storage tiering, autoscaling), and current limits.
  3. QoS Negotiation: A negotiation mechanism to reconcile producer/consumer expectations with broker capabilities.
  4. Observability Integration: Define standard metrics to report actual observed QoS (e.g., end-to-end latency, data freshness, throughput).
  5. QoS in Topic Configuration: Enable topic-level QoS annotations that can act as policy templates or governance guides.

Proposed Public Interfaces

This KIP introduces the concept of a QoS grammar that can be expressed in topic-level configurations, cluster-level capabilities, producer/consumer client metadata, and observability endpoints. 

QoS Metrics Measurements and Responses

The system measures and responds to Quality of Service (QoS) requirements in order to optimize cost, latency, and availability on a per-topic basis. Each topic has a requestedQoS policy attached, which may specify targets for characteristics such as:

...

To ensure compliance with these policies, each topic continuously reports both the requested QoS and the achieved QoS. The achieved QoS is calculated by the system based on runtime metrics collected by the control plane and brokers:

QoS Metadata Format (example schema)


{
  "desired-latency": {
    "p99": "50ms"
  },
  "desired-max-freshness": "00:01:00:00",
  "expected-throughput": "2M/s",
  "expected-payload-size": {
    "min": "500B",
    "max": "10MB",
    "mode": "<1KB"
  },

  "elasticity": {
    "min-partitions": 10,
    "max-partitions": 1000

  },
  "priority": "high",
  "retention": {
    "duration": "30d",
    "tier": "hot"

  },
  "schema-evolution-aware": true
}

Achieved QoS Measurements

Metrics collected per topic may include:

...

Topics could self-report these metrics periodically or brokers could aggregate and emit them. Consumers and producers could subscribe to these metrics via an internal API or external metrics collection endpoint. These reports may include timestamps, moving average windows, and confidence levels.

QoS Metrics Format (example schema)


{
  "topic": "sensor-stream",
  "window": "5m",
  "desired-latency-p99": "100ms",
  "actual-latency-p99": "127ms",
  "desired-throughput": "500k msgs/sec",
  "actual-throughput": "491k msgs/sec",
  "max-payload-size-requested": "1MB",
  "max-payload-size-observed": "987KB"
}

QoS Evaluation and Reporting

Each topic periodically emits a QoS Status Report containing:

...

These reports are exposed via the system dashboard, APIs, and alerting tools. This allows users and the control plane to understand whether a topic's SLAs are being met.

System Response to QoS Drift or Violation

When a drift between requested and achieved QoS is detected, the control plane may take corrective action:

  • Storage Tier Promotion/Demotion: Automatically promote hot topics from object storage to SSD or memory cache if latency SLAs are unmet.

  • Replica Adjustment: Increase replication or change acknowledgment quorum if durability or availability is not achieved.

  • Dynamic Resource Allocation: Scale up compute or networking resources for affected topics.

  • User Notification and Guidance: Inform topic owners when they are consistently out of bounds and suggest policy or configuration changes (e.g., increase cost tolerance or adjust throughput targets).

Kafka Cluster Characteristics

While individual Kafka topics will be the primary QoS policy level, clusters themselves can provide descriptions of their overall QoS based upon various factors: topology, capacity, performance, scalability, availability and even affordability. For example, a cluster may have an overall network bandwidth capacity, against which each individual topic that is supported deprecates a remaining availability metric.

...

Producers may or may not be able to specify the storage media their topics are deployed to in the cluster. For example, serverless deployments may entirely obscure such details. The producer should at least be able to request general requirements, such as throughput and latency, leaving the server to interpret how to fulfill them in a serverless manner, opaque to the producer.

Compatibility, Deprecation, and Migration Plan

  • No existing APIs are broken.
  • QoS capabilities will be opt-in and backward compatible.
  • Clients and brokers that don’t support QoS negotiation or observability will function as-is.

Use Cases

  • Vendor Differentiation: Describe the capabilities of a Kafka-compliant service transparently.
  • SLA Negotiation: Form the basis of SLO/SLA/SLC between producers, brokers, and consumers.
  • Throttling and Backpressure: Use QoS metrics to drive intelligent throttling.
  • Autoscaling: Preemptively scale infrastructure based on declared throughput expectations.
  • Storage Tiering: Route data to appropriate storage based on lifecycle and access patterns.
  • Cost Optimization: Align cost and performance by selecting appropriate QoS levels.

Prior Art and References

Future Work

  • Define a formal grammar (e.g., JSON Schema or protobuf) for QoS metadata
  • Add QoS negotiation APIs to Kafka clients
  • Integrate QoS metrics into JMX and logging outputs
  • Establish community-led working groups for specific QoS dimensions (latency, elasticity, retention, etc.)
  • Explore alignment with schema evolution and data contracts

Conclusion

As Kafka evolves to serve more diverse workloads and industries, the need for transparent, standardized, and negotiable QoS has become essential. This KIP aims to establish the foundational work for a vendor-neutral, extensible QoS framework for Apache Kafka.

...