Status

Current state: Draft

Discussion thread: here

JIRA: TDB

Released: target 4.1.0 – effective 5.0.0

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Historically, Apache Kafka did provide very generous backward compatibility guarantees. For example, up to Apache Kafka 4.0.0, all older clients could connect to brokers, no matter how old they are, as all RPC versions were supported up to, including, 3.9.x. While Apache Kafka 4.0.0 did drop support for some very old versions, the provided backward compatibility guarantees are still very generous, and the main issue is really, that there is still no official contract. The changes made in Apache Kafka 4.0.0 were backed by individual KIPs only, what is basically some "random patchwork".

For a mature project like Kafka, there should be a clearly defined, and well document contract with users, about what compatibilities are guaranteed. Of course, we can still apply additional exception (ie, maintain something longer, or shorter) based on individual KIPs if there is a need. Nevertheless, having a public "base contract" that we would only diverge from in very specific and rare circumstances would be a great improvement. – In addition, the currently provided backward compatibility guarantees are still very generous, making it hard to maintain the code base, so we should also consider to reduce the guaranteed backward compatibility window, allowing us to stay agile and move faster. While a good backward compatibility story is a great trait, being too generous has the drawback that people don't upgrade their dependencies, running for example ancient client versions against newer brokers, resulting in all sort of issues (it's also a security concern if outdated software is used).

Given that we just release Apache Kafka 4.0.0, it is the perfect time to define this contract now. It would only apply to Apache Kafka 5.0.0, and thus user will have the maximum amount of time to prepare.

Public Interfaces

The is no direct public interface changes. However, if we agree on a contract, we are allowed to remove support for older versions, ie, upgrade from version X to version Y, or cross component compatibility (ie, use client version X with broker version Y), without the need to do a KIP, if a major release happens. – The agree on contract must be clearly documented in the Kafka webpage.

Proposed Changes

There is different backward compatibility guarantees we need to consider:

• Component upgrade compatibitliy: Can I upgrade brokers (or clients / connect / streams) directly from version X to version Y?
• Client-Server compatibility: Can I use clients version X with brokers version Y?
• Kafka Connect framework and connectors compatibility: Can I run a version X connector using Connect cluster version Y?
• Maybe: Binary compatibility vs source compatibility (for clients, streams, connectors): Can I just switch the jar and re-deploy, or do I need to recompile my code using the new jar first?

We propose to maintain compatibility across two major version going forward.

Component upgrades:

• Broker and Kafka Connect can be directly upgraded from 4.x to 5.x version. However, there is no guarantee that 3.x brokers / connect workers can be directly upgraded to 5.x.
• Clients (producer/consumers/admin/Kafka Streams) API compatibility:
  • We are allowed break API compatibility with a major release, i.e., 5.x will remove deprecated API
  • We still guarantee a one year API compatibility window, i.e., APIs deprecated in the last three minor versions of 4.x series will not be removed with the 5.0.0 release
  • These last three minor version of 4.x act as bridge release version, and upgrading to any 5.x version is supported
  • Upgrading clients directly from 3.x to 5.x is not supported any longer.

Client-Server compatibility:

• Clients with version 4.x can connect to 5.x brokers, but older clients 3.x and older are not supported any longer.
• Clients with version 5.x can connect to 4.x brokers, but broker 3.x and older are not supported any longer.
• The clients backward compatibility directly translates into the same guarantees for Kafka Connect and Kafka Streams.

Kafka Connect:

• Connectors compiled with version 4.x can be deployed into a 5.x Connect cluster. However, Connector complied with version 3.x are not supported with Connect 5.x any longer.

Compatibility, Deprecation, and Migration Plan

TDB

Test Plan

n/a

Documentation Plan

The Kafka webpage must be updated, with a dedicate "Backward Compatibility Guarantees" page, explaining the contract, plus showing with concrete versions number what will be supported.

Rejected Alternatives

n/a