Page History

...

Author: Viquar Khan (Vaquar.khan@gmail.com)

Discussion thread: TBD

Jira:

Motivation The open-source community is currently seeing a massive sharp increase in automated pull requests. While code generation tools can be helpful to help developers, they frequently submit produce pull requests that look correct on the surface but are actually full of contain empty boilerplate, unnecessary comments, and hallucinated logic. This puts a heavy burden on our maintainers , who have to spend their limited review time reviewing on code that has with no real substance.

Other major Apache projects are already dealing with this issue and taking action. For example, Apache Airflow recently had to change changed their contribution policies because after automated bots were started taking over open issues, and . Apache Iceberg recently had to write wrote strict new contribution guidelines to protect their reviewers.

Why AGENTS.md is not enough: While adding Adding an AGENTS.md or CLAUDE.md file to the repository is a great first step to instruct well-behaving bots, it acts only as a "soft control." . AI models use probabilistic reasoning; if their context window gets too full, or if fills up or a user overrides the prompt, the agent will silently ignore the markdown instructions and submit the bad code anyway. We cannot rely on the "good intentions" of an AI model to protect our codebase. We need a deterministic "hard control" to catch that catches these failures before they consume reach a human reviewer's time.

Rather than waiting for our reviewers to burn out, we should be proactive. I propose we add custom, native validation logic to our Gradle build system to mathematically check the quality and substance of a pull request before a human reviewer is ever notifiedI propose adding two focused validation tasks to Kafka's Gradle build that catch the two most damaging categories of low-quality contributions: scaffolding-heavy PRs with no real logic, and code that violates Kafka's specific architectural rules.

Public Interfaces This KIP introduces no changes to the Kafka protocol, public APIs, client behaviors, or broker metrics. It strictly modifies only the project's internal build tools and continuous integration ( CI ) workflow.

Proposed Changes I propose we add adding two new custom Java-based validation tasks directly into Gradle tasks to Kafka's buildSrc / build-logic directory. By building this directly into our Gradle configuration, the checks run entirely locally without needing external API keys, introducing zero supply-chain risk, and allowing developers to run them on their laptops just like our existing ./gradlew checkstyleMain tasks.

These tasks will provide a two-layered defense against low-quality automated contributions:

directory. These are native Java classes that live inside the Kafka repository — no external dependencies beyond JavaParser (build-time only, same pattern as checkstyle/spotbugs). Developers run them locally with ./gradlew checkContributionQuality, and CI runs them automatically as part of ./gradlew check.

Important: These tasks complement, not duplicate, Kafka's existing checks. Kafka already enforces FinalLocalVariable, CyclomaticComplexity (max 16), MethodLength (max 170), and NPathComplexity (max 500) via checkstyle/checkstyle.xml. The proposed tasks target problems that Checkstyle structurally cannot detect.

Task 1: Logic Density Validation (LDR) The problem Checkstyle cannot solve: Checkstyle measures the complexity of individual methods. It cannot answer the question: "Did this PR add real logic, or is it 300 lines of scaffolding wrapping 2 lines of actual work?" 1. Logic Density Validation (LDR) Task AI tools are notorious for generating massive pull requests PRs with very little actual code, such as generating 12 lines of sophisticated documentation for a single line of logic. I propose adding a Gradle task that calculates a "Logic Density Ratio" (LDR), which mathematically measures the amount of logical or executable statements relative to the overall size of a code block. It evaluates the ratio of actual working code against the amount of boilerplate and comments. If a pull request submits 300 lines of setup and documentation but only 2 lines of actual logic, the task will flag it as low-effort scaffolding and fail the build.

2. AST Architecture Linter Task Automated tools often fail to understand Kafka's specific design paradigms. I propose adding a deterministic Abstract Syntax Tree (AST) parser task to systematically enforce our architecture based on our existing contributor guidelines.

Examples of problems solved:

executable logic—empty class hierarchies, verbose documentation for trivial getters, and copy-pasted boilerplate. These PRs pass all existing checks because each individual method is simple.

How it works: The task uses JavaParser to build an Abstract Syntax Tree (AST) of each changed Java file in the PR diff and counts two categories of nodes:

1. Logic nodes (weighted): if, for, while, switch (weight 5); method calls, binary expressions (weight 2).

2. Structure nodes (weighted): class declarations, method declarations (weight 1). Formula: Logic Density Ratio (LDR) = logicNodes / (logicNodes + structureNodes)

Default Thresholds & Exceptions:

• LDR Threshold (Java): 0.25 (Configurable). We validated this against 50 recent Kafka PRs. Legitimate PRs score 0.3+ because constructors, validation logic, and builder method calls count as logic nodes. Known AI-generated scaffolding PRs score 0.05-0.15.

• Entropy Threshold (non-Java): 3.8 bits. For Scala, shell scripts, and configs, the task falls back to Shannon entropy. Copy-pasted boilerplate has low character entropy; real code has higher entropy.

• Refactoring Exception: If a PR has a net-negative line count (e.g., <= -50 lines), the density check is skipped entirely so legitimate cleanups are never blocked.

• Test/Generated Code Bypass: Paths matching **/generated/** or **/test/** are automatically excluded to prevent false positives.

Task 2: Design Compliance (Kafka Architecture Linter) The problem Checkstyle cannot solve: Checkstyle's Regexp module can block individual patterns globally (e.g., System.exit). But it cannot express conditional rules like "if a file uses KafkaConsumer, then it must NOT use ExecutorService." It also cannot express required patterns like "if a file creates a KafkaProducer, it should call close()."

How it works: Rules are defined in a simple YAML file (.validation/design-rules.yaml) at the repository root. This allows committers to add new rules without writing Java code.

Kafka-Specific Rules to Ship With:

1. consumer-thread-safety:

   • Trigger: KafkaConsumer, consumer.poll

   • Forbidden: ExecutorService, ThreadPoolExecutor, newFixedThreadPool

   • Why

...

1. • : KafkaConsumer is explicitly not thread-safe.

...

Configuration Rules: AI agents frequently assume Kafka is a standard database and introduce dangerous defaults. The parser will block pull requests that blindly set enable.auto.commit=true (which leads to data loss during failures) or leave critical metrics like retention.ms at unsafe defaults.

...

1. • AI tools frequently hallucinate and wrap it in thread pools.

2. producer-close:

   • Trigger: KafkaProducer, new KafkaProducer

   • Required: producer.close

   • Why: Unclosed producers leak connections and memory.

3. no-deprecated-metrics:

   • Trigger: MetricName, KafkaMetricsGroup

   • Forbidden: kafka.metrics.KafkaMetricsGroup

   • Why: The old metrics API is deprecated; new code must use the updated API.

Implementation Details

Where the code lives: buildSrc/ is Gradle's standard convention for custom build logic. This directory does not currently exist in the Kafka repo, but introducing it is the idiomatic Gradle approach to prevent polluting the root 

buildSrc/
├── build.gradle
└── src/
    ├── main/java/org/apache/kafka/gradle/integrity/
    │   ├── DensityAnalyzer.java          # LDR + entropy calculation
    │   ├── DesignComplianceChecker.java  # YAML rule enforcement
    │   ├── ContributionQualityTask.java  # Gradle task entry point
    │   └── DiffParser.java              # Git diff extraction
    └── test/java/org/apache/kafka/gradle/integrity/
        ├── DensityAnalyzerTest.java
        └── DesignComplianceCheckerTest.java

Build-Time Dependencies:

• com.github.javaparser:javaparser-core (AST parsing for density analysis. Build-time only, does not ship in Kafka binaries).

• org.yaml:snakeyaml (Already a transitive dependency in Kafka's build, used for rule parsing).

Human Overrides:

• Emergency bypass: Adding /skip-validation in any commit message skips all gates.

• Trusted authors: Committers can add their emails to a trusted_authors list in .validation/config.yaml to bypass density checks entirely.

Compatibility, Deprecation, and Migration Plan This proposal does not change the Kafka protocol or public APIs. It adds only

...

internal

...

build logic

...

To ensure this does not slow down regular developers, the proposed tasks will include built-in exceptions:

• Refactor Exception: If a developer is cleaning up code and deleting more lines than they add (net negative lines), the Logic Density check is automatically skipped so legitimate refactoring is never blocked.

• Emergency Override: Anyone can add a /skip-validation flag to their commit message to bypass the checks during an urgent hotfix.

Test Plan To ensure this solution is robust and does not disrupt developer velocity, we will implement the following testing strategy:

...

Unit Testing the Gradle Tasks: The new logic in buildSrc will include its own JUnit tests. We will add test cases containing known "AI slop" patterns and concurrency violations to mathematically prove the AST parser and LDR calculator accurately reject bad code while allowing valid code.

. Migration: None. Existing code passes both checks (validated against trunk). The tasks only evaluate changed files in a PR diff, not the entire codebase.

Test Plan

To guarantee zero disruption to developer velocity, this KIP relies on strict unit testing and a phased CI rollout.

1. Unit Tests (JUnit 5 in buildSrc/src/test/):

• DensityAnalyzerTest:

  • Case: 150 lines of Javadoc with a single return statement -> LDR = 0.05 -> FAILS.

  • Case: 20 lines of branching if/while logic -> LDR = 0.45 -> PASSES.

  • Case: PR removes 100 lines of dead code -> Net LOC negative -> SKIPS check.

• DesignComplianceCheckerTest:

  • Case: File initializes KafkaConsumer and passes it to Executors.newFixedThreadPool(5) -> FAILS (consumer-thread-safety violation).

  • Case: File initializes KafkaConsumer in a standard single-threaded poll loop -> PASSES.

2. Shadow Mode (30-Day Rollout): Upon merge, the GitHub Action will run with continue-on-error: true. It will log results (Pass/Fail, LDR score, AST violations) to the GitHub Actions step summary for 30 days without blocking PRs. Once the PMC validates that the false-positive rate is practically zero, we will remove the continue-on-error flag

...

to make it a mandatory, blocking check

...

.

...

Rejected Alternatives

• Relying only on human reviewers:

...

• Does not scale. Maintainer time is the scarcest resource.

• Using third-party GitHub Actions:

...

• External binaries introduce supply-chain

...

• risk and cannot be run locally. Using buildSrc ensures developers run the exact same checks

...

• on their laptops

...

• (./gradlew checkContributionQuality).

• Extending Checkstyle only: Checkstyle operates per-file on the full source. It cannot measure per-PR logic density, nor can it easily enforce conditional architectural rules ("if file contains X, then block Y").

• Adding only AGENTS.md: A soft control that AI models probabilistically ignore. This KIP provides deterministic hard control.