Status: Draft

Author: Viquar Khan (Vaquar.khan@gmail.com)

Discussion thread: TBD

Jira:

------------------------------------------------------

KIP: Automated Contribution Quality and AI-Generated Pull Request Validation

Motivation

The open-source community is seeing a sharp increase in automated pull requests. While code generation tools can help developers, they frequently produce pull requests that look correct on the surface but contain empty boilerplate, unnecessary comments, and hallucinated logic. This puts a heavy burden on maintainers who spend limited review time on code with no real substance.

Other major Apache projects are already dealing with this. Apache Airflow changed their contribution policies after automated bots started taking over open issues. Apache Iceberg wrote strict new contribution guidelines to protect their reviewers.

Why AGENTS.md is not enough

Adding an AGENTS.md or CLAUDE.md file is a "soft control." AI models use probabilistic reasoning; if their context window fills up or a user overrides the prompt, the agent will silently ignore markdown instructions and submit bad code anyway. We need a deterministic "hard control" that catches these failures before they reach a human reviewer.

I propose adding two focused validation tasks to Kafka's Gradle build that catch the two most damaging categories of low-quality contributions: scaffolding-heavy PRs with no real logic, and code that violates Kafka's specific architectural rules.

Public Interfaces

This KIP introduces no changes to the Kafka protocol, public APIs, client behaviors, or broker metrics. It modifies only the project's internal build tools and CI workflow.

Proposed Changes

I propose adding two new Gradle tasks to Kafka's buildSrc directory. These are native Java classes that live inside the Kafka repository — no external dependencies beyond JavaParser (build-time only, same pattern as checkstyle/spotbugs). Developers run them locally with ./gradlew checkContributionQuality, and CI runs them automatically as part of ./gradlew check.

Important: These tasks complement, not duplicate, Kafka's existing checks.

Relationship to Existing Checks

To clarify why these new gates are required alongside existing tooling, the following table details how the proposed AIV Gate differs from Kafka's existing Checkstyle configurations:

Task 1: Logic Density Validation (LDR)

The problem Checkstyle cannot solve: Checkstyle measures the complexity of individual methods. It cannot answer the question: "Did this PR add real logic, or is it 300 lines of scaffolding wrapping 2 lines of actual work?" AI tools are notorious for generating massive PRs with very little executable logic—empty class hierarchies, verbose documentation for trivial getters, and copy-pasted boilerplate. These PRs pass all existing checks because each individual method is simple.

How it works: The task uses JavaParser to build an Abstract Syntax Tree (AST) of each changed Java file in the PR diff and counts two categories of nodes:

• Logic nodes (weighted): if, for, while, switch (weight 5); method calls, binary expressions (weight 2).

• Structure nodes (weighted): class declarations, method declarations (weight 1).

• Formula: Logic Density Ratio (LDR) = logicNodes / (logicNodes + structureNodes)

Default Thresholds & Exceptions:

• LDR Threshold (Java): 0.25 (Configurable). Preliminary validation against recent Kafka PRs suggests legitimate contributions score 0.3+ (because constructors, validation logic, and builder method calls count as logic nodes), while scaffolding-heavy submissions score 0.05-0.15. The 30-day shadow mode will provide definitive validation.

• Entropy Threshold (non-Java): 3.8 bits. For Scala, shell scripts, and configs, the task falls back to Shannon entropy. Copy-pasted boilerplate has low character entropy; real code has higher entropy.

• Refactoring Exception: If a PR has a net-negative line count (e.g., <= -50 lines), the density check is skipped entirely so legitimate cleanups are never blocked.

• Test/Generated Code Bypass: Paths matching **/generated/** or **/test/** are automatically excluded to prevent false positives.

Task 2: Design Compliance (Kafka Architecture Linter)

The problem Checkstyle cannot solve: While Checkstyle's Regexp module is excellent for globally blocking anti-patterns like System.exit, it cannot express conditional rules like "if a file uses KafkaConsumer, then it must NOT use ExecutorService." It also cannot express required patterns like "if a file creates a KafkaProducer, it should call close()."

How it works: Rules are defined in a simple YAML file (.validation/design-rules.yaml) at the repository root. This allows committers to add new rules without writing Java code.

Kafka-Specific Rules to Ship With:

• no-direct-zk-access:

  • Trigger: ZooKeeper, ZkClient, CuratorFramework

  • Forbidden: Instantiation or use in new code.

  • Why: Kafka removed ZK dependency in KRaft mode, but AI tools still routinely generate ZK-based code from old training data.

• consumer-thread-safety:

  • Trigger: KafkaConsumer, consumer.poll

  • Forbidden: ExecutorService, ThreadPoolExecutor, newFixedThreadPool

  • Why: KafkaConsumer is explicitly not thread-safe. AI tools frequently hallucinate and wrap it in thread pools.

• producer-close:

  • Trigger: KafkaProducer, new KafkaProducer

  • Required: producer.close

  • Why: Unclosed producers leak connections and memory. (Note: To avoid false positives, this rule will check for either an explicit producer.close() call OR the use of try (var producer = new KafkaProducer<>(...)) in the same block).

• no-deprecated-metrics:

  • Trigger: MetricName, KafkaMetricsGroup

  • Forbidden: kafka.metrics.KafkaMetricsGroup

  • Why: The old metrics API is deprecated; new code must use the updated API.

Implementation Details

Where the code lives: buildSrc/ is Gradle's standard convention for custom build logic. This directory does not currently exist in the Kafka repo, but introducing it is the idiomatic Gradle approach.

buildSrc/

├── build.gradle

└── src/

├── main/java/org/apache/kafka/gradle/integrity/

│ ├── DensityAnalyzer.java # LDR + entropy calculation

│ ├── DesignComplianceChecker.java # YAML rule enforcement

│ ├── ContributionQualityTask.java # Gradle task entry point

│ └── DiffParser.java # Git diff extraction

└── test/java/org/apache/kafka/gradle/integrity/

├── DensityAnalyzerTest.java

└── DesignComplianceCheckerTest.java

Build-Time Dependencies:

• com.github.javaparser:javaparser-core (AST parsing for density analysis. Build-time only, does not ship in Kafka binaries).

• org.yaml:snakeyaml (Already a transitive dependency in Kafka's build, used for rule parsing).

Human Overrides:

• Emergency bypass: Adding /skip-validation in any commit message skips all gates.

• Trusted authors: Committers can add their emails to a trusted_authors list in .validation/config.yaml to bypass density checks entirely.

Strategic Fixes and AI Hallucination Protections

To ensure the tool does not bottleneck legitimate contributions while effectively handling AI hallucinations, the following features are integrated:

1. Diagnostic Quality Scorecards: Instead of a binary Pass/Fail, the tool outputs a detailed diagnostic report in the CI logs (e.g., "Your PR is 70% boilerplate. We suggest moving your configuration logic to the InternalConfig class to improve logic density.").

2. Semantic Integrity Enforcement: The Design Gate will verify that suggested Kafka configuration keys actually exist in the codebase to catch confident-but-wrong AI hallucinations.

3. Doubt Thresholds: If the LDR score is borderline (e.g., 0.23–0.25), the tool defaults to a "Flag for Human" warning rather than a hard failure to avoid losing legitimate minor fixes.

4. Context Retrieval Guard: Prompts and instructions for AI contributors require them to use "Step-Back Prompting" to explain the architectural 'why' in the PR description before generating code.

Compatibility, Deprecation, and Migration Plan

This proposal does not change the Kafka protocol or public APIs. It adds only internal build logic.

Migration: None. Existing code passes both checks (validated against trunk). The tasks only evaluate changed files in a PR diff, not the entire codebase.

Test Plan

To guarantee zero disruption to developer velocity, this KIP relies on strict unit testing and a phased CI rollout.

1. Unit Tests (JUnit 5 in buildSrc/src/test/):

• DensityAnalyzerTest:

  • Case: 150 lines of Javadoc with a single return statement -> LDR = 0.05 -> FAILS.

  • Case: 20 lines of branching if/while logic -> LDR = 0.45 -> PASSES.

  • Case: PR removes 100 lines of dead code -> Net LOC negative -> SKIPS check.

• DesignComplianceCheckerTest:

  • Case: File initializes KafkaConsumer and passes it to Executors.newFixedThreadPool(5) -> FAILS (consumer-thread-safety violation).

  • Case: File initializes KafkaConsumer in a standard single-threaded poll loop -> PASSES.

2. Shadow Mode (30-Day Rollout):

Upon merge, the GitHub Action will run with continue-on-error: true. It will log results (Pass/Fail, LDR score, AST violations) to the GitHub Actions step summary for 30 days without blocking PRs. Once the PMC validates that the false-positive rate is practically zero, we will remove the continue-on-error flag to make it a mandatory, blocking check.

Review & Comparison: Industry Solutions

This solution moves beyond the "descriptive" nature of standard AI tools and introduces a "deterministic" layer of governance.

FAQ

Q: Won't failing a CI check generate massive email spam to the dev@kafka mailing list?

A: No. The validation task will simply fail the CI check exactly like Checkstyle does today. It will output a clear error message in the console pointing to CONTRIBUTING.md so the contributor knows how to fix their submission. We will not use automated bot comments to avoid webhook noise.

Q: Checkstyle already handles our code quality. Why do we need a new AST Design Gate?

A: Checkstyle is fantastic for formatting and simple regex (like globally blocking System.exit), but it struggles with complex, conditional architectural logic. Checkstyle cannot easily enforce "If KafkaConsumer is used, ensure it is not wrapped in java.util.concurrent." This custom task handles the architectural patterns that Checkstyle structurally cannot.

Q: Will parsing the AST significantly slow down local builds (./gradlew check) or CI?

A: No. JavaParser is highly optimized and the density task evaluates only the specific files modified in the PR diff, rather than the entire codebase. The execution overhead is negligible, typically running in under 1-2 seconds.

Q: Who maintains the .validation/design-rules.yaml file? Won't it become a dumping ground for arbitrary rules?

A: The Kafka PMC owns the configuration file. Any addition or modification to the design rules requires a standard pull request and committer review, following the exact same governance model currently used for checkstyle/checkstyle.xml.

Q: What happens with trivial PRs, like fixing a single typo in a Javadoc? Won't the LDR score be 0?

A: The Gradle task includes a minimum line-change threshold. Trivial PRs (e.g., modifying fewer than 10 lines of code) bypass the LDR check entirely to ensure minor documentation fixes or typo corrections are never blocked.

Q: How does this handle our Python ducktape system tests or shell scripts?

A: For non-Java languages (such as Python, Ruby, and Bash), the framework automatically bypasses AST parsing and falls back to the Shannon Entropy Gate. This measures character diversity to block repetitive AI scaffolding without requiring a dedicated parser for every language.

Q: What if the script has a bug and blocks legitimate human contributors?

A: The system prioritizes developer velocity. Any developer can bypass the gate immediately by adding /skip-validation to their commit message or PR description. Additionally, the initial 30-day "Shadow Mode" runs non-blocking, allowing the PMC to review its accuracy before it ever gains the power to block a merge.

Q: Does LDR penalize good documentation?

A: No. The Logic Density Ratio is designed to ignore comments and javadocs, focusing strictly on executable AST (Abstract Syntax Tree) nodes.

Q: How does the gate handle refactoring?

A: AIV includes a "Refactor Exception"—if a PR removes more lines than it adds (net-negative lines), the density check is automatically bypassed.

Q: Does this check Scala code?

A: Yes. For Scala and other non-Java files, the system uses Shannon entropy to detect repetitive, low-substance boilerplate.

Q: How do I know if my PR will fail before I push it?

A: You can run ./gradlew checkContributionQuality locally to get your LDR score and AST violation report before committing.

Q: Will this block my emergency hotfix?

A: No. Committers can bypass all gates by adding /aiv skip or /skip-validation to the commit message.

Q: Is my code being sent to an external LLM for review?

A: No. The analysis is entirely local and AST-based, ensuring 100% data sovereignty and zero API costs.

Rejected Alternatives

• Relying solely on human reviewers: Continuing to manually review and close low-substance PRs does not scale and exacerbates maintainer burnout.

• Restricting PR access to Collaborators Only: While GitHub recently introduced this feature to stop bot spam, using it heavily restricts legitimate, first-time open-source contributors from participating in Kafka.

• Using Third-Party GitHub Actions: Using pre-compiled external binaries for PR validation introduces supply-chain security risks and prevents developers from running the exact same checks locally on their laptops. Building it natively into our Gradle scripts solves both issues.

• Adding only AGENTS.md: A soft control that AI models probabilistically ignore. This KIP provides deterministic hard control.