Page History

Status: Draft

Author: Viquar Khan (Vaquar.khan@gmail.com)

Discussion thread:

...

TBD

Jira:

Motivation

The open-source community is

...

seeing a

...

sharp increase in automated pull requests. While code generation tools can

...

help developers, they frequently

...

produce pull requests that look correct on the surface but

...

contain empty boilerplate, unnecessary comments, and

...

hallucinated logic. This puts a heavy burden on

...

maintainers

...

who

...

spend

...

limited review time

...

on code

...

with no real substance.

While this trend is accelerated by AI, the necessity for structural validation applies to all pull requests. Other major Apache projects are already dealing with this

...

.

...

Apache Airflow

...

changed their contribution policies

...

after automated bots

...

started taking over open issues

...

. Apache Iceberg

...

wrote strict new contribution guidelines to protect their reviewers.

...

Rather than waiting for our reviewers to burn out, we should be proactive. I propose we add a set of native, automated quality scripts to our continuous integration (CI) pipeline. This will mathematically check the quality and substance of a pull request before a human reviewer is ever notified.

...

Why AGENTS.md is not enough

Adding an AGENTS.md or CLAUDE.md file is a "soft control." AI models use probabilistic reasoning; if their context window fills up or a user overrides the prompt, the agent will silently ignore markdown instructions and submit bad code anyway. We need a deterministic "hard control" that catches these failures before they reach a human reviewer.

I propose adding two focused validation tasks to Kafka's Gradle build known collectively as the Automated Integrity Validation (AIV) Gate that catch the two most damaging categories of low-quality contributions: scaffolding-heavy PRs with no real logic, and code that violates Kafka's specific architectural rules. This effectively addresses the "Reviewer Overload" crisis while maintaining Kafka's domain-specific invariants and ensuring 100% local data sovereignty.

Relationship to Existing Checks

To clarify why these new gates are required alongside existing tooling, the following table details how the proposed AIV Gate differs from Kafka's existing Checkstyle configurations:

Public Interfaces

This KIP introduces no changes to the Kafka protocol, public APIs, client behaviors, or broker metrics. It

...

modifies only the project's

...

internal build tools and CI workflow.

Proposed Changes

...

1. Logic Density Validation (LDR) AI tools are notorious for generating massive pull requests with very little actual code. I have written a script that calculates a "Logic Density Ratio" (LDR), which measures the amount of logical or executable statements relative to the size of a code block. It evaluates the ratio of actual working code against the amount of boilerplate and comments. If a pull request submits 300 lines of setup and documentation but only 2 lines of actual logic, the CI pipeline will flag it as low-effort scaffolding and block it before it wastes a maintainer's time.

2. AST Architecture Linter Automated tools often fail to understand Kafka's specific concurrency rules. I have built a deterministic Abstract Syntax Tree (AST) parser that uses a simple design-rules.yaml file to enforce our architecture. For example:

• Concurrency Rules: KafkaConsumer is explicitly not thread-safe, and automated tools often make the mistake of wrapping it in standard java.util.concurrent threads. This script will automatically block that exact mistake.

• Configuration Rules: It will block pull requests that try to blindly set enable.auto.commit=true (which leads to data loss) or leave critical metrics like retention.ms at unsafe defaults.

• Code Standards: It enforces the usage of the final keyword wherever possible and ensures new public APIs have JavaDocs, as mandated by our contributor guide.

3. Contributor Instructions (AGENTS.md) We will add an AGENTS.md file to the root of our project. This acts as a clear instruction manual for automated tools, telling them our formatting rules and our requirement to disclose automated code with a "Generated-by" token in the commit message before they even write the code.

Proposed Architecture Workflow

Code snippet

graph TD
    A --> B{Is Core Maintainer?}
    B -- Yes --> C
    B -- No --> D{Commit contains '/skip-validation' flag?}
    D -- Yes --> C
    D -- No --> E{Dependency Gate}
    E -- Fails --> F
    E -- Passes --> G{Logic Density Gate}
    G -- Fails --> H
    G -- Passes --> I{AST Design Gate}
    I -- Fails --> J
    I -- Passes --> C

Compatibility, Deprecation, and Migration Plan This proposal does not change the Kafka protocol, public APIs, client behaviors, or broker metrics. It only updates our GitHub Actions workflow.

To ensure this does not slow down regular developers, the scripts include built-in exceptions:

• Refactor Exception: If a developer is cleaning up code and deleting more lines than they add (net negative lines), the Logic Density check is automatically skipped so legitimate refactoring is never blocked.

• Emergency Override: Anyone can add a /skip-validation flag to their commit message to bypass the checks during an urgent hotfix.

Test Plan We will run these scripts in a non-blocking "Shadow Mode" for 30 days. It will log its findings on incoming pull requests without actually blocking them or notifying contributors. The community can review these logs to ensure there are no false positives before we decide to make it a mandatory check.

Documentation We will update the CONTRIBUTING.md file to explain the new continuous integration checks and document how the AGENTS.md file works.

Rejected Alternatives

• Doing nothing: Relying entirely on human reviewers to catch low-effort automated submissions and concurrency mistakes does not scale and will lead to maintainer fatigue.

• Restricting pull requests to collaborators only: While this stops bot spam natively in GitHub, it heavily restricts legitimate first-time contributors and contradicts the open-source ethos.

Subject: SPIP: Automated Pull Request Validation and Architectural Compliance

Q1. What are you trying to do? The Apache Spark project is seeing an increase in automated pull requests. These submissions often pass standard syntax checks but contain padded boilerplate, made-up dependencies, and excessive comments without real logic. I am proposing we add a set of custom, native continuous integration (CI) scripts to mathematically filter out this low-effort code and catch common Spark coding mistakes before a human maintainer reviews the code.

Q2. What problem is this proposal NOT designed to solve? This is not designed to replace human review for complex architecture decisions. It is also not designed to ban automated tools for legitimate human contributors. It only filters out zero-substance submissions and catches known API mistakes natively in GitHub Actions.

Q3. How is it done today, and what are the limits of current practice? Right now, Spark maintainers rely on manual review and standard compiler checks. A pull request with 400 lines of useless documentation and a highly inefficient withColumn loop will pass basic CI tests. A human committer then has to waste time explaining why the code is structurally flawed.

Q4. What is new in your approach and why do you think it will be successful? I have written a set of lightweight validation scripts that I want to donate to the project's CI pipeline. The two main features are:

Logic Density Ratio (LDR): A mathematical formula that checks if a pull request has enough actual execution logic compared to its scaffolding.

...

AST Architecture Linter: A parser that uses a design-rules.yaml file to scan the code structure and enforce Spark's evolving API standards.

Q5. Who cares? Project maintainers and committers who are suffering from review fatigue due to an influx of low-quality, automated pull requests.

Q6. What are the risks? The main risk is a "false positive" where a legitimate pull request is blocked. We prevent this by including a "Refactor Exception" that automatically skips the density check if a developer is deleting code (net negative lines). We also allow trusted maintainers to bypass the checks with a simple commit flag.

Q7. How long will it take? Adding the scripts to our GitHub Actions and writing the initial design-rules.yaml will take about two weeks of testing and feedback on the developer mailing list.

Q8. What are the mid-term and final “exams” to check for success?

• Mid-term: Run the new checks in a non-blocking "shadow mode" for one month to monitor accuracy.

• Final: Achieve a measurable drop in the time maintainers spend reviewing low-quality pull requests.

Appendix A. Proposed Design Rules Examples The AST linter will actively scan the code for common mistakes that automated tools frequently make:

...

Production Mistakes: Flagging pull requests that try to use display(), collect(), or print(df.count()) in core execution paths.

...

Loop Mistakes: Automatically failing pull requests that place withColumn() inside a for loop, prompting the contributor to use select() with multiple columns instead.

...

Deprecation Checks: Rejecting code that introduces outdated configurations.

I propose adding an AIV Gate using Gradle's includeBuild composite build feature. This relies on native Java classes that live inside the Kafka repository   no external dependencies beyond JavaParser (build-time only, same pattern as Checkstyle/SpotBugs). Developers run them locally with ./gradlew checkContributionQuality, and CI runs them automatically as part of ./gradlew check.

Important: These tasks complement, not duplicate, Kafka's existing checks.

Task 1: Logic Density Validation (LDR)

The problem Checkstyle cannot solve: Checkstyle measures the complexity of individual methods. It cannot answer the question: "Did this PR add real logic, or is it 300 lines of scaffolding wrapping 2 lines of actual work?" High-volume, low-effort tools often generate massive PRs with very little executable logic empty class hierarchies, verbose documentation for trivial getters, and copy-pasted boilerplate.

How it works: The task uses JavaParser to build an Abstract Syntax Tree (AST) of each changed Java file in the PR diff and counts two categories of nodes:

• Logic nodes (weighted): if, for, while, switch (weight 5); method calls, binary expressions, variable assignments (weight 2).

• Structure nodes (weighted): class declarations, method declarations (weight 1).

• Calculation: The tool simply divides the total score of logic nodes by the combined score of logic + structural nodes. To ensure accuracy and prevent gaming the system, the AST parser entirely ignores whitespace, comments, and Javadoc. (Note: If a PR diff contains no countable structural nodes in Java files, the LDR check is skipped to prevent division by zero).

Weighting Rationale:

Control-flow nodes (if, for, while) dictate cyclomatic complexity and represent actual human or algorithmic problem-solving. Class and method declarations are merely structural scaffolding. Weighting logic nodes at 5x ensures we measure the density of the solution, not the structure.

Worked Examples (LDR in Practice):

To understand the provisional threshold of 0.25, consider these three PR profiles using the exact weights defined above:

1. Legitimate Core Fix (LDR = 0.91): A PR modifying replication logic inside ReplicaManager.java. It adds no new classes and one new private method (weight 1). It contains one if block (weight 5) and three variable assignments (weight 6). Total logic = 11, total structure = 1. LDR = 11 / 12 = 0.91. It passes easily.

2. Interface/Config Addition (Bypassed): A PR adding a new ConfigDef or Java interface. Files where ≥80% of top-level type declarations are interface or @interface types are dynamically evaluated under a "declarative profile" exception, allowing them to pass regardless of the LDR score. The 80% threshold will be validated during Shadow Mode and adjusted if necessary.

3. AI Slop/Boilerplate (LDR = 0.18): A PR generating an expansive new module wrapper. It contains 3 new classes (weight 3), 15 empty methods or simple getters (weight 15), massive Javadoc blocks (ignored), and only 2 actual method calls (weight 4). Total logic = 4, total structure = 18. LDR = 4 / 22 = 0.18. It falls below the 0.25 threshold, and the PR is blocked.

Default Thresholds & Exceptions:

• LDR Threshold Calibration (Java): The provisional threshold is 0.25. The definitive threshold will be finalized by publishing a sensitivity analysis of the 5th percentile distribution across 100-200 historically merged Kafka PRs during the "Shadow Mode" phase.

• The "Annotation Attack" Protection: AI models sometimes pad code with complex type annotations to "fake" logic density. The logic parser is tuned to ignore annotations and focus strictly on executable control-flow nodes.

• Refactoring Exception: If a PR has a net-negative line count (e.g., net -50 lines), the density check is skipped entirely so legitimate cleanups are never blocked. Note: The Refactoring Exception applies only to the Logic Density check (Task 1). Design Compliance rules (Task 2) are always enforced regardless of net line count.

• Language Scope: This initial KIP focuses strictly on Java files via JavaParser. Support for Scala and Kotlin is deferred to a future KIP to keep this proposal focused and rigorous.

Task 2: Design Compliance (Kafka Architecture Linter)

The problem Checkstyle cannot solve: While Checkstyle's Regexp module is excellent for globally blocking anti-patterns like System.exit(), it cannot express conditional rules like "if a file uses KafkaConsumer, then it must NOT use ExecutorService."

Governance Risk & Mitigation:

To prevent "rule creep," rules are defined in .validation/design-rules.yaml. Adding or removing a rule requires a standard pull request and lazy consensus on the dev@kafka.apache.org mailing list with a minimum 72-hour review period. A single PMC member can veto a rule addition to prevent bloat. The YAML schema explicitly requires an added-in-version and rationale field.

Kafka-Specific Rules to Ship With:

• no-direct-zk-access:

  • Trigger: ZooKeeper, ZkClient, CuratorFramework

  • Forbidden: Instantiation or use. (Applies to all lines added in the PR diff).

  • Rationale: Kafka removed ZK dependency in KRaft mode, but AI tools still routinely generate ZK-based code from old training data.

• consumer-thread-safety:

  • Trigger: KafkaConsumer, consumer.poll

  • Forbidden: ExecutorService, ThreadPoolExecutor, newFixedThreadPool

  • Rationale: KafkaConsumer is explicitly not thread-safe. AI tools frequently hallucinate and wrap it in thread pools.

• producer-close:

  • Trigger: KafkaProducer, new KafkaProducer

  • Required: producer.close

  • Rationale: Unclosed producers leak connections and memory. (Note: The checker performs intra-method dataflow analysis using JavaParser's symbol resolution to handle arbitrary variable names like var p = new KafkaProducer, and successfully detects try-with-resources closures).

Implementation Details

Where the code lives: To prevent compilation errors from breaking the entire Gradle build (a common risk with buildSrc/), this tool will be implemented as an isolated composite build using includeBuild("build-plugins").

build-plugins/

├── build.gradle

└── src/

├── main/java/org/apache/kafka/gradle/integrity/

│ ├── DensityAnalyzer.java # LDR calculation

│ ├── DesignComplianceChecker.java # YAML rule enforcement

│ ├── ContributionQualityTask.java # Gradle task entry point

│ └── DiffParser.java # Reads GitHub PR.patch natively

└── test/java/org/apache/kafka/gradle/integrity/

├── DensityAnalyzerTest.java

└── DesignComplianceCheckerTest.java

Secure Human-in-the-Loop (HITL) Overrides:

• Secure Emergency Bypass: Adding /aiv skip in any commit message skips all gates, but only if the GitHub Actions actor is verified against the official Apache Gitbox LDAP COMMITTERS list. Bots and external contributors cannot bypass the gate.

• Auditability: All bypasses are explicitly logged in the GitHub Actions CI step summary for transparency.

Diagnostic Reports (A Mentorship Framework)

Instead of a cryptic failure once Shadow Mode ends, the tool posts a diagnostic summary in the CI output: "This PR's Logic Density is 0.18 (Threshold: 0.25). It appears to be mostly scaffolding. Try moving logic to the internal Config classes to improve density." This provides a clear, mentorship-oriented path to "ready" without bottlenecking legitimate contributions.

Compatibility, Deprecation, and Migration Plan

This proposal does not change the Kafka protocol or public APIs. It adds only internal build logic.

Migration: None. Existing code passes both checks (validated against trunk). The tasks only evaluate changed files in a PR diff, not the entire codebase.

Test Plan

To guarantee zero disruption to developer velocity, this KIP relies on strict unit testing and a phased CI rollout.

1. Unit Tests (JUnit 5 in build-plugins/src/test/):

• DensityAnalyzerTest:

  • Case: 150 lines of Javadoc with a single return statement -> LDR = 0.05 -> FAILS.

  • Case: 20 lines of branching if/while logic -> LDR = 0.45 -> PASSES.

  • Case: PR adds a pure interface with 10 method signatures -> Declarative Profile -> PASSES.

  • Case: PR removes 100 lines of dead code -> Net LOC negative -> SKIPS check.

• DesignComplianceCheckerTest:

  • Case: File initializes KafkaConsumer and passes it to Executors.newFixedThreadPool(5) -> FAILS (consumer-thread-safety violation).

  • Case: File initializes KafkaConsumer in a standard single-threaded poll loop -> PASSES.

  • Case: File contains only interface declarations with no trigger patterns -> PASSES (no rules applicable).

2. Shadow Mode (30-Day Data Collection):

Upon merge, the GitHub Action will run with continue-on-error: true. It will log results (Pass/Fail, LDR score, AST violations) to the GitHub Actions step summary for 30 days without blocking PRs. During this time, the PMC will publish a sensitivity analysis of the historical data.

Exit Criteria: The gate will be promoted to blocking status via lazy consensus on the dev list once the false-positive rate is below 2% over a minimum of 50 evaluated PRs.

Future Work

To keep this KIP tightly scoped to achievable build validation tasks, the following enhancements are deferred to future proposals:

• Kotlin and Scala Support: Extending the density and design parsing to cover Kafka's non-Java modules.

• BOM-Grounding (Supply Chain): A dependency validation phase that cross-references AI-suggested method calls against the current trunk AST to stop "hallucinated" API calls that look like logic but do not exist in the codebase.

• Context Retrieval Guard: A prompt-engineering standard requiring AI contributors to utilize "Step-Back Prompting" in the PR description to explain the architectural intent before generating code.

Review & Comparison: Industry Solutions

This solution moves beyond the "descriptive" nature of standard AI tools and introduces a "deterministic" layer of governance.

FAQ

General Concept

• Q: Why build a custom tool instead of using standard AI code reviewers like GitHub Copilot or CodeRabbit?

  A: Standard AI tools are purely descriptive and lack project-specific context (e.g., they don't natively know that wrapping a KafkaConsumer in an ExecutorService is a threading violation in our core). Furthermore, cloud-based tools require sending Kafka’s internal logic to a third-party cloud for indexing. AIV executes 100% locally via AST parsing, ensuring no data leakage, zero ongoing API costs, and providing a mathematical proxy (LDR) for substance that descriptive AI summaries lack.

• Q: Why rely on hard controls instead of just adding instructions to an AGENTS.md file?

  A: Markdown instructions like AGENTS.md are a "soft control." Research shows that AI models use probabilistic reasoning and frequently ignore markdown instructions when their context window is saturated or overridden by a user prompt. By embedding deterministic Gradle tasks into our pipeline, we ensure that Kafka's architectural invariants are enforced by hard code, not just suggestions.

• Q: Will a PR failure silently reject or move a submission to "Draft," discouraging new human contributors?

  A: No. We strictly want to avoid "Silent Rejection," which can be hostile to newcomers. The system acts as a Mentorship Framework. If a PR falls below the threshold, the GitHub Action posts a Diagnostic Scorecard in the CI logs providing a clear, actionable path to getting the PR ready.

Logic Density Ratio (LDR)

• Q: How does the gate handle refactoring?

  A: AIV includes a "Refactor Exception" if a PR removes more lines than it adds (net-negative lines), the density check is automatically bypassed.

• Q: What happens with trivial PRs, like fixing a single typo in a Javadoc? Won't the LDR score be 0?

  A: The Gradle task includes a minimum line-change threshold. Trivial PRs (e.g., modifying fewer than 10 lines of code) bypass the LDR check entirely to ensure minor documentation fixes or typo corrections are never blocked.

• Q: Does LDR penalize good documentation?

  A: No. The AST analyzer ignores Javadoc and comments entirely, focusing only on executable nodes.

• Q: What about Kotlin and Scala code?

  A: This initial KIP applies exclusively to Java files. Kotlin, Scala, Shell scripts, and configuration files are completely bypassed until dedicated parsers are proposed in a future KIP.

• Q: What if a PR modifies both Java and non-Java files?

  A: The AIV Gate evaluates only the Java files in the diff. Non-Java files are ignored entirely in this initial KIP.

Design Gate (Architecture)

• Q: Checkstyle already handles our code quality. Why do we need a new AST Design Gate?

  A: Checkstyle is fantastic for formatting and simple regex (like globally blocking System.exit()), but it struggles with complex, conditional architectural logic. Checkstyle cannot easily enforce "If KafkaConsumer is used, ensure it is not wrapped in java.util.concurrent." This custom task handles the architectural patterns that Checkstyle structurally cannot.

• Q: Who maintains the .validation/design-rules.yaml file? Won't it become a dumping ground for arbitrary rules?

  A: The Kafka PMC owns the configuration file. Adding or removing a design rule requires a standard pull request and lazy consensus on the mailing list (72 hours), and the schema dictates providing a rationale to prevent rule creep. A single PMC veto blocks the rule.

CI & Build Integration

• Q: Will parsing the AST significantly slow down local builds (./gradlew check) or CI?

  A: No. By isolating the tool via includeBuild and utilizing the native GitHub Actions .patch payload (via DiffParser), there is no shelling out to the git binary. The JavaParser cold-start and AST traversal takes ~2-5 seconds total per PR, not per file.

• Q: How do I test my PR locally?

  A: Run ./gradlew checkContributionQuality to see your LDR score and design violations before you push.

Security & Edge Cases

• Q: Is my code being sent to a third party?

  A: No. 100% local execution ensures Kafka's IP never leaves our controlled environment.

• Q: Won't failing a CI check generate massive email spam to the dev@kafka mailing list?

  A: No. The validation task will simply fail the CI check exactly like Checkstyle does today. It will output a clear error message in the console pointing to CONTRIBUTING.md. We will not use automated bot comments to avoid webhook noise.

• Q: What if the script has a bug and blocks legitimate human contributors?

  A: The system prioritizes developer velocity. Any Kafka Committer can bypass the gate immediately by adding /aiv skip to their commit message. The CI runner verifies the actor against the committers list to prevent abuse.

• Q: Does a "Green" AIV status mean the PR is safe from sophisticated exploits like ShadowRay?

  A: No. AIV is a first line of defense against structural slop and known design anti-patterns. It does not replace human reviewers for semantic security. Complex, logic-dense code generated by AI could still contain subtle unauthenticated execution paths. A passing AIV score simply ensures the code has enough substance to warrant human architectural review.

• Q: How do we prevent AI models from "faking" logic density by hallucinating redundant method calls?

  A: AI-generated code occasionally attempts to boost its density score by inventing redundant or hallucinated method calls. To combat this, future enhancements (see the Future Work section) outline "BOM-Grounding." This process will cross-reference and verify that the suggested Kafka API methods actually exist in the current trunk before counting them as valid logic nodes.

Rejected Alternatives

• Using LLM-based code review bots (e.g., CodeRabbit): Rejected due to the severe risks of vendor lock-in, data sovereignty violations, and API costs associated with transmitting proprietary Apache codebases to third-party providers. Generic LLMs also lack the domain-specific context to enforce deep Apache invariants.

• Relying solely on human reviewers: Continuing to manually review and close low-substance PRs does not scale and exacerbates maintainer burnout.

• Restricting PR access to Collaborators Only: While GitHub recently introduced this feature to stop bot spam, using it heavily restricts legitimate, first-time open-source contributors from participating in Kafka.

• Using Third-Party GitHub Actions: Using pre-compiled external binaries for PR validation introduces supply-chain security risks and prevents developers from running the exact same checks locally on their laptops. Building it natively into our Gradle scripts solves both issues.

• Adding only AGENTS.md: A soft control that AI models probabilistically ignore. This KIP provides deterministic hard control

...

• .