Page History

Status: Draft

Author: Viquar Khan (Vaquar.khan@gmail.com)

Discussion thread:

...

TBD

Motivation The open-source community is currently seeing a massive increase in automated pull requests. While code generation tools can be helpful to developers, they frequently submit pull requests that look correct on the surface but are actually full of empty boilerplate, unnecessary comments, and made-up dependencieshallucinated logic. This puts a heavy burden on our maintainers, who have to spend their limited time reviewing code that has no real substance.

Other major Apache projects are already dealing with this issue and taking action. For example, Apache Airflow recently had to change their contribution policies because automated bots were taking over open issues without delivering actual work. , and Apache Iceberg also recently had to write strict new contribution guidelines to protect their reviewers.

Why AGENTS.md is not enough: While adding an AGENTS.md or CLAUDE.md file to the repository is a great first step to instruct well-behaving bots, it acts only as a "soft control". AI models use probabilistic reasoning; if their context window gets too full, or if a user overrides the prompt, the agent will silently ignore the markdown instructions and submit the bad code anyway. We cannot rely on the "good intentions" of an AI model to protect our codebase. We need a deterministic "hard control" to catch these failures before they consume a human reviewer's time.

Rather than waiting for our reviewers to burn out, we should be proactive. I propose we add

...

custom, native validation logic to our Gradle build system to mathematically check the quality and substance of a pull request before a human reviewer is ever notified.

Public Interfaces This KIP introduces no changes to the Kafka protocol, public APIs, client behaviors, or broker metrics. It strictly modifies the project's internal build tools and continuous integration (CI) workflow

...

.

Proposed Changes I

...

propose we add two new custom Java-based validation tasks directly into Kafka's buildSrc / build-logic directory. By building this directly into our Gradle configuration, the checks run entirely locally without needing external API keys, introducing zero supply-chain risk, and allowing developers to run them on their laptops just like our existing ./gradlew checkstyleMain tasks.

These tasks will provide a two-layered defense against low-quality automated contributions:

1. Logic Density Validation (LDR) Task AI tools are notorious for generating massive pull requests with very little actual code, such as generating 12 lines of sophisticated documentation for a single line of logic. I

...

propose adding a Gradle task that calculates a "Logic Density Ratio" (LDR), which mathematically measures the amount of logical or executable statements relative to the overall size of a code block. It evaluates the ratio of actual working code against the amount of boilerplate and comments. If a pull request submits 300 lines of setup and documentation but only 2 lines of actual logic, the

...

task will flag it as low-effort scaffolding and

...

fail the build.

2. AST Architecture Linter Task Automated tools often fail to understand Kafka's specific

...

design paradigms. I

...

propose adding a deterministic Abstract Syntax Tree (AST) parser

...

task to systematically enforce our architecture

...

based on our existing contributor guidelines.

Examples of problems solved:

• Concurrency Rules: KafkaConsumer is explicitly not thread-safe

...

• . Automated tools often make the mistake of wrapping it in standard java.util.concurrent threads. This

...

• AST check will automatically

...

• parse the syntax tree to detect and block thread-pool wrapping around consumer instances.

• Configuration Rules:

...

• AI agents frequently assume Kafka is a standard database and introduce dangerous defaults. The parser will block pull requests that

...

• blindly set enable.auto.commit=true (which leads to data loss during failures) or leave critical metrics like retention.ms at unsafe defaults.

• Code Standards: It enforces the usage of the final keyword wherever possible and ensures new public APIs have JavaDocs, as mandated by

...

3. Contributor Instructions (AGENTS.md) We will add an AGENTS.md file to the root of our project. This acts as a clear instruction manual for automated tools, telling them our formatting rules and our requirement to disclose automated code with a "Generated-by" token in the commit message before they even write the code.

Proposed Architecture Workflow

Code snippet

...

• the current Kafka contributor guide.

Compatibility, Deprecation, and Migration Plan This proposal does not change the Kafka protocol , or public APIs, client behaviors, or broker metrics. It only updates our internal Gradle build logic and GitHub Actions workflow.

To ensure this does not slow down regular developers, the scripts proposed tasks will include built-in exceptions:

• Refactor Exception: If a developer is cleaning up code and deleting more lines than they add (net negative lines), the Logic Density check is automatically skipped so legitimate refactoring is never blocked.

• Emergency Override: Anyone can add a /skip-validation flag to their commit message to bypass the checks during an urgent hotfix.

Test Plan To ensure this solution is robust and does not disrupt developer velocity, we will implement the following testing strategy:

1. Unit Testing the Gradle Tasks: The new logic in buildSrc will include its own JUnit tests. We will

...

1. add test cases containing known "AI slop" patterns and concurrency violations to mathematically prove the AST parser and LDR calculator accurately reject bad code while allowing valid code.

2. CI Shadow Mode: We will implement these Gradle tasks and run them in a non-blocking "Shadow Mode" in our CI pipeline for 30 days. It will log its findings on incoming pull requests without actually

...

1. failing the build or notifying contributors. The community can review these logs to ensure there are no false positives before we decide to make it a mandatory, blocking check.

Documentation We will update the CONTRIBUTING.md file to explain the new

...

Gradle checks and

...

instruct contributors to run ./gradlew checkPullRequestQuality locally before submitting a PR.

Rejected Alternatives

...

• Relying

...

• only on human reviewers

...

• : Catching low-effort automated submissions and concurrency mistakes manually does not scale and will lead to maintainer fatigue.

...

Restricting pull requests to collaborators only: While this stops bot spam natively in GitHub, it heavily restricts legitimate first-time contributors and contradicts the open-source ethos.

Subject: SPIP: Automated Pull Request Validation and Architectural Compliance

Q1. What are you trying to do? The Apache Spark project is seeing an increase in automated pull requests. These submissions often pass standard syntax checks but contain padded boilerplate, made-up dependencies, and excessive comments without real logic. I am proposing we add a set of custom, native continuous integration (CI) scripts to mathematically filter out this low-effort code and catch common Spark coding mistakes before a human maintainer reviews the code.

Q2. What problem is this proposal NOT designed to solve? This is not designed to replace human review for complex architecture decisions. It is also not designed to ban automated tools for legitimate human contributors. It only filters out zero-substance submissions and catches known API mistakes natively in GitHub Actions.

Q3. How is it done today, and what are the limits of current practice? Right now, Spark maintainers rely on manual review and standard compiler checks. A pull request with 400 lines of useless documentation and a highly inefficient withColumn loop will pass basic CI tests. A human committer then has to waste time explaining why the code is structurally flawed.

Q4. What is new in your approach and why do you think it will be successful? I have written a set of lightweight validation scripts that I want to donate to the project's CI pipeline. The two main features are:

Logic Density Ratio (LDR): A mathematical formula that checks if a pull request has enough actual execution logic compared to its scaffolding.

...

AST Architecture Linter: A parser that uses a design-rules.yaml file to scan the code structure and enforce Spark's evolving API standards.

Q5. Who cares? Project maintainers and committers who are suffering from review fatigue due to an influx of low-quality, automated pull requests.

Q6. What are the risks? The main risk is a "false positive" where a legitimate pull request is blocked. We prevent this by including a "Refactor Exception" that automatically skips the density check if a developer is deleting code (net negative lines). We also allow trusted maintainers to bypass the checks with a simple commit flag.

Q7. How long will it take? Adding the scripts to our GitHub Actions and writing the initial design-rules.yaml will take about two weeks of testing and feedback on the developer mailing list.

Q8. What are the mid-term and final “exams” to check for success?

• Mid-term: Run the new checks in a non-blocking "shadow mode" for one month to monitor accuracy.

• Final: Achieve a measurable drop in the time maintainers spend reviewing low-quality pull requests.

Appendix A. Proposed Design Rules Examples The AST linter will actively scan the code for common mistakes that automated tools frequently make:

...

Production Mistakes: Flagging pull requests that try to use display(), collect(), or print(df.count()) in core execution paths.

...

Loop Mistakes: Automatically failing pull requests that place withColumn() inside a for loop, prompting the contributor to use select() with multiple columns instead.

...

Deprecation Checks: Rejecting code that introduces outdated configurations.

...

• Using third-party GitHub Actions: Using pre-compiled external binaries introduces supply-chain security risks and prevents developers from running the exact same checks locally on their laptops. Building it natively into our Gradle buildSrc solves both issues.