Skip to content

Claude Review Biased by Outdated & Bot Comments, Ignores Current Code #590

@arthurtaborda

Description

@arthurtaborda

Describe the bug
The PR review generated by Claude is heavily biased by outdated and resolved previous conversation comments. Specifically, Claude includes suggestions or points out issues in its review that were already addressed and resolved in subsequent commits/updates to the Pull Request. This leads to noise and requires manual effort to ignore or dismiss irrelevant automated comments.

The two core problems are:

Stale Comments: The model is biased by previous comments, even if the corresponding code issues are resolved in the latest commit.

Bot Noise: Comments from automated users/bots (e.g., security scanners, dependency checkers) are included in the context, consuming valuable context space and potentially influencing Claude's review with irrelevant, non-human discussion points. We need a way to filter these bot users from the context input.

This bias increases with the length of the PR discussion.

To Reproduce
Steps to reproduce the behavior:

  1. Add a comment to a PR, for example: "Please rename temp_var to user_id in line 42."
  2. Push a new commit to the PR that resolves the issue (i.e., renames temp_var to user_id).
  3. Trigger Claude Code
  4. See error: Claude's review includes a comment reiterating the need to rename temp_var to user_id, despite the code already reflecting this change.

Expected behavior
Claude's review should focus only on the current diff and unresolved concerns. Specifically:

It should use the full PR conversation history to understand context, but it must prioritize the current state of the code.

It should not include comments regarding issues that have been addressed in the latest commit of the Pull Request.

The action should provide a configuration option to filter out comments made by specific GitHub users or bots (e.g., Dependabot, security scanners, or other automated tools).

Screenshots
First comment:
Image

Claude review after fix:

Image

API Provider

[ ] Anthropic First-Party API (default)
[x] AWS Bedrock
[ ] GCP Vertex

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or requestp2Non-showstopper bug or popular feature requestprovider:bedrockAWS Bedrock API

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions