What you’ll be able to do
- Replace vague review instructions with specific criteria that define which issues to report and which to skip
- Explain why confidence-based instructions fail to improve precision compared with categorical criteria
- Protect developer trust by temporarily disabling high false-positive categories while their prompts are improved
- Define severity levels with concrete code examples so classification stays consistent
What you need to know
Vague instructions leave the decision boundary to guesswork
An automated reviewer told to "check that comments are accurate" has to invent its own definition of accurate. It will flag stale wording, imprecise phrasing, comments that are merely incomplete, and the occasional comment that is genuinely wrong, and it will do so inconsistently from one run to the next. The instruction states a goal, not a condition for reporting.
Compare: "Flag a comment only when the behavior it claims contradicts what the code actually does." That sentence is testable. For any comment, the model can check the claim against the code and give a yes or no answer. Everything that isn't a contradiction (terse comments, outdated TODOs, style) is out of scope by definition. Precision improves because the category is narrower and checkable, not because the model has been asked to try harder.
“Be conservative” doesn’t move the boundary
The tempting fix for a noisy reviewer is a general instruction: "be conservative," "only report high-confidence findings," "avoid nitpicks." These feel like they should cut false positives, but they don't tell the model which findings to drop. The model has no reliable sense of which of its findings are wrong. If it did, it wouldn't produce them. So it trims somewhat at random, often losing real issues along with the noise, and the false positive rate in the worst categories barely changes.
Specific categorical criteria work where confidence filtering doesn't. They tell the model what is in scope and what isn't:
Do not report:
- Naming, formatting, or other minor style preferences
- Patterns used consistently elsewhere in this codebase (local conventions), even if you would write them differently
- Suggestions for refactors that don’t fix a defect
The skip list matters as much as the report list. Most false positives in code review come from style opinions and from flagging a team's local conventions as problems. Naming those explicitly as out of scope removes whole classes of noise at once.
False positives cost trust across every category
Developers don't judge a reviewer finding by finding. If one category, say "possible performance issue," is wrong most of the time, people learn to skim or dismiss the bot's output as a whole, and the accurate security findings sitting next to the noise get ignored too. A high false positive rate in one category undermines confidence in the categories that are working.
That makes a practical remedy legitimate: temporarily disable the high false-positive category while you rework its prompt. Turning off a category that produces some real findings feels like losing coverage, but leaving it on costs more, because its noise drags down how seriously people take every other finding. Restore trust first by shipping only the categories that are reliably right, improve the noisy category's criteria offline against real examples, and re-enable it once its precision holds up.
Severity needs examples, not adjectives
"Critical," "major," and "minor" mean different things to different readers, and to the model on different runs. Defining them with adjectives ("critical: severe issues") produces inconsistent classification: the same missing null check is "major" in one file and "minor" in the next.
Define each severity level by its criterion and a concrete code example:
MAJOR — wrong behavior on a realistic input, no security impact. e.g. for i in range(len(items) - 1): # silently skips the last item
MINOR — correct today but fragile; would break under a plausible change. e.g. status == 3 # magic number duplicated from another module
Examples give the model a reference point it can compare each new finding against, which is what makes classification consistent. They also make disagreements easy to settle: a reviewer can point to the example the finding most resembles.
The same principle beyond code review
Explicit criteria beat vague instructions anywhere a model makes a judgment call. A support agent told to "escalate anything difficult" escalates inconsistently. One given explicit triggers (customer asks for a human, request needs a policy exception, no progress possible with available tools) plus a few examples behaves predictably. Proxies like sentiment are a version of the same failure: they substitute an easy-to-measure signal for the actual criterion.
Key concept
To raise precision, define what to report and what to skip in specific, checkable categories, and anchor severity levels with code examples. Instructions to “be conservative” or report only high-confidence findings don’t tell the model which findings are wrong, so they don’t fix false positives.
When a scenario describes developers ignoring an automated reviewer, look for the noisiest category first. Narrowing its criteria, or switching it off until they're fixed, usually does more than any general instruction about confidence.
Practice scenario
Work it through, then open this
The confidence instruction won’t fix it: the model can’t tell which of its code-quality findings are wrong, so it will cut some at random and may drop real security issues too. The root cause is that the code-quality category has no explicit criteria and sweeps in style and local conventions. Disable that category now so the reliable security findings regain credibility. Then rewrite its criteria with an explicit report list (defects only) and skip list (naming, formatting, patterns used consistently in this codebase), test it against the dismissed findings, and re-enable it once its precision is acceptable.
Build exercise — Turn a vague review prompt into explicit criteria
Intermediate · 30 min
What you’ll learn
- Writing report and skip criteria that a model can check
- Using dismissal data to decide which category to disable first
- Anchoring severity levels with code examples
-
Collect 20–30 findings from an existing review prompt and label each as accepted or dismissed, grouped by category.
- Why: False positives cluster by category, and you need to know which category is eroding trust before you change anything.
- You should see: One or two categories with far higher dismissal rates than the rest, usually style- or convention-related.
-
Rewrite the prompt with an explicit “report” list and an explicit “do not report” list, and disable the worst category entirely for now.
- Why: Categorical criteria change what the model looks for; switching off the noisiest category protects trust in the accurate ones while you fix it.
- You should see: A prompt with no “be conservative” language, and a re-run in which the dismissed findings from step 1 largely disappear while accepted ones remain.
-
Add a severity section with one concrete code example per level, then re-classify the accepted findings twice.
- Why: Example-anchored definitions give the model a fixed reference, which is what makes severity consistent across runs.
- You should see: The same finding receiving the same severity on both runs, where adjective-only definitions previously varied.
Exam traps
Adding “be conservative” or “only report high-confidence findings” to fix false positives
General caution doesn’t tell the model which findings are wrong, so it doesn’t reliably improve precision. Specific categorical criteria do.
Writing criteria as goals (“check that comments are accurate”) instead of report conditions
A goal leaves the model to invent the boundary. A condition like “flag only when claimed behavior contradicts the code” is checkable.
Filtering findings by the model’s self-reported confidence instead of by category
Confidence-based filtering trims noise and real issues alike. Define which issue types to report (bugs, security) and which to skip (minor style, local patterns).
Keeping a noisy category enabled because some of its findings are real
A high false positive category teaches developers to ignore the bot entirely. Disable it temporarily, improve its prompt, then bring it back.
Defining severity levels with adjectives alone and no code examples
“Critical” and “minor” drift between runs without a reference point. A concrete code example per level makes classification consistent.
Treating false positives as a cosmetic annoyance rather than a trust problem
Noise in one category undermines confidence in accurate categories, which is how real findings end up ignored.
Sources
- Claude Developer Platform Docs
- Anthropic, Claude Certified Architect – Foundations Exam Guide v1.0 (July 2026)