Claude Certification Guide CCAR-P

Curriculum › Evaluation, Testing & Optimization · 16% of the exam

Defining evaluation metrics

What you’ll be able to do

  • Define task-specific accuracy metrics that measure the business outcome, not a convenient proxy
  • Specify latency and cost metrics at the right unit: percentiles, time to first token, cost per successful task
  • Define safety and security metrics, including over-refusal and prompt-injection resistance, as release gates
  • Structure a metric set as one primary target plus guardrail thresholds, segmented by user or input type

What you need to know

Success criteria come before the build

An evaluation metric is a definition of "good enough to ship," written down before anyone tunes a prompt. If you define it afterwards, you tend to pick whatever the current system already does well. Good criteria are specific (what counts as correct for this task), measurable (a number or a pass/fail rule that two people would score the same way), and tied to a decision (the threshold at which you ship, roll back, or escalate). "The assistant should be helpful" is a goal. "At least 90% of answers are fully supported by the retrieved policy text, judged against a rubric" is a metric.

The exam guide names five dimensions. A production system needs a position on each, even if some are thresholds rather than targets.

Accuracy: measure the outcome, not a proxy

  • Pick the measure the task actually allows. Classification and extraction have verifiable answers: exact match, field-level accuracy, precision and recall per class. Open-ended generation needs a rubric: is it grounded in the sources, does it answer the question, does it follow the required format and tone.
  • Weight errors by their cost. In a claims triage system, missing a fraud signal and over-flagging a clean claim aren't equally bad. Report precision and recall separately, or a weighted error, rather than one accuracy figure that treats them alike.
  • For agents, measure the end state. Did the ticket end up correctly resolved, the record correctly updated? Trajectory checks (did it call the right tools in an acceptable order) are useful diagnostics, but the outcome is the metric.

Latency: the right number at the right point

Averages hide the users who wait longest. Track percentiles (p50 and a tail such as p95 or p99) and separate time to first token, which drives perceived responsiveness in a streamed chat, from total completion time, which matters for a back-end pipeline step. For agentic and RAG systems, measure end to end and by stage: retrieval, each model call, each tool call. A latency target that only covers the model call will miss a slow vector search or a tool that retries three times. The configuration choices that trade accuracy against latency are covered in 3.3; here the job is to define what you're measuring.

Cost: per outcome, not per call

Cost per request is easy to compute and easy to mislead with. A cheaper configuration that fails more often, forcing retries, human escalation, or a second agent turn, can cost more per successfully completed task. Define cost at the unit the business pays for: per resolved ticket, per processed document, per thousand active users per month. Break it down by input tokens, output tokens and cached tokens (the API's usage block reports these), because each responds to a different optimization (4.5).

Safety: both directions

Safety metrics measure whether the system produces content or actions it shouldn't: policy-violating output, disallowed advice, unsafe tool actions. They must also measure the opposite failure, over-refusal: declining or hedging on legitimate requests. A system tuned only to drive harmful outputs to zero will often get there by refusing too much, which users experience as broken. Track both rates on a dedicated test set, and treat a regression in either as a release blocker. The guardrails themselves are designed in domain P5; this lesson defines how you'd know they work.

Security: adversarial, and continuous

Security metrics ask whether an attacker can make the system do something it shouldn't. Typical measures: success rate of a maintained suite of prompt-injection attempts (direct, and indirect via retrieved documents or tool results), data-leakage rate (system prompt, other users' data, secrets), and unauthorized tool use (did any test case cause a write action outside the user's permissions). Because new attack patterns appear and every prompt or tool change can reopen a hole, these run as a regression suite on every release, not as a one-time test.

One target, several guardrails, and segments

Optimizing five numbers at once doesn't work; someone will trade one for another without saying so. A practical structure is one primary metric you try to improve (say, resolution accuracy) and guardrail metrics with hard thresholds that must not regress (p95 latency under the SLA, cost per resolution under budget, zero critical safety failures, injection success rate below the agreed bar). A change ships only if it improves or holds the primary metric and passes every guardrail.

Finally, segment every metric. A 94% aggregate can hide a language, document type or customer tier at 70%. Report by the segments the business cares about, and set thresholds per segment where the stakes differ.

Key concept

A metric is only useful if it is defined before the build, measures the outcome the business pays for, and comes with a threshold that triggers a decision; structure the set as one primary target plus guardrails, segmented.

In questions, watch for options that improve an easy-to-measure proxy (average latency, cost per call, a blended score) while the stem describes a business outcome that the proxy doesn't capture. The right answer usually redefines the metric at the correct unit or adds the missing guardrail.

Practice scenario

ScenarioAn insurer's claims-intake assistant is being prepared for launch. The product owner's dashboard shows one "quality score" (an average of five judge ratings), average latency, and cost per API call. Legal wants assurance the assistant won't give coverage advice; the support director complains that in pilots it "refuses half of what people ask." The CFO wants cost cut by a third.
Work it through, then open this

The dashboard can’t answer any of the stakeholders. Replace the blended score with a primary outcome metric, such as the share of claims captured completely and correctly, measured per claim type. Track p95 latency and time to first token, since users watch a streamed reply. Measure cost per completed intake, not per call, so a cheaper configuration that causes more abandoned or escalated intakes doesn’t look like a win. Legal’s concern and the support director’s are two sides of one safety metric set: the rate of disallowed coverage advice on a targeted test set, and the over-refusal rate on legitimate questions. Both become release gates. Then the CFO’s target can be pursued as an optimization under those guardrails rather than at their expense.

Build exercise — Write a metric spec for a real system

Advanced · 40 min

What you’ll learn

  • Turning stakeholder goals into measurable, decision-linked metrics
  • Choosing the right unit for latency and cost
  • Separating a primary target from guardrails

  1. Pick a Claude system you know (or a hypothetical support agent) and list each stakeholder’s goal in their own words, then rewrite each as a metric with a unit, a measurement method, and a threshold.

    • Why: Goals stated as adjectives (“fast,” “safe”) can’t gate a release; a number with a method can.
    • You should see: Five or more rows covering accuracy, latency, cost, safety and security, each with a threshold someone could check without arguing.

  2. Mark exactly one row as the primary metric and every other row as a guardrail, and write the ship rule in one sentence.

    • Why: Without an explicit primary metric, trade-offs between dimensions get made silently.
    • You should see: A rule of the form “ship if the primary metric improves or holds and no guardrail is breached.”

  3. Add the segments you’ll report on (input type, language, customer tier, document type) and decide which guardrails need per-segment thresholds.

    • Why: Aggregate numbers hide concentrated failures, and the exam repeatedly tests that.
    • You should see: At least one segment where you’d accept a different threshold, with the reason written next to it.

Exam traps

Reporting a single blended “quality score” that hides which dimension regressed

A composite moves for reasons nobody can see. Report each dimension separately so a regression points to its cause.

Tracking average latency instead of tail percentiles and time to first token

Averages hide the slowest users, and total time misses what a streaming user perceives. Use percentiles and measure the stage that matters.

Measuring cost per API call instead of cost per successfully completed task

A configuration that’s cheaper per call but fails more often can cost more per outcome once retries and escalations are counted.

Counting only harmful outputs as safety failures and ignoring over-refusal

Driving harmful output to zero by refusing legitimate requests is its own failure. Measure both directions.

Treating security as a one-off penetration test instead of a regression metric

Every prompt, tool or retrieval change can reopen an injection or leakage path, so the adversarial suite runs on every release.

Defining success criteria after the system is built, so the metric fits whatever it already does

Criteria written after the fact describe the system rather than test it. Write them first, with thresholds tied to decisions.

Sources

Quick check

A SaaS company's monitoring shows that, after a release, cost per conversation rose 30% while traffic, model and output length were unchanged. The usage logs show input tokens flat but cache-read tokens near zero, where they had previously covered most of the input. What is the most likely cause to investigate first?