CCAR-P practice › Evaluation, Testing & Optimization · Question 59 of 110

Evaluation, Testing & Optimization · Select TWO

A customer-support assistant must (1) return a JSON object matching a fixed schema, (2) call the order-lookup tool before quoting any delivery date, and (3) write replies that are empathetic and accurately reflect the retrieved order history. The team wants the evaluation to run on every pull request without becoming slow or expensive. Which two grading choices are most appropriate? Select two.

Show answer and explanation

Answer: A and C — Grade schema conformance and the tool-call ordering with code-based checks on every run / Grade empathy and grounding with a rubric-based model judge whose scores are checked against human ratings on a sample

(a) and (c) are right. Schema conformance and whether a tool was called before a date was quoted are verifiable exactly by code, which is fast and deterministic. Empathy and grounding are open-ended, so a rubric-based model judge fits, provided its agreement with human graders is measured. (b) uses a costlier, noisier method where code is exact. (d) uses humans for everything, which can't run on every pull request; humans belong on samples and calibration.