CCAR-P practice › Evaluation, Testing & Optimization · Question 59 of 110
Evaluation, Testing & Optimization · Select TWO
A customer-support assistant must (1) return a JSON object matching a fixed schema, (2) call the order-lookup tool before quoting any delivery date, and (3) write replies that are empathetic and accurately reflect the retrieved order history. The team wants the evaluation to run on every pull request without becoming slow or expensive. Which two grading choices are most appropriate? Select two.
Show answer and explanation
Answer: A and C — Grade schema conformance and the tool-call ordering with code-based checks on every run / Grade empathy and grounding with a rubric-based model judge whose scores are checked against human ratings on a sample
(a) and (c) are right. Schema conformance and whether a tool was called before a date was quoted are verifiable exactly by code, which is fast and deterministic. Empathy and grounding are open-ended, so a rubric-based model judge fits, provided its agreement with human graders is measured. (b) uses a costlier, noisier method where code is exact. (d) uses humans for everything, which can't run on every pull request; humans belong on samples and calibration.
Keep going
More Evaluation, Testing & Optimization questions
- An LLM judge is used to compare two versions of a research summarizer pairwise. The new…
- An agent that files insurance claims passes a test case on its first run, and the team…
- A travel company's booking assistant is a multi-turn chat. The team wants to A/B test a…
- After an online test, a new retrieval configuration for an internal IT helpdesk…
- A team is comparing two prompt variants for a claims-summary feature over a four-week…