Ethical AI: bias, fairness, transparency

What you’ll be able to do

• Identify where bias enters a Claude system: prompts, few-shot examples, retrieval corpora, input features and proxies, and downstream use
• Evaluate fairness with disaggregated metrics and counterfactual tests rather than aggregate accuracy
• Choose and document a fairness criterion appropriate to the use case, recognizing that criteria can conflict
• Design transparency and accountability: AI disclosure, grounded explanations, documentation, recourse and named ownership

What you need to know

Where bias enters a system you didn’t train

Architects building on Claude don't control pretraining, but they do control nearly everything else, and that's where much of a deployed system's bias actually comes from:

• Prompts and few-shot examples. If every example of a "strong candidate" shares a background, or every escalated complaint example uses one dialect, the model learns the pattern you showed it.
• Retrieval corpora. A knowledge base that covers some products, regions or languages thoroughly and others thinly produces uneven answer quality, and that unevenness often tracks customer groups.
• Inputs and proxies. Names, addresses, postcodes, schools, employment gaps and writing style can stand in for protected characteristics. Dropping a "gender" field doesn't remove gender from a CV.
• Task framing and labels. Historical decisions used as ground truth ("approved", "hired") encode past bias. An eval that rewards matching them rewards reproducing it.
• Downstream use. A summary that's fine for internal triage can be unfair when it becomes the main input to a decision about a person.

The model's own tendencies matter too, and the provider works on them, but your design choices decide whether those tendencies get amplified or contained.

Measure fairness, don’t assert it

Aggregate accuracy hides group-level failure the same way it hides document-type failure (CCAR-F 5.5). Make fairness evaluation part of your eval suite (domain P4):

• Disaggregated metrics. Report error rates, refusal rates, escalation rates, score distributions and answer quality by relevant group or segment (language, region, customer tier and, where lawful and appropriate, demographic attributes), not just overall.
• Counterfactual tests. Hold the case constant and vary only a sensitive attribute or a proxy: a name, a pronoun, a dialect, an address. Outcomes shouldn't change when the attribute is irrelevant to the decision. These tests are cheap with an LLM and very revealing.
• Representative eval data. Make sure smaller groups have enough cases to measure. A segment with five examples can't show you a problem.
• Production monitoring. Track the same segmented metrics after launch, since input mix and corpus coverage drift.

Collecting sensitive attributes to measure fairness can itself raise privacy and legal questions (5.4). Decide with privacy counsel how you'll measure: synthetic counterfactual sets, consented samples or aggregated analysis.

Fairness criteria are a choice, and they can conflict

"Fair" has several technical definitions. Equal selection rates across groups, equal error rates across groups, and equally well-calibrated scores across groups are all reasonable goals, and when base rates differ between groups you generally can't satisfy all of them at once. That isn't a technicality to hide. It's a decision for the business, legal and affected-stakeholder representatives, made explicitly for the use case and documented with its rationale. The architect's role is to make the trade-off visible with data and to build the system so the chosen criterion is measured and enforced.

Transparency: disclose, ground, document

• Disclose AI involvement to the people affected, especially when they're interacting with an assistant or when AI contributes to a decision about them.
• Ground explanations in evidence. Cite the policy clause, the retrieved document or the extracted fields a recommendation relied on. Be careful with the model's generated reasoning: it's useful for debugging and review, but it isn't guaranteed to be a faithful account of how the output was produced, so don't present it to regulators or customers as the audit-grade explanation. Ground the explanation in inputs and rules you can verify.
• Document the system: intended use and out-of-scope uses, data sources, known limitations, evaluation results by segment, the human oversight design and the change history. Make it a living document that's updated with each significant change.

Accountability and recourse

Every AI-assisted decision about people needs a named owner, a way for an affected person to ask for review and contest the outcome, and a human with the authority and the information to change it (5.3). Ethical commitments become real when they're enforced in the design: blocked use cases enforced in code, human review on consequential decisions, segment-level fairness gates in the release process. A "be fair and unbiased" line in the system prompt is fine as guidance. It isn't a control, and it isn't evidence. Check your use case against Anthropic's Usage Policy as well, including any extra requirements it places on higher-risk uses.

Key concept

Fairness is measured per group with disaggregated metrics and counterfactual tests, against a criterion chosen deliberately for the use case, and made accountable through disclosure, evidence-grounded explanations, documentation and a real path to contest decisions.

Questions tend to offer a prompt instruction ("be unbiased"), dropping a protected field, or a high aggregate accuracy as false comfort. Prefer the answer that measures outcomes by group, tests counterfactuals, deals with proxies, or adds recourse and a human owner. When the stem shows stakeholders disagreeing about what "fair" means, the right move is to make the trade-off explicit and get it decided, not to pick a metric silently.

Practice scenario

Build exercise — Run a counterfactual fairness check

Advanced · 45 min

What you’ll learn

• Building counterfactual test pairs that isolate a sensitive attribute or proxy
• Reporting disaggregated results and spotting proxy effects
• Turning findings into a documented fairness criterion and release gate

1. Take 30–50 representative inputs for a Claude feature that affects people and create variants that change only one attribute or proxy (name, pronoun, dialect, location, a career break) while holding the substance constant.

   • Why: Counterfactual pairs isolate the attribute’s effect from everything else, which aggregate metrics can’t do.
   • You should see: A paired dataset where any difference in outcome can be attributed to the changed attribute.

2. Run the feature on all variants and compare the outcomes (scores, decisions, refusal or escalation rates, tone) per attribute, as illustrative script output in a table.

   • Why: Disaggregated, paired comparisons reveal effects that a 90%+ aggregate hides.
   • You should see: Either consistent outcomes across variants, or specific attributes with measurable shifts to investigate in prompts, examples or retrieval.

3. Write a one-page fairness note: the criterion you propose and why, the results, the mitigations, the owner, and a release gate (illustrative: “counterfactual outcome shift below an agreed threshold on the test set before each prompt or model change”).

   • Why: Fairness has to be decided, documented and re-checked on change to stay true.
   • You should see: A criterion that stakeholders can approve or challenge, and a gate that runs automatically with your evals.

Exam traps

Concluding a system is fair because overall accuracy is high

Aggregates hide group-level failure. Report outcomes by segment and test counterfactuals.

Removing a protected attribute and assuming bias is gone while proxies remain

Names, locations, schools and gaps carry the same signal. Test proxies directly and design the task around job- or decision-relevant criteria.

Treating the model’s generated reasoning as a faithful, auditable explanation of how it decided

Generated reasoning helps with review but isn’t guaranteed to reflect the actual process. Ground explanations in verifiable inputs, rules and sources.

Adding a “be unbiased” instruction to the prompt as the main fairness control

It’s guidance, not a control or evidence. Fairness needs measurement, design changes and release gates.

Picking a fairness metric by default instead of deciding it with stakeholders for the use case

Criteria can conflict. Make the trade-off explicit and get it decided and documented by the people accountable.

Deploying an AI-assisted decision without disclosure, a way to contest it, or a named accountable owner

Transparency and recourse are part of the design. Affected people need to know, to be able to challenge the outcome, and to reach a human with authority to change it.

Sources

• Claude Developer Platform Docs
• Anthropic
• Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)