Separate task-level gains from effects on a wider decision process.
About 40 minutes, including the exercise.
Learning objectives
Separate task-level gains from effects on a wider decision process.
Choose a realistic comparator rather than an idealised human or machine.
Companion reading in Better Answers, Worse Decisions: Introduction; Chapters 1, 4 and 9. The book is for separate purchase. This course does not provide the manuscript or chapter downloads.
The lesson
Choose the unit before choosing the metric
Begin with the decision you are trying to improve. A useful model can reduce the time needed to classify a message, prepare a first analysis or identify a candidate. Those gains are real outcomes, not distractions. But an organisation does more than produce its next answer. It must decide which questions to ask, discover when its usual method has failed, develop people capable of diagnosing that failure and change what happens to those affected.
A measure at one level cannot silently stand in for all the others. Faster completion does not establish that the chosen task was valuable. Accuracy on a fixed test does not establish performance under a changed population. A high approval rate does not establish that a reviewer could reverse a recommendation. Write the unit of analysis next to each claim: answer, task, person, team, organisation or wider field. Then ask what evidence would connect one level to the next.
Compare with the alternative people actually have
The comparison is not machine imperfection against flawless human judgement. The available alternative might involve a long queue, inconsistent review, unaffordable expertise or no service at all. A system can be justified because it improves access even when it is not the best imaginable expert. Conversely, a marginal task improvement may not justify losing the only effective route to appeal.
Use two columns in your decision record. The first records present performance, access and cost. The second records the ability to notice and recover from mistakes over time. Do not combine the columns into an arbitrary numerical score. State which trade-offs are acceptable, who bears them and which outcomes you will observe. If the organisation values a safety threshold or a right of appeal, make that constraint explicit rather than hiding it inside an average.
Make the claim falsifiable
Specify a comparison period, a population, an outcome and a reason the comparison might be misleading. A before-and-after improvement could reflect new staffing or an easier caseload. A carefully randomised task experiment may still say little about long-term skill retention. Neither limitation makes the evidence useless; it defines the question it can answer.
Your final recommendation should include a stopping or revision condition. What observation would make you narrow, suspend or expand the deployment? This is not a ritual statement that more research is needed. It connects a measurable result to a decision and names the person with authority to take it.
Invented for this course
The faster claims team
In this invented example, a claims team processes 1,000 cases per month. Assisted review reduces median handling time from 30 to 20 minutes. The available error audit reports 4% errors before and 3% afterwards, but the mix of claims also changed.
The time saving is 10,000 minutes, or about 167 hours per month, if the same 1,000 cases require the reported reduction. Median times do not by themselves establish that aggregate saving; individual or total-time data are needed to verify it.
The one-percentage-point difference in audited error rates is descriptive. Different case mix prevents a clean causal attribution to the tool.
Add observations on overturned decisions, time to remedy and the ability to detect a newly introduced error. These do not negate speed gains; they test a different outcome.
What follows: Keep the operational improvement claim separate from the causal and institutional claims. Ask for the missing denominator and comparison design before turning the result into a deployment guarantee.
Attempt before opening feedback
Write a two-level evaluation
Choose a fictional automated service. Define two task outcomes and two institutional outcomes, a realistic comparator and one condition that would change your deployment decision.
Review criteria
The comparator is a service that could actually be supplied, not a perfect benchmark.
Each metric has a denominator, observation period and identified limitation.
The decision condition is linked to a named authority, not merely a dashboard.
Worked feedback
For an appointment-triage service, task outcomes might include waiting time and inappropriate routing per 1,000 referrals. Institutional outcomes might include time from a documented anomaly to a service change and successful access to human reassessment. A defensible plan could expand routine triage while reserving a separate route for ambiguous cases. It should explain staffing capacity and not assume that every appeal is evidence of a model error.
Optional extension · about 12 minutes, additional to core course time
When the overall score falls while both groups improve
Use paper or a calculator for this optional, no-cost exercise. All counts are invented. A fictional document-routing service is evaluated against independently adjudicated reference labels in two periods. Each record belongs to exactly one of two difficulty groups, defined before evaluation.
The populations differ between periods. Calculate what the observations show before deciding whether the service improved. No random assignment, uncertainty model or evidence about why the case mix changed is supplied.
Hypothetical correct routes and evaluated records
Period
Routine: correct / evaluated
Complex: correct / evaluated
Overall: correct / evaluated
A
80 / 100
8 / 20
88 / 120
B
18 / 20
50 / 100
68 / 120
Attempt first
Calculate accuracy within each group and overall in each period. Which comparisons point in different directions?
For a fixed reference mix containing equal shares of routine and complex records, calculate the weighted accuracy in each period. Explain what this comparison changes and what it does not establish.
Write a two-sentence finding that preserves both the observed overall result and the group results. Name one further check before changing deployment.
Worked comparison and limits
Routine accuracy rises from 80% to 90%; complex accuracy rises from 40% to 50%. Overall accuracy falls from 88 / 120, approximately 73.33%, to 68 / 120, approximately 56.67%. Complex records account for 20 / 120 in A but 100 / 120 in B.
With the stipulated 50:50 reference mix, A scores (80% + 40%) / 2 = 60% and B scores (90% + 50%) / 2 = 70%. This standardised comparison holds the group weights fixed. It does not replace the observed workload or establish a causal effect of the service.
A defensible finding is: accuracy rose within both specified difficulty groups, while the observed overall accuracy fell as the workload shifted toward complex records. An equal-weight comparison favours B, but neither comparison identifies why performance changed.
Before changing deployment, check group definitions, reference-label quality, remaining differences within groups, uncertainty and consequences of misrouting. The small invented counts provide no significance claim. A 50:50 mix is a teaching assumption, not a recommendation for the real service.
Sources and scope
NIST AI RMF 1.0 (2023), Section 3.1, calls for realistic test sets and documented methods, and allows accuracy to be reported for separate data segments. This original exercise illustrates why the evaluation population matters. NIST did not supply the counts or the 50:50 reference mix. The source page notes that a revised framework is in progress.
Optional extension · about 15 minutes, additional to core course time
When the test set becomes the design brief
These are invented teaching counts, not measurements of an AI product. A team tries three prompts on the same 100 labelled review cases. After each attempt it inspects every error and changes the next prompt. It selects C. Before opening a separate 100-case batch, it freezes C, its scoring rule and the case-selection procedure. The separate batch has not informed training, prompt design or selection.
The two batches are disjoint, use the same stated target task and independently checked reference labels. The table alone does not establish representative sampling, absence of other leakage or comparable difficulty. Treat those as audit questions, not facts proved by the scores.
Two batches with different roles (constructed counts)
Prompt
Reused batch: correct / total
Separate batch: correct / total
Role
A
80 / 100
Not tested
Development
B
88 / 100
Not tested
Development
C
94 / 100
82 / 100
Frozen before separate evaluation
Attempt first
Calculate C’s accuracy on each batch and the difference in percentage points. Is the 12-point difference a measured causal effect of prompt tuning?
Can the team claim an independently tested improvement of 14 percentage points over A? Which comparison is missing?
The team now reads the separate batch’s errors and changes C to D. Can it report D on that same batch as an untouched final evaluation? Explain how the batch’s role changes.
Write a four-line evaluation record: what was frozen, who could see which labels, how cases were selected, and what evidence is needed next. Include duplicate cases, shared people/documents and time ordering where relevant.
Worked feedback: what the test can still tell you
C scores 94% on the reused batch and 82% on the separate batch: a difference of 12 percentage points. The difference could reflect selection, case difficulty, sampling variation or other factors. It is not a causal estimate of the effect of tuning, nor an exact estimate of selection bias.
The 14-point gain (94 − 80) is on the batch used for adaptation. A was not tested on the separate batch, so that table cannot establish a held-out gain over A. A pre-specified paired comparison on suitable new cases would supply the missing comparison; report uncertainty and case selection, not just an aggregate score.
Once those errors guide D, that batch contributes to development. Its observations remain useful, but it is no longer an untouched evaluation for D. Preserve C’s frozen result and version history. Evaluate the newly frozen D through a properly separated procedure; merely renaming the old batch does not restore separation.
Keep prompt and pipeline versions, scoring rule, label-access log and sampling record. Split by the unit or time structure needed for the intended claim, and check duplicates and shared upstream records. A clean holdout is a procedural safeguard, not proof that an AI system is safe or effective in every setting.
Sources and scope
The scikit-learn guide explains that tuning against test results can compromise their evaluation role and distinguishes model selection from final testing. This original exercise applies that principle to prompt revision. The source supplies neither these counts nor a claim about any particular AI product.
This is an open self-check, not a secure examination or certificate. Answers and explanations are available below. Nothing you enter is sent to a server.
Keep your working notes
These notes remain only in this tab. They are not submitted or saved automatically. Download a copy before leaving. Use fictional or public information, not confidential personal or client details.