Separate task-level gains from effects on a wider decision process.
About 40 minutes, including the exercise.
Learning objectives
Separate task-level gains from effects on a wider decision process.
Choose a realistic comparator rather than an idealised human or machine.
Companion reading in Better Answers, Worse Decisions: Introduction; Chapters 1, 4 and 9. The book is for separate purchase. This course does not provide the manuscript or chapter downloads.
The lesson
Choose the unit before choosing the metric
Begin with the decision you are trying to improve. A useful model can reduce the time needed to classify a message, prepare a first analysis or identify a candidate. Those gains are real outcomes, not distractions. But an organisation does more than produce its next answer. It must decide which questions to ask, discover when its usual method has failed, develop people capable of diagnosing that failure and change what happens to those affected.
A measure at one level cannot silently stand in for all the others. Faster completion does not establish that the chosen task was valuable. Accuracy on a fixed test does not establish performance under a changed population. A high approval rate does not establish that a reviewer could reverse a recommendation. Write the unit of analysis next to each claim: answer, task, person, team, organisation or wider field. Then ask what evidence would connect one level to the next.
Compare with the alternative people actually have
The comparison is not machine imperfection against flawless human judgement. The available alternative might involve a long queue, inconsistent review, unaffordable expertise or no service at all. A system can be justified because it improves access even when it is not the best imaginable expert. Conversely, a marginal task improvement may not justify losing the only effective route to appeal.
Use two columns in your decision record. The first records present performance, access and cost. The second records the ability to notice and recover from mistakes over time. Do not combine the columns into an arbitrary numerical score. State which trade-offs are acceptable, who bears them and which outcomes you will observe. If the organisation values a safety threshold or a right of appeal, make that constraint explicit rather than hiding it inside an average.
Make the claim falsifiable
Specify a comparison period, a population, an outcome and a reason the comparison might be misleading. A before-and-after improvement could reflect new staffing or an easier caseload. A carefully randomised task experiment may still say little about long-term skill retention. Neither limitation makes the evidence useless; it defines the question it can answer.
Your final recommendation should include a stopping or revision condition. What observation would make you narrow, suspend or expand the deployment? This is not a ritual statement that more research is needed. It connects a measurable result to a decision and names the person with authority to take it.
Invented for this course
The faster claims team
In this invented example, a claims team processes 1,000 cases per month. Assisted review reduces median handling time from 30 to 20 minutes. The available error audit reports 4% errors before and 3% afterwards, but the mix of claims also changed.
The time saving is 10,000 minutes, or about 167 hours per month, if the same 1,000 cases require the reported reduction. Median times do not by themselves establish that aggregate saving; individual or total-time data are needed to verify it.
The one-percentage-point difference in audited error rates is descriptive. Different case mix prevents a clean causal attribution to the tool.
Add observations on overturned decisions, time to remedy and the ability to detect a newly introduced error. These do not negate speed gains; they test a different outcome.
What follows: Keep the operational improvement claim separate from the causal and institutional claims. Ask for the missing denominator and comparison design before turning the result into a deployment guarantee.
Attempt before opening feedback
Write a two-level evaluation
Choose a fictional automated service. Define two task outcomes and two institutional outcomes, a realistic comparator and one condition that would change your deployment decision.
Review criteria
The comparator is a service that could actually be supplied, not a perfect benchmark.
Each metric has a denominator, observation period and identified limitation.
The decision condition is linked to a named authority, not merely a dashboard.
Worked feedback
For an appointment-triage service, task outcomes might include waiting time and inappropriate routing per 1,000 referrals. Institutional outcomes might include time from a documented anomaly to a service change and successful access to human reassessment. A defensible plan could expand routine triage while reserving a separate route for ambiguous cases. It should explain staffing capacity and not assume that every appeal is evidence of a model error.
Formative practice
Check your reasoning
This is an open self-check, not a secure examination or certificate. Answers and explanations are available below. Nothing you enter is sent to a server.
Keep your working notes
These notes remain only in this tab. They are not submitted or saved automatically. Download a copy before leaving. Use fictional or public information, not confidential personal or client details.