Distinguish illustrative cases, coding agreement and causal identification.
About 50 minutes, including the exercise.
Learning objectives
Distinguish illustrative cases, coding agreement and causal identification.
Revise a claim when an alternative reading changes its apparent support.
Companion reading in Better Answers, Worse Decisions: Note on evidence; Chapter 14; Conclusion. The book is for separate purchase. This course does not provide the manuscript or chapter downloads.
The lesson
Match the claim to the evidence
A vivid case can establish that a mechanism is possible and reveal how a process worked. It does not establish how common the mechanism is. A selected comparison can sharpen a question without identifying a causal effect. A randomised task experiment can support a causal claim in its tested setting while leaving institutional and long-term questions unresolved. Begin by stating the type of evidence and the strongest conclusion it can bear.
Keep observation, interpretation and proposal separate. A document may show that an objection was raised. Whether the objection supplied an effective independent challenge can require judgement. Whether a new governance design would prevent recurrence is a further claim. Moving between these levels without an explicit argument creates apparent certainty rather than evidence.
Treat disagreement as information
Two coders can agree often while disagreeing on the few cells that drive the conclusion. Report where disagreement occurs, not only an overall percentage. Test whether the substantive association survives each retained coding and a defensible adjudication rule. Choose the rule for its evidential justification, not because it preserves the preferred result.
A model-based recoding exercise is not independent human validation. Shared training or model lineage can correlate errors. Even genuinely independent human agreement does not make a selected sample representative or prove causation. These limitations do not require abandoning a conceptual framework. They require narrowing the empirical claim and stating which further observation would support or challenge it.
Defend a design without claiming a theorem of institutions
A correction architecture can be a useful proposal even when no universal causal law has been established. Defend it by identifying the failure it targets, its operational mechanism, its costs and the conditions under which it should be revised. Include a strong counterexample: a setting where standardisation improves reliability, independent checks add little, or rapid automation expands access.
Your capstone should therefore contain a decision and a research plan, not merely a list of virtues. State which controls you would implement now, which you would test, and which you would reject. Preserve the distinction between a completed exercise and a verified real-world deployment. The final product is an argument that another person can inspect, challenge and improve.
Invented for this course
Agreement that hides the decisive dispute
In a fictional ten-case exercise, two coders agree on 54 of 60 capacity ratings. All six disagreements concern whether stop authority existed in the cases used to support the main conclusion.
Report the 90% overall agreement, but also identify the disputed field and the affected cases.
Recompute the proposed association under each coding rather than averaging away the disagreement.
Inspect the underlying evidence and apply an explicit rule. If the association changes, qualify or withdraw that empirical claim while retaining separately supported design arguments.
What follows: High overall agreement is not enough when disagreement is concentrated in the evidence that determines the result.
Attempt before opening feedback
Write the strongest objection
Take your proposed correction plan. Write a 200-word objection explaining how it could waste resources, entrench an incumbent or obstruct useful automation. Then revise the plan or defend it with a test that could prove you wrong.
Review criteria
The objection attacks a real weakness rather than a caricature.
The response changes a claim, a design choice or an evidence requirement.
The proposed test has a result that would genuinely alter the recommendation.
Worked feedback
For example, a second reviewer may share the same source and add delay without useful independence. A better design could replace universal duplicate review with targeted reconciliation against original records. The test should compare material error detection and delay, not simply count how many checks were completed.
Formative practice
Check your reasoning
This is an open self-check, not a secure examination or certificate. Answers and explanations are available below. Nothing you enter is sent to a server.
Keep your working notes
These notes remain only in this tab. They are not submitted or saved automatically. Download a copy before leaving. Use fictional or public information, not confidential personal or client details.