Identify which capabilities must survive routine automation.
About 45 minutes, including the exercise.
Learning objectives
Identify which capabilities must survive routine automation.
Design an observable recovery exercise rather than relying on confidence.
Companion reading in Better Answers, Worse Decisions: Chapters 5, 6, 7 and 11. The book is for separate purchase. This course does not provide the manuscript or chapter downloads.
The lesson
Retain a capability, not every old task
Automation changes the work through which people gain experience. If a system handles routine cases, the remaining human task may be unusual, urgent and poorly practised. It does not follow that every old manual activity must be preserved. The relevant question is which knowledge is required to recognise a failure, diagnose it and operate a safe alternative.
Separate three abilities: noticing that the result may be wrong, identifying the cause, and restoring a useful service. Someone may be excellent at one and unable to perform the others. A manager who recognises an anomaly still needs access to the evidence and specialists who can investigate it. A specialist who understands a failure may lack permission to suspend the process.
Exercise the abnormal condition
A fallback plan should be tested under the conditions that make it necessary. An exercise in which every record is available and the original system continues to provide answers may test ordinary operations rather than recovery. Simulate a specific interruption, preserve safety and record the time needed to complete a defined task without the unavailable dependency.
Use the exercise to learn, not to punish a person for finding a weakness. Record missing documentation, inaccessible tools, unclear responsibilities and decisions that require information no one retains. An unsuccessful rehearsal can be valuable evidence if it leads to a concrete repair. A successful rehearsal has a scope: it does not prove recovery from a different failure or a larger disruption.
Fund learning deliberately
When routine practice disappears, an organisation may need simulation, rotation, supervised exceptions or periodic independent work to reproduce diagnostic skill. Each has a cost. Compare that cost with the consequences and likelihood of needing the capability, while recognising uncertainty. Do not justify unlimited training through a vaguely catastrophic scenario.
Track whether people can perform the required recovery tasks over time. Confidence surveys and course attendance are weak substitutes for demonstrated performance. Review whether the maintained skill is still relevant as the system changes. Retaining obsolete procedures can waste resources just as surely as losing an essential competence can create exposure.
Invented for this course
The unavailable recommendation service
A fictional planning team can process routine orders with an automated recommendation service. Its continuity plan says that staff will switch to manual allocation if the service fails.
Choose a representative set of orders and withhold the recommendation output during a safe rehearsal.
Measure whether staff can find the necessary constraints, make an allocation and explain unresolved conflicts within the required time.
If the team cannot access a critical inventory record, repair that dependency and repeat the relevant exercise. Training alone cannot fix missing access.
What follows: Recovery depends on a system of people, information and authority. It is not a personal attribute that can be assumed from job titles.
Attempt before opening feedback
Design a recovery drill
Specify one failure, the capability needed to recover, a safe rehearsal, success criteria and the person responsible for acting on its findings.
Review criteria
The exercise actually removes or changes the dependency under test.
The success criteria measure performance rather than attendance or confidence.
The plan names a repair and retest process for failed elements.
Worked feedback
For a scheduling tool outage, a useful drill might require staff to produce a feasible allocation from preserved inputs within a defined period. It should record constraint violations and unresolved cases, not merely whether someone produced a spreadsheet. Do not introduce real service risk just to make a rehearsal realistic.
Optional extension · about 15 minutes, additional to core course time
Extension: does the fallback survive the missing service?
Review four invented rehearsal records for an AI-assisted order-planning team. The failure being tested removes both the recommendation service and its shared record store. The fallback must therefore use a separately maintained, accessible copy of the necessary inputs and constraints. Hiding the recommendation screen while leaving the shared store available does not reproduce this failure. This is a paper exercise, not a report of an actual drill or a request to interrupt a real service.
The fictional acceptance rule is fixed before the rehearsal: from the moment of the outage, finish all 12 test orders by the end of minute 8, with no constraint violations, using no resource removed by the stated failure. The test orders and constraints are fixed for this case. The approved offline copy is complete and current for them. An order counts only after its required checks are complete; sending an unchecked plan does not count as recovery.
After activation, two independent operator stations work in parallel. Each station completes one different order, including its checks, every 1 minute. Orders cannot be split between stations, no orders arrive during the drill, and no other delays occur. Activation finishes before either station starts. These simplifying assumptions are stipulated for the calculation, not measured productivity or a general staffing rule. Each row is a separate rehearsal of the same 12-order task.
Four fictional recovery records; the target is 12 checked orders by minute 8
Run
Activation time
Reported result
Evidence about the fallback
A
1 minute
All 12 orders by minute 7; no constraint violations.
The recommendation screen was hidden, but operators still queried the shared record store.
B
3 minutes
All 12 orders by minute 9; no constraint violations.
Both unavailable services were blocked in the isolated rehearsal; the approved offline copy was used.
C
2 minutes
All 12 orders by minute 8; 2 orders violate constraints.
Both services were blocked; operators used an obsolete offline copy instead of the approved current copy.
D
2 minutes
All 12 orders by minute 8; no constraint violations.
Both services were blocked; operators used the approved current offline copy and retained checked results.
Attempt first
Which run meets all three acceptance conditions? For each other run, name the particular condition that fails. Explain why the quickest result does not establish recovery from the stated failure.
Reconstruct Run B. How long does processing take after activation? How many orders are complete at minute 8, and how many remain? What is the latest activation completion time compatible with the deadline under these assumptions?
Repeat Run D with one operator station unavailable from the start. Keep the same activation time and per-station rate. Find the earliest possible completion time and the number finished by minute 8. What does this reveal about the scope of the two-station result?
For A, B and C, propose a specific repair and retest. Write a short acceptance record containing the resources made unavailable by the simulated failure, input version, available staff and tools, elapsed times, constraint results, unresolved limits, and the role responsible for deciding whether the fallback can be relied upon.
Recovery drill: worked feedback and evidence limits
D meets the stated deadline, correctness and dependency conditions in this invented record. A is fast and its outputs are correct, but it still relies on the store that the specified outage removes. It establishes performance with that store available, not recovery without it. B has the right dependency boundary and correct results but misses the deadline. C is on time and avoids both services but fails the constraint condition. Calling either B or C a success would change the predeclared acceptance rule after seeing the result.
Two stations complete 12 orders in 6 processing minutes. In B, total time is 3 + 6 = 9 minutes. At minute 8, the stations have worked for 5 minutes and completed 10 orders; 2 remain. With the same task and rate, activation must finish by minute 2 because 8 − 6 = 2. Completing exactly at minute 8 is allowed by this case, but leaves no time margin for any additional delay.
With one station, D takes 2 + 12 = 14 minutes. At minute 8 it has completed only 6 orders. The two-station result therefore supplies no evidence that the same deadline can be met with one station. A plan may add a tested reserve, define a separately authorised reduced service, or revise its requirement through the responsible decision-maker. It cannot claim the original 12-order target was met by silently omitting orders or dropping checks.
For A, repeat a safe isolated drill with both the recommendation service and the shared store unavailable, and verify the separate input path. For B, investigate the activation delay and test a specific repair; a shorter proposed time is not an observed improvement. For C, repair how the current input version is selected and checked, then rerun the constrained task. Do not diagnose these different failures as a lack of individual skill without evidence. Missing access, outdated data and slow mobilisation can require different remedies.
Keep the original failed record alongside the repair and new result. Record the procedure and input versions, actual resources used, the timing start and finish, completed orders and violations, and who may accept the remaining limits. Passing D does not establish recovery under a longer outage, new orders, stale inputs or absent staff. For operational evidence, people would need to perform the tasks safely in the stated isolated conditions and retain the observations. Reading this scenario or discussing its answer does not perform that test.
Sources and scope
NIST SP 800-34 Rev. 1 (2010, updated 11 November 2010), sections 3.5.1–3.5.3, distinguishes testing, training, discussion-based tabletop exercises and functional exercises in a simulated operational environment. NIST AI RMF 1.0 (2023), MAP 3.4 and MEASURE 2.3, addresses documented operator proficiency and evaluation under conditions similar to use. The outage, records, timings and acceptance rule here are original teaching constructions. They are not NIST results, a universal recovery threshold or evidence that any real team has recovered successfully.
This is an open self-check, not a secure examination or certificate. Answers and explanations are available below. Nothing you enter is sent to a server.
Keep your working notes
These notes remain only in this tab. They are not submitted or saved automatically. Download a copy before leaving. Use fictional or public information, not confidential personal or client details.