The work a learner
can do alone.

A student finishes an exercise with an AI assistant and receives a good mark. The next question is what the student can explain, check and apply when that assistance is no longer supplying the reasoning.

Both achievements can matter. Completing useful work with a tool has value. Education also aims to develop capabilities that remain available to the learner across changes in task, setting and support. An evaluation needs to say which achievement its score measures.

My public working paper, The Phaedrus Machine: Artificial Intelligence and the Manufacture of Intellectual Dependence, examines the relation between fluent machine output and the user's capacity for judgment. Its public abstract presents a conceptual argument about prior knowledge, verification incentives and evaluation focused on outputs. The concern is that access to a convincing answer can outpace the ability to assess it.

That argument raises empirical questions which need their own evidence. A particular tutoring experiment cannot settle every philosophical claim about understanding. It can test whether a specified form of assistance changes performance on a later task under declared conditions.

Separate the occasions on which the learner is tested

There are at least four useful outcomes to distinguish:

• Assisted performance: the quality and speed of work while the specified help is available.
• Immediate unaided performance: what the learner can do soon afterwards without that help.
• Delayed retention: what remains available after a declared interval, with intervening practice recorded.
• Transfer: whether the learner can use the idea in a materially different problem or recognise when it does not apply.

The timing, permitted resources and task difficulty belong beside each outcome. A harder final question can lower the score even when learning has occurred. Repeating an identical question can reward memory of an answer rather than command of the method. Comparing assisted and unaided scores within one group alone does not identify the effect of the assistance.

Read the experiment at the level it measures

Bastani and colleagues' 2025 PNAS study involved nearly 1,000 students at one Turkish high school across four mathematics sessions. Compared with control, practice scores improved by 48% with GPT Base and 127% with GPT Tutor. On subsequent unaided exams, the GPT Base group performed 17% worse relative to control, about 5.4 percentage points on the score scale. GPT Tutor's unaided result was statistically indistinguishable from control.

GPT Tutor combined teacher-informed hints, solutions supplied to the system and instructions intended to preserve student effort. The experiment therefore compares instructional packages, without isolating the effect of a single prompt sentence. Its outcomes are short-term, from particular tasks and tools. They do not establish permanent loss of an existing skill or a universal effect of AI. Nor does an indistinguishable result establish equivalence or a positive unaided learning gain.

The contrast matters because a service can look impressive during use while its educational consequences remain unresolved. The relevant comparison follows the learner beyond the assisted session and includes a credible group against which to judge the subsequent outcome.

Ask for an explanation that has work to do

Consider an original teaching illustration. A workshop charges an £18 booking fee and £4 for each item. The customer has £50. A learner should be able to explain why the maximum purchase is eight items: £18 + 8 × £4 = £50, while nine would cost £54.

An assistant might supply that answer immediately. A teacher could instead ask the learner to identify the fixed charge, write the budget constraint and check the largest feasible whole number. Recording a correct final answer would not distinguish those routes.

A further check could present a plausible but incorrect answer of twelve items, obtained by forgetting the booking fee. Can the learner locate the missing term and show that the proposed purchase would cost £66? Another problem could change the commercial arrangement, such as charging the booking fee separately for each session, and ask the learner to decide which parts of the original calculation survive.

These are proposed questions, not a validated assessment instrument. Their purpose is to make the desired capability concrete: represent the constraint, justify the boundary and detect an answer that violates it. A useful evaluation would also need unfamiliar examples, suitable difficulty and a fair account of the support each learner is permitted to use.

Evaluate the teaching arrangement

I would design a comparison around the capability being taught before choosing the interface. Learners could receive a declared form of assistance during practice, followed by an immediate assessment and a later assessment under the same stated resource conditions across groups. Prior knowledge, instructional time and task coverage would need to be recorded. Random assignment, where appropriate, would help distinguish the effect of the arrangement from differences in who chose to use it.

The later tasks should include both familiar structure and a change that requires judgment. Scores should distinguish a correct answer from a correct explanation, and an explanation from successful error detection. The marking rules need to be fixed before inspecting which rule favours a preferred product.

Participation in the follow-up also matters. If the learners who struggled most do not return, the retained sample may tell a more reassuring story than the original group. Report who was assigned, who participated at each stage and what remains uncertain. Record additional tutoring or practice between assessments rather than silently attributing every later difference to the original tool.

The cost account should include preparing tasks, checking the assistant's behaviour, maintaining explanations and supporting teachers. A polished interface can depend on substantial instructional work elsewhere. Compare the complete arrangement, including what happens when a problem changes or the learner's mistake was not anticipated.

Keep assistance aligned with the learning objective

Unaided work is appropriate when independent execution is part of the objective. Other tasks legitimately assess skilled use of tools. An engineer may need to choose, verify and integrate a calculation produced by software rather than reproduce every numerical operation by hand. The assessment should identify the judgment the person remains responsible for exercising.

Accessibility support needs the same functional precision. Removing a transcription aid is a different intervention from removing a source of substantive reasoning. The distinction developed in What did the tool contribute? helps define what is being withheld and whether that change measures the intended capability fairly.

A completed answer provides evidence about one performance under one arrangement. Following the learner into explanation, checking and later application makes the educational claim more demanding and more useful. It gives a tutoring service a clear task: help the person acquire the capability the institution says it is teaching.