The population behind
a detector score.

A detector can perform well on one collection and badly on another. The research question is not whether a score looks convincing, but whether its validation travels to the language, task and population in which it will be used.

Two departments buy the same text detector. One screens short examination essays; the other screens technical reports written in several languages. They adopt the same threshold. Even if the software has not changed, the evidential meaning of a flag may have changed considerably.

My published article, Transcription is not generation, asks institutions to distinguish the operation a tool performed from the intellectual work an assessment measures. That distinction creates a second question for detection: what, exactly, was the detector validated to identify? Recognising generated prose and establishing prohibited assistance are different targets. Neither target becomes clearer merely because the interface produces a percentage.

Carry the comparison with the result

In their 2023 Patterns article, Weixin Liang and colleagues compared seven detectors on 91 TOEFL essays and 88 essays by US eighth-grade students. They reported an average false-positive rate of 61.3% for the TOEFL collection, compared with 5.1% for the US collection. The result identifies a serious failure in those tested comparisons. It is not an estimate for every non-native writer, every assessment or every later detector. Read the primary study.

Adnan Al Ali, Jindřich Helcl and Jindřich Libovický revisit the question in a paper published in the 2026 EACL Student Research Workshop proceedings, examining Czech-language texts and three detector families. It reports no systematic bias against non-native speakers in its tested setting, and finds that their texts do not have the lower perplexity observed in the earlier English-language study. The authors also examine performance across domains. In its separate English check, the commercial detector flags 23.1% of the TOEFL essays and none of the Hewlett student essays as generated (Table 5). The Czech finding is therefore not a declaration that bias has disappeared. Read the published paper.

These findings should not be converted into rival slogans. A result about selected English essays and seven earlier tools does not mechanically contradict a result about Czech texts and different detector families. Language, proficiency, text genre, training data, software version and threshold are parts of the comparison. Changing several of them changes the question being answered.

A flag has a denominator

Consider an explicitly hypothetical screening exercise, not an estimate of student behaviour. There are 10,000 texts, of which 100 are wholly generated. Suppose a detector flags 80% of generated texts and 1% of human texts. It would flag 80 generated texts and 99 human texts. Only about 44.7% of the 179 flagged texts would belong to the generated category.

Keep the same detection rate and population, but raise the false-positive rate to 5%. The count of flagged human texts becomes 495; generated texts then account for about 13.9% of the 575 flags. Neither calculation estimates misconduct: the example's generated-text category does not encode the institution's permission rules, and the assumed prevalence is invented. It shows why a benchmark accuracy figure cannot, by itself, supply the probability that a particular accusation is correct.

The practical implication is to report the counts behind a performance claim. Sensitivity, false-positive rate and the composition of the screened population answer different questions. A balanced test collection can be useful for measurement without representing the prevalence in ordinary submissions.

Validate the use, not just the product

An institution could begin with a declared use: a particular assessment, language and type of text, with an explicit rule about permitted assistance. It would then need examples whose provenance is independently established. Treating every unflagged submission as human-written, or every flagged submission as generated, would build the detector's own decisions into its supposed validation.

The evaluation should preserve meaningful distinctions. Wholly generated answers, human-authored work with faithful transcription, substantive machine revision and permitted language support need not be collapsed into one artificial category. Their treatment depends on what the assessment measures and what the governing rule permits. A writing examination and an assessment of mathematical reasoning may legitimately ask different things.

For each relevant group, record the tool version, threshold, text length, sample size and uncertainty around error rates. Where samples are too small, say that the comparison is unresolved. Pooling the results can conceal an important difference; splitting them too finely can produce unstable estimates. Both choices need an explanation.

Keep a route from the score to the work

A flag may prompt a closer examination. The next step should address the intellectual contribution: contemporaneous notes, original calculations, intermediate versions, disclosed assistance and the student's explanation of the reasoning. Those records also have limitations and require evaluation. They are not an automatic certificate of authorship.

The distinction developed in my transcription article remains important here. A faithful conversion can preserve already-authored reasoning; a generated addition can change it. A detector trained on final prose does not necessarily observe that transition. Its performance on a text-classification task therefore cannot replace evidence about the actual workflow.

There is a constructive research programme in the gap. Test whether panels using declared workflow evidence make more accurate and consistent decisions than panels given a detector score alone; measure time and burden as well as errors; and examine whether results survive a change in language or assessment. That is a proposed study, not a reported trial.

Reliable assessment requires more than choosing a tool with an impressive headline. It requires carrying the conditions of validation into the decision, and recognising when those conditions no longer hold.