Research commentary · AI & scientific evidence
The observation behind
five citations.
Five papers can contain five observations. They can also contain five descriptions of one observation. A research system needs to know which it has before it turns repetition into confidence.
My 2025 BEWA article proposes an architecture for updating beliefs about structured scientific claims. Its public abstract describes author attribution, replication history, citation weighting and the handling of contradiction. That is a useful setting for a narrower question: what should an evidence system preserve when several documents depend on the same underlying result?
The following calculation is an original, hypothetical illustration. It is not a reported BEWA experiment, an assessment of the paper's proofs or a claim that the architecture has solved dependence in scientific evidence. It shows why a document graph needs an account of the observations behind it.
Start with the event, not the bibliography
Consider a precisely defined claim H. Before a test, give H a probability of 10%. Suppose the specified positive test outcome has probability 80% if H is true and 20% if H is false. These are stipulated properties of an imaginary test, not estimated properties of a real scientific field.
The likelihood ratio for that positive outcome is four: 0.8 divided by 0.2. Prior odds of one to nine become four to nine. The updated probability is therefore 4/13, approximately 30.8%.
Now suppose five papers report that same test result. They add no further measurements, and their publication decisions supply no additional evidence in this example. Reading the fifth description does not change the likelihood of the original outcome. The probability remains 30.8%. The documents may add interpretation, criticism or useful context, but their number alone does not produce five independent tests.
Say what would justify a second update
A different experiment gives a different answer. Suppose five separately conducted tests all return positive, each has the same stipulated operating characteristics, and their outcomes are conditionally independent both when H is true and when H is false. The likelihood ratios can then be multiplied. The posterior odds become 4 to the fifth power, divided by nine; the probability is 1024/1033, approximately 99.1%.
The large difference between 30.8% and 99.1% is not produced by a more sophisticated numerical routine. It is produced by an assumption about how the evidence was generated. A correct calculation can be a poor account of the science if the input treats repeated reporting as repeated observation.
Conditional independence is stronger than having different author lists or different institutions. Teams can share a dataset, instrument, measurement error, selection procedure or analysis code. Even genuinely new measurements may share a failure mechanism. Conversely, two analyses of one dataset may ask different questions and supply useful information. They should not be discarded simply because their provenance overlaps. Their contribution must be described at the level at which it changes the evidence.
Represent the shared dependency
Imagine the five reports form two evidence families. Four are exact restatements of one positive test; the fifth reports a second positive test, conditionally independent of the first under both hypotheses. With the same test properties and prior, the probability is now 16/25, or 64%. The appropriate unit of the update is the two tests, not the five documents.
This is a deliberately clean case. For partly dependent results, counting families is not generally enough. The relevant quantity is the probability of the joint evidence under each hypothesis. A researcher may need an explicit dependence model, a sensitivity analysis over plausible models, or a qualified statement that the available information does not support a precise posterior.
That limitation is informative. A system that records an unknown dependency is doing more useful work than one that supplies an impressive decimal by silently assuming the dependency away.
Keep interpretation and observation connected
An evidence record should identify the exact claim, the observation or dataset, its version, the method, the relevant population and the route by which the document obtained it. A replication label should say what was repeated. Re-running an analysis checks something different from collecting new observations with the same instrument, and both differ from testing the claim with a materially different measurement procedure.
A later paper can also reveal an error in an earlier analysis without collecting new data. That changes the interpretation of the original evidence. The correction should propagate to claims that depend on the erroneous result, while preserving the original record and the reason for the change. It should not merely add another favourable or unfavourable vote to an accumulated count.
The June 2025 BEWA preprint is a separately identified public version with a different title and length from the journal article. Keeping those identities separate is itself part of evidence management. A later title, a repository date and a publisher date are not interchangeable evidence of what a reader actually examined.
Test the evidence system with a controlled substitution
A useful engineering test is to take one observation record and attach five documents that merely repeat it. The evidence contribution of that observation should not increase solely because the descriptions multiplied. Then replace one duplicate with a genuinely independent second observation and check that the update changes according to the declared model.
A third test introduces a correction to the shared observation. The system should locate its dependants and distinguish claims affected by that correction from claims supported by other evidence. These are proposed tests, not results from a deployed system or an audit of BEWA.
Scientific authority is valuable when it helps a reader find and assess the evidence. It becomes a liability when the trail disappears behind a score. The useful question is not how many documents agree, but what each document adds to what was already known.