The value of a different
security test.

A second security assessment can produce a substantial report while adding little to the faults already known. Its value depends partly on where it looks differently, what it can establish and which decisions its findings change.

My 2007 paper, A Taxonomy of Information Systems Audits, Assessments and Reviews, separates assessment activities that are often sold under similar names. It compares audit-based methods with external tools-based checks and challenges the substitution of a limited scan for a fuller audit programme. That historical comparison motivates a further question: once one assessment has been completed, which additional procedure would be most useful?

Identify the procedure before comparing its score

The paper distinguishes how much information a tester receives from the type of assessment being performed. White-box and black-box describe different initial knowledge conditions; they are not simply synonyms for audit and penetration test. Its discussion also distinguishes running a scanner from conducting a properly designed penetration test. These distinctions matter when interpreting the historical findings. They are not a measurement of every present-day testing service.

An organisation should specify the intended decision before choosing the headline measure. Finding an exposed service, validating an attack path, examining a configuration and exercising an incident-response process can each supply useful evidence. A count of findings collapses those purposes unless the report retains what was examined and what each finding establishes.

NIST SP 800-115 supplies a complementary planning framework. Section 6.3 relates technique selection to assessment objectives, assessor viewpoint, resources, skills and operational risk. Section 6.5 calls for a plan that specifies the assessment’s boundaries. These are grounds for comparing suitable procedures, rather than treating a larger report as an automatic sign of better assurance.

Count the faults added to the existing record

Consider an original hypothetical exercise with twelve deliberately seeded faults in an isolated test system. Give each fault a stable identifier from 1 to 12. All findings below are confirmed true positives, each fault counts once, and the exercise temporarily assigns every fault equal importance. These are stipulated teaching conditions, not results from the 2007 paper or a measured organisation.

Assessment A finds faults 1, 2, 3, 4, 5 and 6. Assessment B also finds six faults: 1, 2, 3, 4, 5 and 7. Assessment C finds four: 6, 8, 9 and 10. The three procedures run on identical starting copies, without seeing one another’s results. Suppose B and C require the same total additional assessment effort, including preparation and confirmation.

Judged separately, B reports more faults than C: six against four. After A, however, B adds only fault 7. The combined record contains seven distinct faults. C adds faults 8, 9 and 10, taking the combined record to nine. The smaller standalone result supplies three new discoveries rather than one.

The arithmetic is simply the size of the union. A and B cover 6 + 6 − 5 = 7 faults because five discoveries overlap. A and C cover 6 + 4 − 1 = 9 because only one overlaps. No assumption of statistical independence is needed. On the exercise’s equal-importance, equal-effort objective, C is the more useful addition to A.

This is a retrospective coverage comparison. Choosing the next real assessment requires evidence that its contribution is likely to persist in the relevant environment. A supplier’s brand or a different tool name does not establish that it will inspect different failure mechanisms.

Keep the missing faults in view

Because the exercise has a known answer key, A with B leaves five seeded faults undiscovered; A with C leaves three. Combining all three finds faults 1 through 10 and still misses 11 and 12. Agreement between assessors has not established completeness.

That last count is available only because the example specifies the full set of seeded faults. In a production assessment, the undiscovered set is generally unknown. Observing an overlap between reports does not, by itself, reveal how many faults nobody found. Nor does success against a seeded challenge prove that the challenge represents all the failures a real service can suffer.

Repeated findings can still be valuable. A second procedure may independently confirm a disputed result, demonstrate a consequence or show that an attempted repair failed. Those are additional evidential contributions, even when the fault identifier does not change. They should be recorded separately from discovering another fault, so the buyer can see what the extra work supplied.

Bring consequence and cost back into the choice

Equal importance made the example easy to inspect. A real decision cannot stop there. One newly established path to a major loss may matter more than several low-consequence configuration defects. Conversely, a long list of unconfirmed warnings can consume the time that would otherwise be spent fixing a serious problem.

A procurement comparison should therefore retain confirmation status, affected assets, plausible consequences, assessment effort and the work required to investigate or repair each finding. Shared causes also matter. If one correction removes several findings, adding their separate estimated losses can overstate the benefit. Where consequences cannot be credibly priced, show the competing considerations rather than manufacture a precise return.

Design a comparison that can be repeated

I would evaluate candidate procedures on controlled copies of the same authorised test environment. The design would state the allowed information, accounts, time budget and fault classes in advance. It would keep a separate answer key, randomise assignments where practical and use independent adjudication to match reports to confirmed faults. No production intrusion is proposed here.

The record should show standalone discoveries, overlap, added discoveries, disputed findings and faults missed by every procedure. Changing the provided documentation or account permissions would be a separate experimental condition. Results should also be reported by fault class, so an apparent gain cannot hide a complete blind spot in another class.

Such a study would test complementary coverage rather than reward duplicated volume. The historical taxonomy supplies a reason to distinguish assessment methods carefully. The operational task is to buy the next piece of evidence that matters, then connect it to a decision and an accountable repair.