Research commentary · Forecasting and decisions
The customer hidden by a calibrated average.
A probability can be right across a company's whole book of business and wrong for each of the customers who use it. The distinction matters when forecasts decide where scarce protection, stock or staff should go.
A delivery manager is told that tomorrow's shipments have a 20% chance of arriving late. The dashboard is reassuring: across the evaluation period, one shipment in five was late. Yet the manager responsible for one route sees far more delays, while another route rarely has a problem. The company has an accurate overall count and an inadequate guide to allocating its response.
My public preprint, Utility-Weighted Forecasting and Calibration for Risk-Adjusted Decisions under Trading Frictions, treats forecasts as inputs to decisions with costs and constraints. It concerns financial applications. The separate, hypothetical logistics example here develops a related question: which population must a probability describe if it is to support the action someone actually takes? It is an illustration, not an empirical finding from that preprint.
Correct together, misleading apart
Consider 1,000 shipments, all assigned the same predicted probability of delay: 20%. There are 500 shipments on route A and 500 on route B. In an invented completed evaluation set, route A has 175 late deliveries and route B has 25. The observed rates are therefore 35% and 5%.
Together, the routes have 200 late deliveries out of 1,000. The forecast passes an aggregate calibration check exactly: the observed frequency among the shipments given 20% is 20%. Within route A it understates the observed rate by 15 percentage points; within route B it overstates it by the same amount. The two discrepancies cancel in the pooled check.
Nothing has been rounded away. There is only one prediction value in this example, so there is no dispute about how to group the scores into probability bands. The gap arises because conditioning on the forecast alone and conditioning on both forecast and route produce different comparisons.
Follow the probability into a choice
To expose the practical consequence, suppose these route-specific rates are also the true probabilities for the next set of otherwise identical shipments. That is an additional assumption for the calculation, not something the invented historical counts establish. Assume a late delivery causes a loss of £100. A reservation costing £25 per shipment guarantees an alternative delivery if the original is late, avoiding the entire £100 loss. There are no other costs or constraints in this first version.
The reservation is worthwhile on expected cost when the probability of delay exceeds 25%. With a 20% forecast, a buyer following that rule reserves nothing. Expected losses are £17,500 on route A and £2,500 on route B, or £20,000 altogether.
With the route-specific probabilities, the buyer reserves alternatives for route A alone. Its 500 reservations cost £12,500. Route B remains unprotected, with expected losses of £2,500. Total expected cost is £15,000. The difference is £5,000 in this deliberately simplified example.
The calculation does not suggest that sorting customers into more categories always saves money. Its assumptions make the mechanism visible: a pooled probability can cross the wrong action threshold for one group and keep another below a threshold it should not cross. The usefulness of the distinction depends on the available intervention and its cost.
Add the limit that procurement actually faces
Now suppose only 200 alternative-delivery reservations are available. Under the same assumptions, a reservation for route A avoids an expected loss of £35 at a cost of £25, an expected saving of £10. For route B, it avoids £5 at a cost of £25, increasing expected cost by £20.
Allocating all 200 reservations to route A reduces expected total cost from £20,000 to £18,000. A policy based only on the common 20% forecast still buys none. Capacity has reduced the attainable saving from £5,000 to £2,000, but has not removed the importance of identifying where protection is useful.
A real alternative carrier may share the original carrier's weather disruption or warehouse bottleneck. Its protection would then depend on their joint failure behaviour. This example assumes guaranteed protection only to isolate the information lost by pooling. An operating forecast would need evidence about the intervention as well as the original delay.
Make the groups part of the question
The primary research on multicalibration by Ursula Hebert-Johnson, Michael Kim, Omer Reingold and Guy Rothblum gives a formal way to ask for calibrated predictions across a specified family of identifiable subpopulations, including overlapping groups. Their ICML 2018 paper develops algorithms and studies the computational demands of that requirement. It is a distinct research contribution; the arithmetic here does not establish its guarantees.
For the delivery business, route is a starting point rather than a universal solution. The relevant distinction might concern a route during a particular time window, a product needing cold storage or a delivery commitment with an unusually high cost of failure. Some groups overlap. The business must decide which comparisons its service promise requires and whether it has enough observations to estimate them.
That decision belongs before the final performance report. Searching through hundreds of small groups and reporting only the most dramatic discrepancy can mistake noise for a stable pattern. A useful development process separates the data used to identify and adjust a model from the evidence used to evaluate the revised policy.
Record the promise at the level where it is used
I would retain the probability issued, the information available at that time, the relevant route or service class, the action taken, its cost and the eventual outcome. The reporting window and missing outcomes should remain visible. A shipment still in transit has not yet demonstrated an on-time arrival.
The evaluation should show pooled calibration and the specified within-group comparisons, with uncertainty appropriate to the data. It should also show the realised costs of the actual decision policy. A probability estimate, a reservation rule and a capacity allocation solve connected but different parts of the business problem.
This extends the question in When a forecast becomes a decision. There the issue was how an error near a cost threshold changes an action. Here the threshold is held fixed while an apparently correct forecast conceals information about who faces the risk. Keeping that distinction visible helps turn a dashboard's assurance into a decision that can be explained, tested and improved.