During the Second World War, Allied bombers returned with bullet holes concentrated on the wings and fuselage. The obvious conclusion was to armor those areas.

Abraham Wald, a statistician working on the problem, pointed out what the data was actually showing. The holes were in the planes that came back. Damage in those areas was survivable. The areas with no holes on returning aircraft — the engines, the cockpit — were where a hit meant the plane did not return at all.

Armor the places with no holes.

The story is the clearest statement of the error there is. The sample had been filtered by the very thing being studied, and the filter was invisible because the missing cases were at the bottom of the Channel.

The shape

Some process removes cases from view. What remains is not a sample of everything — it is a sample of what got through, and the criterion that let it through is usually the outcome you are trying to explain.

So the conclusion drawn is about the filter rather than about the world. And the more selective the filter, the more confident and the more wrong the conclusion.

Where it operates

Business advice. Interviews, case studies and books come from companies that exist. A strategy followed by a hundred companies of which three survived will look, from the survivors' accounts, like an excellent strategy. The failures are not writing books.

Your own past decisions. You remember the risks that worked out. The ones that did not are unpleasant and get revisited less, so an internal sense of how well a kind of bet pays is built from a filtered set.

Client and candidate pipelines. The clients you have are the ones who said yes to your pricing and process. Their feedback is real and it is not evidence about the ones who walked, who are the ones you would learn most from.

Track records. A fund, an agency or an individual with ten years of results is selected on having lasted ten years. Everyone with the same method and worse luck stopped reporting.

Old buildings and old products. Things that survive are the things built to survive, which is why the past looks better made than it was. Most of it is gone.

What to do

Ask who is missing, first. Before drawing anything from a set of cases, name the filter that produced it and describe the cases it removed. This one question does most of the work.

Go and find the failures deliberately. They are harder to reach — people who failed are less available and less willing — and they carry nearly all the information. A conversation with somebody whose agency closed is worth several with people whose agencies are open.

Prefer base rates to cases. How many attempted this and what share succeeded beats any number of detailed accounts from the ones that did. This is base rate neglect from the other direction: the vivid case is available and the denominator is not.

Record your own decisions when you make them. A written record is unfiltered by outcome, which is the only defense against your memory selecting on it.

The general form is worth carrying: an absence produced by the thing you are studying is data, and it is data you cannot see. Whenever a pattern looks unusually clean, the first question is what had to happen for a case to appear in the set at all.