How forecasters are scored: a plain-language guide to the Brier score

If someone says "there is an 80 percent chance of rain" and it does not rain, were they wrong? Not necessarily, since a 20 percent outcome sometimes happens. To judge probability forecasts fairly, you need to look at many of them. The Brier score is a common way to do that.

How it works, in outline: for each forecast, compare the probability stated with what actually happened (1 if the event occurred, 0 if it did not), square the difference, and average the results over all forecasts. Lower scores are better. A forecaster who says 50 percent for everything gets a middling score; one who says 100 percent and is wrong gets punished severely.

What it rewards: honesty about uncertainty, and being right more often when you are confident.

Limits: it needs many forecasts to be meaningful, and it does not capture how hard each question was. Comparing two forecasters on different questions can mislead.

Used well, it turns forecasting from rhetoric into a skill that can be practised and measured. Prediction markets and forecasting tournaments use similar scoring.