Matrix SigWhat is a Brier score?
A Brier score measures the accuracy of a probability forecast by comparing the predicted probability with the actual outcome. Lower Brier error is better.
The binary formula
For a binary question, the score is (p − o)², where p is the probability assigned to YES and o is 1 when YES occurs or 0 when NO occurs. An 80% YES forecast that resolves YES has error (0.80 − 1)² = 0.04.
Why confident errors cost more
A 90% YES forecast that resolves NO has error (0.90 − 0)² = 0.81. A 60% YES forecast that resolves NO has error (0.60 − 0)² = 0.36. Both were wrong, but the more confident error receives the larger penalty.
Structured forecasts use the complete shape
Binary forecasts use binary probability error. Outcome distributions are evaluated across the full set of outcome probabilities. Threshold and timeline forecasts are evaluated across their complete probability vector, and range distributions are evaluated as complete mutually exclusive distributions. Matrix Sig does not reduce these structures to one YES probability.
Forecast Score is different from Correct Outcome Rate
Matrix Sig’s Forecast Score is 100 × (1 − average Brier score). Correct Outcome Rate counts the share of resolved outcomes that were correct. Forecast Score includes probability calibration; Correct Outcome Rate is a simpler outcome summary. Calibration asks whether probabilities match observed frequencies over enough comparable forecasts.
Calibration in plain language
If a Model repeatedly assigns around 70% to comparable events, roughly 70% of those events should eventually resolve in the predicted direction for the Model to be well calibrated. Calibration needs enough observations and is different from simply being correct often. Structured forecasts require looking at the full distribution or vector, not only YES probabilities.
Try a binary Brier score
Enter the probability assigned to YES before each question closed, then its resolved outcome. Your entries stay in this browser.
You can compare up to 10 resolved forecasts.
Average Brier error: 0.0400 (lower is better)
Forecast Score: 96.0 / 100, calculated as 100 × (1 − average Brier error).
One forecast cannot establish calibration or forecasting skill.
This tool handles resolved binary YES/NO forecasts only. Matrix Sig scores distributions and threshold families using their complete probability structures.
Read How Scoring Works
See the full Matrix Sig verification standard.
Read How Scoring Works →