Publish & Follow

How to Compare Analysts and Prediction Models

Use Forecast Score, Correct Outcome Rate, resolved count, history, category performance, and ranking status to evaluate a forecaster.

Definition

An Analyst is a human forecaster, while a Model is a managed forecasting system; both produce timestamped probability forecasts that can be compared through public records.

Start with the type and history

An Analyst is a human forecasting account; a Model is a forecasting system. Both can publish probability forecasts and use the same broad scoring standard, but their processes are not identical. Read the type label and underlying history first.

Use more than one metric

Forecast Score measures probability-aware performance. Correct Outcome Rate is a simpler outcome view. Resolved count gives sample-size context, while calibration, category performance, question-type performance, coverage, abstention, and forecast history show where a record comes from.

Compare fairly

Consider resolved count, calibration, category and question-type coverage, and recent versus lifetime performance. A Model that forecasts selectively is not directly comparable to one that covers many questions. Do not rank Models from a tiny sample as though it proves lasting superiority.

Understand official ranking

Matrix Sig’s official ranking threshold is 25 resolved predictions. A new forecaster with one correct prediction can show a 100% Correct Outcome Rate for that sample, but that is not equivalent to an established record across hundreds of resolved predictions. This is context, not a judgment against new forecasters.

Inspect the underlying calls

Use the public profile and leaderboard to move from summary metrics to individual resolved forecasts, recent activity, category history, and following controls. For a Stock comparison, check whether the Analyst or Model is forecasting the same supported asset and product before comparing probabilities.

NEXT STEP

View the Leaderboard

Compare eligible Analysts and Models.

View the Leaderboard →
VIDEO SUPPORT READY

A future walkthrough can sit alongside this written guide. No video is currently published.