Check past forecasts against outcomes

Full script: score_history.py.

Read the outcome

Once a forecast resolves, GET /questions/{question_id} returns the outcome in resolution:

{
  "id": 301,
  "title": "What will the settlement price of Brent crude oil be on 31 July 2026?",
  "resolution": {
    "resolved_at": "2026-08-01T09:00:00Z",
    "value": "90.12",
    "notes": "Brent settled at $90.12 on 31 July 2026."
  }
}

resolution is null until then. value is a string, or null if there's no outcome value:

question = client.request("GET", f"/questions/{question_id}").json()
resolution = question["resolution"]
if resolution is None or resolution["value"] is None:
    return None
return float(resolution["value"])

Keep one run per day before the outcome

Fetch the run history and keep each day's last run. Drop runs from 00:00 UTC on the date the forecast is for, when the outcome may already be known:

history = paginate(
    client,
    f"/questions/{question_id}/forecasts",
    "forecasts",
    {"show_forecasts": "true"},
)
last_each_day: dict[str, dict[str, Any]] = {}
for run in history:
    made = datetime.fromisoformat(run["date"])
    if run["type"] == "numerical" and made < before:
        last_each_day.setdefault(run["date"][:10], run)
return sorted(last_each_day.values(), key=lambda run: run["date"])

The history also includes runs that you or your organisation made with your own context, and the stable channel doesn't mark them. If you have made any, read the history from /beta/questions/{question_id}/forecasts and skip runs with user_context: true, as in Forecast with your own context.

Find where the outcome fell

Sample each run's CDF at the outcome:

try:
    response = client.request(
        "GET", f"/forecasts/{run['id']}/cdf", params={"ticks": json.dumps([outcome])}
    )
except httpx.HTTPStatusError as error:
    if error.response.status_code == 404:
        return None
    raise
return next(iter(response.json()["cdf"].values()))

The result is the outcome's position within that day's forecast: 0.5 means the outcome was the median, and 0.95 means the forecast gave a 95% chance of a lower value. A position of 0 or 1 usually means the outcome fell beyond the range the run covers. A run with no stored distribution returns 404, and the script skips that day.

For a quantity that moves in steps, such as a policy rate, the position includes the chance of the exact outcome, so positions lean high.

This makes one request per day of history, so pace the requests when you check many forecasts.

Measure how often the ranges held

The outcome is inside the 50% range when its position is between 0.25 and 0.75, and inside the 80% range when it is between 0.1 and 0.9:

return {
    "inside 50% range": sum(0.25 <= p <= 0.75 for p in positions) / len(positions),
    "inside 80% range": sum(0.10 <= p <= 0.90 for p in positions) / len(positions),
}

Run it with a QUESTION_ID=DATE pair for each resolved forecast, using the date the forecast is for:

python score_history.py 301=2026-07-31 302=2026-06-30

| Forecast | Outcome | Days | Inside 50% range | Inside 80% range |
| --- | ---: | ---: | ---: | ---: |
| 301 | 90.12 | 80 | 41% | 98% |
| 302 | 69.5 | 75 | 5% | 15% |

Read the result

Calibrated forecasts' ranges hold as often as they claim: over many forecasts, the outcome lands inside the 50% range about half the time and inside the 80% range about 80% of the time.

Judge over many forecasts, not one. A forecast's days, and forecasts of the same market, move together, so they are fewer independent checks than they look. When you combine forecasts, counting each day once gives more weight to forecasts that ran longer than counting each forecast once.

For an event forecast, use the Brier score instead: the average of (probability − outcome)², with the outcome 1 if the event happened and 0 if it didn't. Lower is better, and always answering 50% scores 0.25.