Accuracy
We do not need to beat the market. We need to not be lying. This page publishes our model's own calibration results, unedited, whether they pass or fail.
Model 20260928-1724
gates passedEvaluated on 58226 held-out matches across 11 walk-forward splits. Intervals are 80% conformal.
What we said, against what happened
Each dot is a tenth of the probability range: what we said, against how often it happened. On the dashed line we were exactly right; above it we were too cautious, below it too confident. The rule through each dot is one standard error, and the dot's area is how many matches it rests on — the sparse ones at the ends will always scatter.
Furthest out: when we said 92%, it happened 94% of the time, across 18 matches.
All 36 gate values, unedited — 5 not passing
| Gate | Value | Threshold | Result |
|---|---|---|---|
| rps within 5% of marketreported only | — | 1.0500 | not evaluated |
| ece home | 0.0098 | 0.0300 | pass |
| max reliability gap pp homereported only | 2.6136 | 3 | pass |
| max reliability gap pp home measurablereported only | 2.4611 | 3 | pass |
| reliability within sampling envelope home | 2.4611 | 4.4364 | pass |
| ece draw | 0.0108 | 0.0300 | pass |
| max reliability gap pp drawreported only | 30.9590 | 3 | fail |
| max reliability gap pp draw measurablereported only | 3.3422 | 3 | fail |
| reliability within sampling envelope draw | 3.3422 | 4.9933 | pass |
| ece away | 0.0099 | 0.0300 | pass |
| max reliability gap pp awayreported only | 8.1446 | 3 | fail |
| max reliability gap pp away measurablereported only | 4.8845 | 3 | fail |
| reliability within sampling envelope away | 4.8845 | 5.2728 | pass |
| interval coverage within 5pp of 80% | 0.0001 | 0.0500 | pass |
| ece over under 1 5 | 0.0012 | 0.0300 | pass |
| max reliability gap pp over under 1 5reported only | — | 3 | not evaluated |
| max reliability gap pp over under 1 5 measurablereported only | 0.2500 | 3 | pass |
| reliability within sampling envelope over under 1 5 | 0.2500 | 1.3607 | pass |
| brier over under 1 5 | 0.1841 | 0.2500 | pass |
| ece over under 2 5 | 0.0020 | 0.0300 | pass |
| max reliability gap pp over under 2 5reported only | — | 3 | not evaluated |
| max reliability gap pp over under 2 5 measurablereported only | 2.5369 | 3 | pass |
| reliability within sampling envelope over under 2 5 | 2.5369 | 4.4823 | pass |
| brier over under 2 5 | 0.2457 | 0.2500 | pass |
| ece over under 3 5 | 0.0009 | 0.0300 | pass |
| max reliability gap pp over under 3 5reported only | — | 3 | not evaluated |
| max reliability gap pp over under 3 5 measurablereported only | 1.1219 | 3 | pass |
| reliability within sampling envelope over under 3 5 | 1.1219 | 5.8094 | pass |
| brier over under 3 5 | 0.2030 | 0.2500 | pass |
| ece btts | 0.0005 | 0.0300 | pass |
| max reliability gap pp bttsreported only | — | 3 | not evaluated |
| max reliability gap pp btts measurablereported only | 0.1652 | 3 | pass |
| reliability within sampling envelope btts | 0.1652 | 2.0646 | pass |
| brier btts | 0.2476 | 0.2500 | pass |
| sampler converged forecast rhat | 1.0059 | 1.0100 | pass |
| sampler converged max rhatreported only | 1.0212 | 1.0100 | fail |
Why some gates read “fail” while the model publishes
Our original specification asked that predicted and observed rates agree within 3 percentage points in every probability decile. We since measured what that bound demands: simulating a perfectly calibrated model against our own results, it breaches 3 points in 100% of runs, and would need on the order of 150,000 matches — roughly eighty years of five-league football — to clear it reliably. A decile holding a few dozen matches simply cannot resolve 3 points.
So those rows are shown but do not decide. What decides is the same question asked in a form this much data can answer: is the disagreement larger than chance alone would produce at this sample size? That is the sampling-envelope row for each outcome, and it is a real bar — a genuinely miscalibrated model fails it.
We are telling you this rather than quietly deleting the gate we could not meet.
How the forecasts we published have actually done
Everything above comes from backtesting — a model refitted over past seasons and scored on matches no reader ever saw a forecast for. That is the right way to decide whether to publish at all, and the wrong thing to leave as the only answer to “were you right?”. This counts only forecasts we actually published, scored against the market on the same matches.
Did the market come to us?
+0.40pp
Average move in the price of the outcome we backed, between publishing and kick-off, across 19 bets. The market moved toward us on 47% of them.
Positive means the price rose toward our number after we published it, which is what it looks like when a forecast is finding something before the market does. It is not profit and not a win rate: a bet can beat the closing line and lose, and one that drifts the wrong way can still come in. It is evidence about the rule rather than about any single match, and — unlike a settled record — it does not need the season to finish before it says anything.
| Market | Scored | Ours | Market | Difference |
|---|---|---|---|---|
| Match result (RPS) | 479 | 0.2085 | 0.2009 | 0.0076 [0.0016 to 0.0136] |
| Goals over 2.5 (BRIER) | 302 | 0.2346 | 0.2247 | 0.0099 [0.0024 to 0.0174] |
| Goals over 1.5 (BRIER) | 462 | 0.1640 | 0.1648 | -0.0008 [-0.0048 to 0.0032] |
| Goals over 3.5 (BRIER) | 302 | 0.2347 | 0.2241 | 0.0106 [0.0015 to 0.0198] |
| Both teams to score (BRIER) | 462 | 0.2379 | 0.2350 | 0.0029 [-0.0024 to 0.0082] |
Lower is better, and negative means we beat the price. The match result is judged by ranked probability score and the two-way markets by Brier, so a number is comparable with the market beside it but not with another row.
These two halves of the page answer different questions and deliberately treat a thin sample in opposite ways. The gates above decide whether a market is allowed to publish at all, so a market with too little data to check fails them: not being able to verify something is not the same as it being fine. This table only describes how the forecasts we did publish have turned out, so too little data here means too early, not too bad. A market can therefore be cleared to publish and still have nothing to say for itself down here.
A market is only called either way once it has enough settled forecasts for the interval to mean something, and the interval clears zero.
What these gates mean
- The sampler converged on the parameters a forecast is built from. Reported beside it is the same figure across every parameter in the model, which is a maximum over tens of thousands of statistics and exceeds the bound however well the fit went — so it does not decide.
- RPS within 5% of the market's own RPS on the same matches. This reads “not evaluated” and always will: the provider keeps odds for one to two weeks and our corpus ends in May, so the prices for those matches no longer exist anywhere and cannot be bought at any price. It is not a gate awaiting data — it is one that can never be computed on history, and the only route is forward accumulation against prices captured from now on.
- Expected Calibration Error under 3% for each outcome (home, draw, away), and for every other market we publish. Goals over 2.5 and both teams to score come off the same scoreline distribution as the match result, but they are judged on their own numbers: a page vouching for one market while the site shows three is not doing what this one claims. The two-way markets are scored with Brier rather than RPS, which is built for an ordered three-outcome forecast and would give a figure that looks comparable and is not.
- No probability decile disagreeing with observed results by more than chance allows at the sample size we actually have. The original fixed 3-point bound is still reported above, and still fails — see the note on why it no longer decides.
- Our published interval contains the true outcome close to its stated rate.