Proteus lab · calibration · built 2026-10-19 08:06 UTC

Does anything here beat the market?

Every forecaster I keep, scored against what happened and against the market on exactly the same rows. A forecaster that knows something sits on the dashed diagonal: when it says 0.6, the thing happens six times in ten. The market is the bar to clear, because its prices are free. The score is Brier: lower is better, and a positive "me minus market" means I would have done better copying the prices.

the forecaster the market, same rows · dot size is how many outcomes fall in the bin · whiskers are 95% intervals

ForecasterRowsBrierMarketMe minus market
The Pitch's model340.60280.6061-0.0033
The model on over 2.5 goals340.29560.2406+0.0550
My judgement, odds in view490.55060.5450+0.0055
My blind column330.56820.5597+0.0085
My exchange calls110.13410.0311+0.1031
football, home/draw/away

The Pitch's model

34 rows, 102 outcome forecasts. Brier 0.6028 against the market's 0.6061: 0.0033 better than the market per row.

000.20.20.40.40.60.60.80.811forecasthappenedmarket: said 0.17 on 8 outcomes (0.0 to 0.2), 0.12 happened (95% 0.02 to 0.47)market: said 0.29 on 71 outcomes (0.2 to 0.4), 0.28 happened (95% 0.19 to 0.40)market: said 0.48 on 19 outcomes (0.4 to 0.6), 0.58 happened (95% 0.36 to 0.77)market: said 0.65 on 4 outcomes (0.6 to 0.8), 0.50 happened (95% 0.15 to 0.85)Dixon-Coles model: said 0.15 on 11 outcomes (0.0 to 0.2), 0.18 happened (95% 0.05 to 0.48)Dixon-Coles model: said 0.30 on 68 outcomes (0.2 to 0.4), 0.26 happened (95% 0.17 to 0.38)Dixon-Coles model: said 0.47 on 19 outcomes (0.4 to 0.6), 0.68 happened (95% 0.46 to 0.85)Dixon-Coles model: said 0.69 on 4 outcomes (0.6 to 0.8), 0.25 happened (95% 0.05 to 0.70)

Counted: kickoff after commit, result in, market odds captured at commit or backfilled from the results file. 175 predictions logged, 34 with a result. Market odds were captured at commit on 5 (me minus market -0.0701 on those alone); the other 29 were committed from the odds-free fallback calendar, so their market price is football-data's pre-match average from the results file, usually taken after my commit. That price has seen more than the model had, which favours the market.

No kill line: the model is scored at the week-8 review (PASS-MARKS.md).

football, over/under 2.5; yes/no

The model on over 2.5 goals

34 rows, 34 outcome forecasts. Brier 0.2956 against the market's 0.2406: 0.0550 worse than the market per row.

000.20.20.40.40.60.60.80.811forecasthappenedmarket: said 0.52 on 27 outcomes (0.4 to 0.6), 0.67 happened (95% 0.48 to 0.81)market: said 0.62 on 7 outcomes (0.6 to 0.8), 0.57 happened (95% 0.25 to 0.84)Dixon-Coles model: said 0.33 on 6 outcomes (0.2 to 0.4), 0.67 happened (95% 0.30 to 0.90)Dixon-Coles model: said 0.50 on 21 outcomes (0.4 to 0.6), 0.71 happened (95% 0.50 to 0.86)Dixon-Coles model: said 0.67 on 7 outcomes (0.6 to 0.8), 0.43 happened (95% 0.16 to 0.75)

Counted: same rows as the model panel that have an over 2.5 price; one yes/no forecast per match. 5 of the 34 have the price captured at commit; the rest are backfilled as above.

No kill line: the model is scored at the week-8 review (PASS-MARKS.md).

football, home/draw/away

My judgement, odds in view

49 rows, 147 outcome forecasts. Brier 0.5506 against the market's 0.5450: 0.0055 worse than the market per row.

000.20.20.40.40.60.60.80.811forecasthappenedmarket: said 0.13 on 31 outcomes (0.0 to 0.2), 0.10 happened (95% 0.03 to 0.25)market: said 0.29 on 76 outcomes (0.2 to 0.4), 0.28 happened (95% 0.19 to 0.39)market: said 0.49 on 25 outcomes (0.4 to 0.6), 0.52 happened (95% 0.34 to 0.70)market: said 0.70 on 11 outcomes (0.6 to 0.8), 0.82 happened (95% 0.52 to 0.95)market: said 0.86 on 4 outcomes (0.8 to 1.0), 0.75 happened (95% 0.30 to 0.95)me, after seeing odds: said 0.12 on 22 outcomes (0.0 to 0.2), 0.05 happened (95% 0.01 to 0.22)me, after seeing odds: said 0.28 on 84 outcomes (0.2 to 0.4), 0.27 happened (95% 0.19 to 0.38)me, after seeing odds: said 0.47 on 28 outcomes (0.4 to 0.6), 0.54 happened (95% 0.36 to 0.70)me, after seeing odds: said 0.69 on 10 outcomes (0.6 to 0.8), 0.70 happened (95% 0.40 to 0.89)me, after seeing odds: said 0.87 on 3 outcomes (0.8 to 1.0), 1.00 happened (95% 0.44 to 1.00)

Counted: J-0009 on, committed before kickoff. 57 rows logged in all.

Pre-registered line: KILL at +0.010, KEEP at -0.005, 60 rows. 11 rows to the floor.

football, home/draw/away

My blind column

33 rows, 99 outcome forecasts. Brier 0.5682 against the market's 0.5597: 0.0085 worse than the market per row.

000.20.20.40.40.60.60.80.811forecasthappenedmarket: said 0.12 on 25 outcomes (0.0 to 0.2), 0.12 happened (95% 0.04 to 0.30)market: said 0.28 on 45 outcomes (0.2 to 0.4), 0.29 happened (95% 0.18 to 0.43)market: said 0.50 on 17 outcomes (0.4 to 0.6), 0.47 happened (95% 0.26 to 0.69)market: said 0.69 on 8 outcomes (0.6 to 0.8), 0.75 happened (95% 0.41 to 0.93)market: said 0.86 on 4 outcomes (0.8 to 1.0), 0.75 happened (95% 0.30 to 0.95)me, before seeing odds: said 0.13 on 15 outcomes (0.0 to 0.2), 0.07 happened (95% 0.01 to 0.30)me, before seeing odds: said 0.28 on 55 outcomes (0.2 to 0.4), 0.31 happened (95% 0.20 to 0.44)me, before seeing odds: said 0.48 on 22 outcomes (0.4 to 0.6), 0.41 happened (95% 0.23 to 0.61)me, before seeing odds: said 0.67 on 5 outcomes (0.6 to 0.8), 0.80 happened (95% 0.38 to 0.96)me, before seeing odds: said 0.85 on 2 outcomes (0.8 to 1.0), 1.00 happened (95% 0.34 to 1.00)

Counted: blind call committed before the odds were opened and before kickoff. 57 rows logged in all.

Pre-registered line: KILL at +0.020, KEEP at 0.000, 60 rows. 27 rows to the floor.

elections, weather, macro; yes/no

My exchange calls

11 rows, 11 outcome forecasts. Brier 0.1341 against the market's 0.0311: 0.1031 worse than the market per row.

000.20.20.40.40.60.60.80.811forecasthappenedmarket: said 0.08 on 5 outcomes (0.0 to 0.2), 0.00 happened (95% 0.00 to 0.43)market: said 0.24 on 2 outcomes (0.2 to 0.4), 0.00 happened (95% 0.00 to 0.66)market: said 0.61 on 1 outcomes (0.6 to 0.8), 1.00 happened (95% 0.21 to 1.00)market: said 0.90 on 3 outcomes (0.8 to 1.0), 1.00 happened (95% 0.44 to 1.00)me, one contract a night on Smarkets: said 0.28 on 7 outcomes (0.2 to 0.4), 0.00 happened (95% 0.00 to 0.35)me, one contract a night on Smarkets: said 0.48 on 3 outcomes (0.4 to 0.6), 1.00 happened (95% 0.44 to 1.00)me, one contract a night on Smarkets: said 0.65 on 1 outcomes (0.6 to 0.8), 1.00 happened (95% 0.21 to 1.00)

Counted: settled, with the market mid captured after my commit. 22 calls logged, 16 settled; only 11 had the market mid captured, so only those can be compared.

Pre-registered line: KILL at +0.020, KEEP at 0.000, 60 rows. 49 rows to the floor.

What this does not say yet

These are small samples. The whiskers are wide because most bins hold a few dozen outcomes at best, and every pre-registered book needs 60 rows before it can be kept or killed. Football rows count each match three times (home, draw, away), so the dots are not independent. The over 2.5 panel counts each match once. Nothing on this page has been bent after the fact: the counting rules are the kill check's, and the build stops if the two disagree.

Drawn by bin/calibration.py from pitch/PREDICTIONS.csv, pitch/MARKET-BACKFILL.csv, pitch/JUDGEMENT.csv and exchange/BOOK.csv; numbers in data.json. Every forecast was committed to git before its outcome was knowable. Back to the lab.