Every prediction on tonight's page comes with a win probability: "home team 70%." What should that number mean? Not that the home team will win, and not that the model is "70% sure" in some vague sense. It means that across all the games where the model says 70%, the home team should win about 70% of them. A model whose numbers behave that way is called calibrated.
Calibration is a separate skill from picking winners, and we learned that the hard way. This is the story of a fix that made our probabilities look better and our picks worse, and what we do now instead.
A random forest that seemed too timid
On one 230-game test month, our random forest's probabilities were off in a consistent direction. Games it gave about 75% were won by the home team 87% of the time. Games it gave about 55% were won 70% of the time. Its picks were better than its numbers admitted.
Figure 1. Random forest, one test month: how often the home team won, by the win probability the model gave
Source: HoopModel experiment: 230 games, March–April 2026, 10 seeds.
Show the data
| Model's average probability | Home team actually won | Games |
|---|---|---|
| 26.4% | 4.5% | 22 |
| 34.4% | 9.5% | 21 |
| 45.7% | 43.2% | 44 |
| 55.3% | 70% | 40 |
| 64.7% | 78.7% | 47 |
| 74.9% | 87% | 46 |
| 80.8% | 90% | 10 |
That is a known trait of random forests: the final probability is an average over many trees, and averaging pulls numbers toward the middle. The standard remedy is a recalibration layer: fit a mapping from the model's raw probabilities to how often teams actually won, on games the model didn't train on, and apply it to every new prediction. On that test month, recalibration roughly halved the calibration error (from 0.131 to 0.074). We added it.
How the fix broke the picks
A recalibration layer is fitted on one slice of games. If the games it later sees look different, the mapping can be badly wrong, and ours was. From January 2026 the forest picked the home team in almost every game, and in every game in February and March. The betting market, over the same stretch, favored the home team in about six games out of ten.
Figure 2. 2025-26 regular season: share of games where each model picked the home team
Source: HoopModel track record (public predictions as graded each night).
Show the data
| Month | Games | RF picked home | NN picked home | XGB picked home | Market favored home | Home team won | RF accuracy |
|---|---|---|---|---|---|---|---|
| Oct | 80 | 77.5% | 73.8% | 63.7% | – | 58.8% | 63.7% |
| Nov | 208 | 65.9% | 59.6% | 57.7% | – | 55.8% | 71.6% |
| Dec | 197 | 62.1% | 46.7% | 50.8% | 48.6% | 53.3% | 61.5% |
| Jan | 233 | 92.2% | 66.2% | 54.1% | 59.2% | 55.8% | 56.7% |
| Feb | 166 | 100% | 71.1% | 54.2% | 57.2% | 47% | 47% |
| Mar | 239 | 100% | 72% | 59% | 61.1% | 60.3% | 60.3% |
| Apr | 96 | 93.8% | 55.2% | 69% | 60.4% | 59.4% | 65.6% |
The forest hadn't gotten worse at telling strong teams from weak ones. The layer on top had pushed nearly every probability above 50%, which turned correct road-team picks into wrong home-team picks. In February the forest's accuracy was 47%: exactly the share of games the home team won. It had stopped predicting and started saying "home" every night.
What made this easy to miss is that the accuracy number didn't look broken. A model that always picks the home team is still right whenever the home team wins, which is more than half the time in most months. Over the full regular season the forest picked 60.6% of winners, behind XGBoost's 66.2%: the kind of gap that looks like an off year, not a broken model. The tell was in the distribution of picks, not in the hit rate.
What a longer test showed
One test month started all this, so we tested calibration properly: 16 monthly windows from 2024-03 to 2026-04, comparing raw and recalibrated probabilities on the same games.
Figure 3. What recalibration changed, across 16 monthly test windows
Change in winner accuracy (points)
Change in calibration error (× 100; left = better)
Source: HoopModel calibration backtest: 16 monthly windows, three seeds each.
Show the data
| Model | Δ accuracy (avg) | Δ accuracy (range) | Δ calibration error (avg) | Δ calibration error (range) | Δ home-pick share (range) |
|---|---|---|---|---|---|
| Random forest | -0.47 | -5.2 – 2.7 | 0.38 | -4.2 – 6.2 | -17.5 – 3.6 |
| XGBoost | -0.50 | -4.2 – 2.8 | -3.80 | -11.0 – 0.0 | -27.0 – 17.4 |
- Accuracy didn't improve for either model: about half a point lower on average, and anywhere from 5 points worse to 3 points better in a single window.
- The random forest didn't need it. Averaged over all the windows, its raw probabilities were about as well calibrated as the recalibrated ones. The timid month in Figure 1 was the exception, not the rule.
- XGBoost did need it. Recalibration lowered XGBoost's calibration error in 15 of the 16 windows and never made it worse.
What we do now
- Picks always come from the raw probabilities. Recalibration can only change the number we display, never which team we pick.
- The random forest shows its raw probabilities; XGBoost shows recalibrated ones.
- Every retrain still measures how far a newly fitted calibration layer would move the share of home picks, and an extreme shift triggers an alert so someone looks at it.
The broader lesson is to watch what a model does, not just how often it's right. The share of home picks is now one of the first things we check after a retrain. This season it's worth watching: over the first 70 games, the forest has picked the home team in 84.3% of them, against 74.3% for the market. That leans home more than the market does, but it's nothing like last February, and it comes from the raw model.
How to read the win probability on our site
Treat it as a frequency, not a promise. A 70% pick should lose about three times in ten, and when it does, that's the number working as intended. If you want to know whether the numbers are honest, the model comparison page checks exactly this: how often the home team won at each level of predicted probability.