Early in this project, a new group of features went into the team model and the results looked great: winner accuracy up 1.3 points for one model and 2.6 points for another, measured on a month of late-season games. It's the kind of number that ends up in a changelog with an exclamation mark.
It was noise. Not "mostly noise": when we re-ran the comparison properly, the new features failed every test we threw at them. This post is about how we found that out, and the routine every change to HoopModel now has to pass.
A month of games is a small sample
Our standard test month has about 230 games. That sounds like plenty until you ask how much the accuracy number would move if nothing about the model changed at all. Two things move it:
- Which games happen to be in the window. A month full of lopsided matchups is easy; a month of coin-flip games between playoff-bound teams is hard.
- Randomness in training. Most machine-learning models start from a random seed. Retrain the identical model with a different seed and it makes slightly different picks.
We measured both directly, retraining each model with ten different seeds and resampling the test games a thousand times per seed. On 230 games, a model's accuracy wobbles by roughly ±5 percentage points from those two sources alone. Against that backdrop, "76.1% vs 76.5%" isn't a result. It's two numbers whose error bars almost completely overlap.
Figure 1. Four versions of the team model, scored on the same 230 games
Source: HoopModel experiments. Accuracy and splits: one 230-game test month (March–April 2026), 10 training seeds, 1,000 resamples per seed. Player tuning: three seasons, three seeds each.
Show the data
| Version | Model | Accuracy | 95% range |
|---|---|---|---|
| Baseline | RF | 76.1% | 70.4–81.7% |
| Baseline | XGB | 77.3% | 71.3–83% |
| + Elo | RF | 78% | 72.2–83.5% |
| + Elo | XGB | 76.9% | 70.9–82.6% |
| + Opp-adjusted | RF | 75.4% | 69.6–80.9% |
| + Opp-adjusted | XGB | 76.6% | 70.4–82.6% |
| + Lineups | RF | 76.3% | 70.4–81.7% |
| + Lineups | XGB | 76.5% | 70.4–82.2% |
Compare on the same games, not side by side
The fix is to stop comparing two separate accuracy numbers and compare the two models game by game. Both versions predict the same 230 games, so whatever makes a game easy or hard affects both equally and cancels out. What's left is the handful of games where they disagree:
- For winner picks, we count the disagreements. If the new version is right on 10 of them and the old one on 4, that's suggestive; 8 versus 9 is nothing. (This is McNemar's test.)
- For margins, we take each game's error under both versions and test whether the difference is consistently negative (a paired t-test).
- For ranking quality (how well the win probabilities order the games), we resample the games and look at the distribution of the difference between the two versions.
Paired tests are far more sensitive than eyeballing two percentages. They can confirm a small effect if it shows up on the same games run after run, and they're merciless with effects that only exist in one lucky draw.
What survived
We put three candidate feature groups through this routine:
- Team Elo ratings (a running strength rating that updates after every game) passed. The gain was modest, a small but consistent improvement in how well the probabilities ranked games, and it replicated on two different versions of the dataset. We kept it.
- Five-man lineup statistics were the source of that exciting "+2.6 points." They failed every paired test, on every model and every metric. We dropped them.
- Opponent-adjusted rolling stats were worse than useless: they made margin predictions measurably less accurate, because they mostly repeated information the model already had in a noisier form. Dropped.
One keeper out of three is a fairly typical hit rate. Most plausible-sounding basketball features are already captured, in some form, by the ones a model has.
Figure 2. Games where the old and new versions picked different winners
Source: HoopModel experiments. Accuracy and splits: one 230-game test month (March–April 2026), 10 training seeds, 1,000 resamples per seed. Player tuning: three seasons, three seeds each.
Show the data
| Change | New version right | Old version right | p (McNemar) |
|---|---|---|---|
| + Elo (RF) | 8 | 3 | 0.23 |
| + Elo (XGB) | 5 | 8 | 0.58 |
| + Opp-adjusted (RF) | 1 | 3 | 0.62 |
| + Opp-adjusted (XGB) | 5 | 8 | 0.58 |
| + Lineups (RF) | 1 | 2 | 1 |
| + Lineups (XGB) | 2 | 4 | 0.68 |
| Tuned settings (XGB) | 7 | 7 | 0.79 |
The same trap, in tuning
Months later we ran our first systematic search over the team model's settings: how deep its trees go, how fast it learns, how hard it's regularized. The search scored each candidate on four earlier test months and found a configuration that clearly beat the original on those months.
Then we tested it on a fifth month the search had never seen. Scored one seed at a time, it looked better there too: 79.9% against 78.2%. But once we averaged across seeds and compared game by game, the two configurations disagreed on 14 games and split them exactly 7–7 (the bottom row of Figure 2). The margin errors were indistinguishable. The search had found settings that fit those four months, not settings that predict basketball better. We kept the original.
The interesting twist came when we ran the same procedure on the player projection model. There the tuned settings did hold up on held-out data: 9 of the 11 projected stats cleared the bar and turnovers improved slightly, with better fantasy-point accuracy in every test period. The one exception, blocks, got slightly worse, so that stat kept its original settings while the rest switched. Same method, opposite verdict. That's the point of having a method.
Figure 3. Player model: change in average error after tuning, by stat
Source: HoopModel experiments. Accuracy and splits: one 230-game test month (March–April 2026), 10 training seeds, 1,000 resamples per seed. Player tuning: three seasons, three seeds each.
Show the data
| Stat | Avg change in error | Best run | Worst run | Cleared the bar |
|---|---|---|---|---|
| MIN | -0.40% | -0.70% | -0.08% | Yes |
| PTS | -0.66% | -1.29% | -0.42% | Yes |
| REB | -0.63% | -1.00% | -0.33% | Yes |
| AST | -0.70% | -1.11% | -0.16% | Yes |
| STL | -0.29% | -0.50% | -0.15% | Yes |
| BLK | 0.25% | -0.41% | 0.72% | No |
| TOV | -0.27% | -0.52% | -0.03% | No |
| 3PM | -0.24% | -0.34% | -0.16% | Yes |
| 3PA | -0.39% | -0.63% | -0.20% | Yes |
| FTM | -0.39% | -0.58% | -0.04% | Yes |
| FTA | -0.40% | -0.66% | -0.02% | Yes |
The rules we work by now
- Decide the pass/fail bar before running the experiment, and write it down.
- Train with multiple seeds; never trust a single run.
- Compare versions with paired tests on the same games.
- Keep a stretch of data the selection process never touches, and let it decide.
- Treat a sub-one-point accuracy gain as noise until proven otherwise.
None of this makes the models dramatically better on its own. What it does is stop them from getting quietly worse: every feature that adds noise makes the next real improvement harder to detect. It's also why the track record page shows long windows by default. A week of results tells you very little about a model, in either direction.