What We Changed After Being Wrong

Every projection system carries assumptions its authors believed on the day they wrote them. Some of those assumptions are wrong. That is not an indictment of anyone's competence; it is the ordinary condition of modeling a world you cannot fully observe. The interesting question is what happens after you find out, and whether anyone outside the building ever hears about it.

We argued in why we publish our misses that a record only means anything if it includes the parts you would rather delete. This piece is the concrete version of that argument. Three real defects in our own engines, what each one actually was, how it got caught, what changed, and why we still refuse to call two of the three fixes proven.

What a real model error looks like

First, a correction to the usual mental image. A model error is almost never a bad night. Any honest projection loses constantly, because most props sit close to a coin flip and a good estimate still misses a great deal of the time. A losing week tells you approximately nothing.

A real defect looks different. It is a bias that persists across hundreds of outcomes and points the same direction in markets that should be unrelated. If a system is quietly low on points, and also low on rebounds, and also low on assists, the problem is rarely in the points logic, the rebound logic, and the assist logic all at once. It is upstream of all three, in something they share. That signature, one direction, everywhere, is the thing to hunt for, and it is only visible in aggregate. It is exactly what a calibration curve is built to expose.

The phantom bench

Our WNBA engine simulates a game possession by possession, and to do that it has to decide who is on the floor. The earlier version made an assumption that sounds harmless: every player who is not injured is available to play in every simulated game. Availability was treated as a health question.

The engine also conserves total minutes exactly, because a basketball game contains a fixed number of them. Those two rules interact in a way nobody intended. If a fixed pool of minutes is shared among more players than actually take the floor on a given night, everyone's share shrinks. The players who lost the most were the ones who play the most, because they had the most to lose. Our simulated stars were spending less time on the court than real stars do.

The fix was to stop assuming and start measuring: how often does a player at each spot in the rotation actually appear, as decided by a coach rather than by an injury report? Deep bench players turn out to appear far less often than always, and once that is measured, the minutes they were absorbing flow back to the players who really play. The uniform low bias across markets went away with it.

A sign pointing the wrong way

The second one is less subtle and more embarrassing, which is why it is worth telling. Our MMA engine prices volume props off fight duration, for the reason laid out in how fight duration shapes UFC props: a striker's pace is worth nothing if the fight ends in the first round. So the engine needs a view on how finish risk is distributed across rounds.

The original view was authored rather than measured, and it assumed that risk escalates as a fight goes on: tired fighters get finished. It is an intuitive story. When we finally measured it against real fight histories, the relationship ran the other way. The opening round carried the highest finish hazard, and the risk declined from there. The assumption was not merely imprecise. Its sign was backwards.

When the right answer is to stop publishing

The third case is different because the response mattered more than the diagnosis. In one MLB market family, our stated probabilities were running far above the rate at which those outcomes were actually happening. That is the worst kind of defect, because a confident number that is wrong is more damaging than no number at all.

Two things were broken. The distribution the probabilities came from was too narrow, so it treated uncertain outcomes as more predictable than they were. And the scoring profiles behind those composite stats did not match the scales the platforms actually use, which is precisely the failure mode described in fantasy score props.

The response was to block the market. The engine kept computing, and the board stopped publishing recommendations there, with the reason stated in product rather than hidden. It stayed blocked until the probabilities came from properly simulated distributions and the scoring charts had been verified against what each platform publishes. Refusing to show a number is a feature, not an outage. Anyone can produce a projection for anything; the discipline is declining to.

Why we still call two of these unconfirmed

Here is the part that gets left out of most write ups like this one. Both engine revisions improved things in replay, and we still do not describe them as proven.

The reasons are ordinary and they matter. Each revision changed more than one mechanism at once, so the replay cannot attribute the improvement to any single change. Some parameters were estimated using data from the same era they were then evaluated on, which is the classic route to overfitting: a model can look better on the window it was tuned to and carry none of that forward. And the samples involved were small, particularly in MMA, where a handful of events supplies far fewer independent observations than the raw fight count suggests.

• Bundled changes mean you cannot say which change helped.
• Parameters fitted and judged on the same window can flatter themselves.
• A small number of events is a small sample even when the row count looks large.
• Improvement in a replay is a hypothesis about the future, not a measurement of it.

So the honest status is that the defects were real and are fixed, and the size of the improvement is not yet established. Confirming it requires accumulating results going forward, on slates the parameters were never allowed to see. That is slower and less satisfying than declaring victory, and it is the only version that would survive someone checking.

What this should mean to you

Do not take any of this as a reason to trust us more. Take it as a set of questions worth asking of any research product, including this one. Does it keep the inputs behind old projections, or only the outputs? Can it describe a specific thing it got wrong, with the mechanism rather than a vague admission of imperfection? Does it distinguish between a fix that is deployed and a fix that is proven? Has it ever stopped publishing a market on purpose?

A product that cannot answer those has not necessarily been dishonest with you. It may simply have never looked. But you should weight its confidence accordingly, and you should apply the same standard to your own process, which is most of what keeping a research log is for.

None of this changes what a projection is. It remains an estimate with uncertainty around it, published by people who expect to be wrong again in ways they cannot currently see. What we can promise is the part that is actually within our control: when we find the next one, you will read about it here. Our graded records and methodology for every live sport stay public in the Model Room.