The Research Mindset: Judging Decisions, Not Outcomes

Ask anyone how their research went last night and they will answer with a result. It is the most natural thing in the world and it is the single habit that does the most damage to a research process, because in this domain the result and the decision are only loosely related, and the looseness is not a flaw to be engineered away. It is the structure of the thing.

Everything in this library has been building toward this, and it is worth saying directly at the end. If you keep only one idea from all of it, keep the one where you grade the decision on what you knew when you made it, and let the result be a separate fact you record without letting it grade anything.

Outcome bias, named precisely

Outcome bias is the documented tendency to evaluate the quality of a decision by how it turned out rather than by the quality of the reasoning available when it was made. The classic demonstration gives people the same described decision with different endings and watches the ratings move. The reasoning is identical. Only the ending changed, and the judgment follows the ending.

It travels with a companion. Hindsight bias is the sense, once you know what happened, that it was foreseeable all along. Together they produce a very specific illusion: after the fact, a losing decision looks not only unlucky but obviously wrong, and the reasons it was obviously wrong assemble themselves effortlessly. The pitcher's velocity was down in his last start. The number had been moving all day. You knew. You did not know, and if you go back to your own notes from before the game you will usually find that you did not write any of it.

In most of life these biases are mildly costly. In an activity where a genuinely good estimate loses four nights in ten, they are corrosive, because they attach the strongest emotional signal to the least informative variable. You are being taught something after every event. What you are being taught is mostly the outcome, and the outcome is mostly noise.

Why this is practical rather than philosophical

The case against outcome thinking is not that it is unfair to you. It is that it makes you worse, through two concrete mechanisms.

The first is that it teaches the wrong lessons in the wrong direction. When a careful decision loses, outcome thinking asks what you should have done differently, and the honest answer is often nothing. But the question demands an answer, so you invent one: a factor you should have weighted more, a hunch you should have followed. Now your process has a new rule derived from a single noisy event. Do that a dozen times over a season and you have a process built entirely out of overreactions to individual nights, which is indistinguishable from having no process.

The second is more dangerous because it feels good. A run of luck will validate whatever you happened to be doing while it ran. If you spent a week ignoring your own checks, skipping the opportunity question, acting late in tired sessions, and the week went well, outcome thinking will file all of that as working. Nothing in the results will contradict it, because a short run of results cannot distinguish a good process from a bad one. The feedback that would have corrected you is simply absent, and confidence grows anyway.

What to judge instead

Refusing to grade on outcomes only helps if something else does the grading. The replacement has to be specific enough to fail, or it becomes a way of never being wrong about anything. These are the questions that can be answered honestly the morning after, without knowing the result.

1. Was the offer real and sellable as researched. If the conclusion was that an over was bad, and the offer was sold over only, the honest output was a pass rather than an under. A conclusion that could not be acted on is not a decision you get credit for.
2. Was opportunity established before anything else. Minutes, plate appearances, lineup status, scheduled rounds or maps. Rate work on an unknown denominator is not analysis, whatever it produced.
3. Was the estimate formed before you looked at the number. An estimate that arrived after the line is contaminated by it, and you will not be able to feel the contamination.
4. Was the uncertainty sized honestly. Did the range you carried reflect how thin the evidence actually was, or did a small sample get treated like a settled fact.
5. Was the decision written down, with its reasoning, before the event resolved. If it was not, you have no way to check any of the above later, because memory will supply a tidier version.

Notice that every item is checkable without the result, and that a decision can fail several of them and still win. That is the point of the list. It gives you something to be wrong about that is actually within your control, which is the only kind of wrong worth studying.

Calibration is the only honest scoreboard

If results cannot grade an individual decision, something has to grade the process eventually, or the whole idea collapses into unfalsifiable comfort. Results do grade a process. They just need far more of them than instinct suggests, and they need to be read as calibration rather than as a record.

Calibration asks a different question than winning does. It asks whether the things you called seven in ten happened about seven times in ten, and whether the things you called six in ten happened about six times in ten. That is a claim about honesty rather than about profit, and it is the claim a research process is actually making. Reading a calibration curve walks through the mechanics, including how thin the early sections of any such curve are and how easily a hundred results can flatter or condemn a model that deserves neither.

The important structural fact is that calibration is a volume measure and cannot be rushed. Any short window will look like something. A stretch of losses on genuinely good estimates and a stretch of wins on bad ones are both perfectly ordinary at small counts, which is precisely why the story your last two weeks tells you is not evidence of anything. Waiting for a scoreboard that means something is uncomfortable and there is no shortcut, and anyone offering you one is describing a record rather than a method.

This is also why volume is the wrong ambition. More decisions do not make a scoreboard readable faster if the additional decisions are worse, because the noise grows with them. A smaller number of decisions you can defend produces a cleaner signal than a larger number you cannot, and the checklist exists to keep that number where you can defend it.

The freedom hidden in the standard

This all sounds like an austerity measure and it is closer to the opposite. Detaching your self assessment from results removes the two worst experiences in research, and it removes them in both directions.

You can lose an entire night and have done everything right. That is not a rationalization, it is arithmetic: close probabilities produce losing nights routinely, and a night where four careful estimates all came in under is completely unremarkable. There is nothing to fix, no adjustment to make, no lesson buried in it. You are allowed to close the laptop and be finished, and that permission is worth more over a season than any single insight, because it is the thing that prevents the chase.

And you can win a night and owe yourself a hard look. That one is harder to actually do, because nothing in the experience prompts it. A won night that skipped the opportunity check, or acted on an offer you could not explain, is a warning arriving in the costume of a reward. The research log is the mechanism that makes this survivable, because a log written before the result is the only artifact that can tell you what you actually knew rather than what you now feel you knew.

Hold any research product to the same standard

It would be inconsistent to argue all of this and then ask to be judged by a highlight reel. So the standard we have described applies to us, and it should be applied to anyone selling you analysis, including us.

That is why we publish our misses and why the graded record sits in the Model Room rather than in a marketing claim. What belongs there is calibration across enough decided events to mean something, a clear statement of which sports have enough graded history to say anything at all, and the places the model has been wrong in a direction we have not yet fixed. A record of selected wins is not evidence. A curve that shows where the estimates were overconfident is.

The questions worth asking of any tool follow the same shape as the questions worth asking of a decision. Does it show the losses. Does it state its sample size. Does it distinguish a model estimate from a market reference, and say which one it has. Does it decline to answer when the answer is not knowable, or does it produce a confident recommendation for every offer on a board. Does it promise anything about outcomes, because nothing honest can.

That is the whole thesis of this library, and it is a modest one. Research narrows uncertainty and never removes it. The decisions are yours, the results belong partly to chance, and the only durable thing you can build is a process you would still defend after seeing how a night turned out. Keep it in proportion, keep it inside limits you set while calm, and if it ever stops being something you enjoy, that matters more than any of the analysis. Support is available at 1 800 MY RESET and through the National Council on Problem Gambling, and our own commitments sit on the responsible gaming page.