How to Compare Two Research Tools Honestly

Ask which of two projection tools is better and you will usually get an answer within a minute. The speed is the problem. A real answer requires two ledgers built under matching conditions, and matching conditions are exactly what competing products almost never have. What gets compared instead is which landing page sounds more certain, which is a measurement of copywriting.

The backtesting article covers how to audit one record on its own terms. This one is about the harder problem: putting two records next to each other without producing a conclusion that is mostly an artifact of how they were assembled.

The comparability problem

Two tools can both be telling the truth and still produce records that cannot be subtracted from each other. One grades every projection it publishes; the other grades only the ones it labeled as recommendations. One counts a push as a loss; the other removes it. One covers seven sports; the other covers two and happens to cover the two that ran well. One evaluates at the line a platform posted at the moment the projection was written; the other evaluates at whatever line existed at settlement, after the market moved. Coverage is a choice worth checking, and our golf article explains one we made deliberately.

None of those choices is dishonest by itself. Every one of them changes the resulting percentage by an amount that can dwarf any genuine difference in modeling skill. So when you see two numbers, the first question is never which is bigger. It is whether the two numbers are answers to the same question.

This is a distinct failure from models simply disagreeing. Two well built models will produce different projections on the same prop for legitimate reasons, as the article on why models disagree works through. That disagreement is informative. Two incomparable records are not informative at all, and treating them as if they were is how a reader ends up confidently wrong about which process to trust.

The conditions a valid comparison has to meet

A head to head is worth reading when five conditions hold at once. Miss any of them and the result reverts to a story.

1. The same markets. Strikeout props and receiving yards props are different difficulties. A tool with a record concentrated in easier markets will look better without being better.
2. The same time window, chosen before the results were known. A window that starts and ends where one side prefers is a selection, not a period.
3. Results recorded at the lines actually posted, from named platforms, at a stated moment. A record graded against numbers nobody could buy describes a market that did not exist.
4. The same treatment of pushes, voids, and cancellations. Exact landings and players who never appear have to be handled identically on both sides, and stated in advance.
5. No filtering by either side after outcomes were known. No dropping a bad sport, no reclassifying a loss as a pass, no retiring a version and archiving its record with it.

The fifth condition is the one that quietly fails most often, because it can fail without anybody deciding to cheat. A team looks at a rough month, finds a defensible technical reason to exclude it, and excludes it. The reason really is defensible. The problem is that the search for it started only after the month came out badly, and a search that only runs after bad results will only ever produce exclusions that help.

Why a short window compares luck

Player prop markets are priced close to fair, which means good processes and mediocre ones both spend most of their time near the middle. Over a hundred settlements the spread of plausible outcomes for a genuinely strong model overlaps almost entirely with the spread for a slightly weak one. The winner of a week is decided by which coin flips landed, and the same two tools compared over a different week would often swap places.

Which is why the question which tool had a better week has no answer worth acting on, while the question which tool is better calibrated over a large sample does. Calibration asks whether the stated probabilities mean what they say: among all the projections a tool called 60 percent, did roughly 60 percent happen? That is a property of the process rather than a property of the schedule, it accumulates evidence from every graded projection instead of only the ones near the decision boundary, and it is difficult to fake across confidence levels. The single number companion is the Brier score, which rewards confidence only when confidence was warranted.

Comparing calibration also removes a distortion that win rates carry. A tool that only publishes its most confident opinions can post a higher percentage than a tool that publishes everything, without being more accurate about anything. Calibration compares the honesty of both tools' probabilities across their full range, which is closer to what you actually want to know before relying on either one.

A checklist you can apply to any tool

Most readers will never have two comparable ledgers in hand. What you can do instead is evaluate each tool separately against a fixed standard, then compare the evaluations rather than the percentages. Five questions carry most of the weight.

• Does it publish a complete record, or a selection? Ask specifically what happened to the markets and sports that did not perform. Absence is data.
• Can the record be edited quietly? A history that can change without a visible trace is not evidence, it is a claim that is refreshed whenever it stops being flattering.
• Does it distinguish real posted lines from theoretical ones? Grading against a line a named platform actually sold is a much stronger claim than grading against a consensus number.
• Does it admit uncertainty? Look for stated limitations, visible caps on new or thin markets, and a difference between what is measured and what is assumed.
• Does it ever say no play? A tool whose output is always an opinion has confused coverage with conviction. Knowing when to pass, as the passing article argues, is part of the product.

These questions have the useful property of being answerable in an afternoon, without statistical training and without either tool cooperating. They are also hard to game, because passing them requires structural choices made long before you looked.

Applying the standard to Slateline

It would be convenient to end an article about fair comparison by declaring ourselves the winner. We are not going to, because the argument above forbids it: we do not have comparable ledgers from other tools, and without them any claim of superiority would be the exact marketing comparison this article was written against. Other research products in this space make real choices we respect and cover things we do not.

What we can state is what we publish, so the checklist can be run on us. Every live sport's graded record is public in the Model Room: hit rate, probability accuracy, calibration, and how model versions have performed, counted at the lines that were posted, with losses and voided legs in the ledger rather than removed from it. Markets without enough graded history carry visible caps instead of confident grades. Projections that fail our own guardrails are published as no plays rather than hidden. We also write about our misses in public, which that article covers directly.

Some of that record is unimpressive, and some markets have too little history to say anything at all. Both of those facts are visible on the same page as the good numbers, which is the only arrangement that makes the good numbers mean anything. If you are evaluating tools, run the five questions on ours the same way you run them on everyone else's, and start from the Signal Board so you can watch the projections before the results exist. A tool that asks to be graded on its own terms is asking you to lower your standard.