Prediction Market Probability Calibration

To assess whether a prediction-market probability forecast is calibrated, do not judge a single forecast by whether the contract it was attached to resolved YES or NO. Judge a group of forecasts that shared roughly the same stated probability by whether that probability matched the frequency with which similar forecasts actually resolved true. A single 70% forecast on a BTC price-threshold contract, a CPI print, or a Fed-decision market that resolves NO is not evidence of a bad forecast on its own — a forecaster who says 70% is supposed to miss about three times in ten. Calibration is a property of a forecasting record across many resolved contracts, not a verdict on any one binary outcome. Whether a calibrated probability also represents a profitable trade against market prices is a separate question, covered later in this article.

This is a narrower question than the trading performance review used for most discretionary trades. That framework separates results, risk, and execution for positions with a continuous P&L shape. An event contract resolves to one of two states — the market settles YES or NO — which removes the usual outcome-magnitude evidence: the contract produces a binary realization, not a continuous result, against which the earlier probability forecast is evaluated. Reviewing that estimate needs its own method: calibration, not a rules-adherence or exposure check.

How should prediction-market probability estimates be reviewed after they resolve?

Log the stated probability, the contract, and its category before the contract resolves — not after.
Wait until enough forecasts in a comparable probability range have resolved before drawing a conclusion — see trading data statistical reliability for how sample size changes how much a bucket’s figure can be trusted.
Group forecasts into probability buckets rather than reading each resolution in isolation.
Compare each bucket’s average stated probability with that bucket’s realized resolve-true frequency.
Separate a calibration gap — stated probabilities not matching outcomes — from the separate question of whether calibrated probabilities produce an edge against market prices.

What calibration measures — and what it doesn’t

A probability forecast can be evaluated on more than one property, and the terms for those properties get used loosely enough to be worth separating precisely. Calibration (or reliability) asks whether the stated probability matches the observed frequency: across every forecast where you said “70%,” did roughly seven in ten resolve true? Glenn Brier’s 1950 paper introduced the probability score for verifying weather forecasts — scoring a set of probabilistic forecasts by the squared distance between the stated probability and the realized outcome — and the Brier score remains a widely used proper scoring rule for evaluating binary probabilistic forecasts.¹ It was Allan Murphy’s 1973 partition of that score, not Brier’s original paper, that split total forecast error into three components — reliability (calibration, in the sense above), resolution, and uncertainty — conceptually, Brier score = reliability − resolution + uncertainty.² Resolution in that decomposition is a specific, narrower idea than it sounds: it measures whether a forecaster’s probability groups actually separate cases with different outcome frequencies from the overall base rate — not how confident or “sharp” the forecaster sounds. A forecaster who assigns the same probability to every contract has zero resolution by definition, even in the case where that single value happens to be well calibrated to the base rate of the population being forecast.

One related term is worth a precise note: sharpness describes how confidently concentrated a forecaster’s probabilities are, and it is desirable only subject to calibration — a sharp but poorly calibrated forecaster is simply confidently wrong.³ The rest of this piece is about calibration specifically, because it is the property a resolved-contract record can test directly.

Grouping forecasts by stated probability and comparing the bucket average against the realized frequency — sometimes plotted as a reliability diagram, with perfect calibration lying on the 45-degree identity line — is the practical tool the rest of this article uses to check calibration. Murphy and Winkler applied that same grouped stated-probability-versus-realized-frequency comparison to evaluate subjective forecasts of precipitation and temperature, which is direct historical precedent for using this kind of comparison to assess a forecaster’s reliability — not a claim that their 1977 paper defined every present-day calibration-visualization method.⁴ For an event-contract trader, the practical version is: a calibration review tells you whether your probability language means what you think it means, nothing more.

The single-resolution trap

Treating one resolved contract as proof a forecast was right or wrong is the probabilistic-forecasting version of outcome bias: letting the known result stand in for the quality of the estimate that preceded it. A 70% forecast that resolves NO and a 70% forecast that resolves YES are both fully consistent with good calibration — the label only makes a claim about the long-run frequency of forecasts made under similar stated confidence, not about any individual case. The reverse trap is just as common: a single correct high-confidence call gets treated as proof of forecasting skill, when one resolution carries no more evidence than one loss does. Both errors evaluate a sample size of one against a claim that is only meaningful in aggregate.

Build the calibration record before resolution

A calibration review is only as good as the record it starts from, and the record has to be built before the answer is known. For each forecast worth tracking, capture at minimum:

the contract and its category or domain (crypto price events, macro releases, rate decisions, commodities — whatever grouping is meaningful to you);
the probability you assigned, and the timestamp you assigned it;
the resolution date and outcome once the contract settles; and
the market-implied probability at the time you recorded your estimate, if you want to test for an edge later rather than calibration alone.

If a probability changes before resolution — 55% at first look, 65% a day later, 80% just before close — that revision history needs a rule set in advance, not chosen after the outcome is known. Pick one observation convention and hold to it: the probability at initial forecast, the probability at a fixed horizon before settlement, the latest probability before a defined cutoff, or every update logged and analyzed as its own separate forecast observation. Mixing conventions — using whichever stated probability looks best once the resolution is known — breaks a calibration bucket the same way logging after resolution does.

Recording the stated probability after the outcome is already known does not just risk memory bias; it contaminates the record with information that was not available at forecast time, which is a different number regardless of how confident the recall feels. If logging a probability at the moment you form it does not already fit inside your existing review habits, a daily trading routine is the place to attach that checkpoint, and a trading journal app built around configurable fields, rather than a fixed trade-only template, is generally easier to adapt to a dedicated probability field. Whichever tool holds the record, trading-journal data-quality checks — duplicate entries, missing forecasts, and consistent timestamping — apply to a logged probability the same way they apply to a logged trade: a calibration bucket built on a duplicated or dropped forecast is wrong for the same reason a duplicated or missing trade breaks a performance calculation.

Read the calibration curve at a review boundary

At a review boundary you set in advance, group every resolved forecast into probability buckets — deciles if you have enough resolved contracts, coarser ranges like 50–70%, 70–90%, and 90%+ if you don’t. For each bucket, compute:

Realized frequency (bucket) = resolved forecasts in the bucket that settled YES
                                / total resolved forecasts in the bucket

A contract only enters a bucket’s denominator once it resolves; a still-open position is excluded until it settles, not treated as a partial hit or miss. Compare that realized frequency with the bucket’s average stated probability. A well-calibrated bucket has the two numbers close together. A bucket where realized frequency runs consistently below the stated probability shows overconfidence in that range; consistently above shows underconfidence.

Before concluding a bucket is genuinely miscalibrated, weigh the sample size behind it. An observed frequency built from a handful of resolved contracts carries substantial sampling uncertainty — the same underlying calibration can produce noticeably different bucket frequencies just from ordinary binomial variation, and that uncertainty narrows only as more resolutions accumulate. You do not need to compute a confidence interval by hand to use this: treat a bucket built from few resolutions as inconclusive by default, prefer a coarser bin when a finer one would leave too few observations to interpret, and treat a gap as more credible once it has shown up consistently as more forecasts resolve into that bucket, rather than from one small sample.

Buckets are a practical tool for this review, not the only valid way to check calibration. Results can shift with where the bin boundaries fall when data are sparse, and more advanced calibration estimators exist for exactly that reason — but grouped buckets, read with the sample-size caution above, are enough for the review this article describes.

Calibration is not the same thing as an edge

Being well calibrated means your stated probabilities are accurate on average. It does not mean trading on them is profitable.

A standard event-contract price can be read as an implied probability under that contract’s payout structure — what the market charges for a claim that pays a fixed amount if the event resolves YES. That reading needs a horizon caveat: prediction-market price calibration has been found to vary with time until expiration,⁵ and more recent, platform-affiliated research on resolved Kalshi markets reports strong aggregate calibration as resolution approaches⁶ — evidence worth knowing, not a settled claim that market prices are ground-truth probability. Treat an executable price as an implied probability under its payout structure, not as a fact to trade against without checking. When the same event is quoted at different prices on different venues, prediction market price disagreement covers how to tell whether the gap reflects a contract difference, a cost, or a real difference in belief. Research on resolved contracts also reports that prices at the extremes can be biased, with low-priced contracts winning less often than their price implies; favorite-longshot bias in prediction markets covers how to test that against your own record.

For a standard binary contract, a theoretical ex-ante advantage exists when the trader’s probability estimate implies positive expected value relative to the executable price after applicable fees and execution costs. Equivalently, the estimate must exceed the cost-adjusted break-even probability for a YES position, with the corresponding logic for NO. That comparison holds at the moment the trade is available and does not require the contract to have resolved. How to compute the executable price for a given size, rather than reading it off the displayed probability, is covered in prediction market liquidity and executable price.

It is not the same as a repeatable edge. A single apparent positive-EV trade can be estimation error or ordinary sampling noise rather than a real advantage. Confirming a persistent edge needs a sufficiently large out-of-sample record of resolved trades — a distinct test that belongs with the broader results and risk layers of a trading-performance review, not with the calibration record itself. A trader can be well calibrated with no ex-ante advantage once costs are counted, or show an apparent edge in one instance that later turns out to be noise. Neither this article nor a calibration record settles which of those applies.

A worked example

A trader logs forecasts on BTC price-threshold and macro-event contracts — “BTC above $X at settlement,” “will CPI exceed 3.0%” — across six months, grouped by stated-probability bucket at each review boundary. In the 60–70% bucket, 40 forecasts have resolved by the second review boundary, 26 of them YES — a 65% realized frequency, consistent with the stated range. No adjustment is made; the bucket is retained for continued tracking.

In the 85–95% bucket (average stated probability around 90%), the first review boundary shows 21 resolved forecasts, 16 of them YES — about 76%, enough to flag the bucket but still based on a limited sample. By the next review boundary the cumulative sample has grown to 42 forecasts, with 30 YES — about 71%. Because the later sample contains the earlier 21 observations, these are not two independent confirmations of a gap; it is one growing sample. What matters is that the discrepancy against the roughly 90% stated average remained substantial rather than shrinking as another 21 forecasts entered the record — that makes the bucket more worthy of investigation than the initial snapshot did, though it still does not by itself establish a stable bias.

That pattern moves the bucket from “possible noise” to something worth a defined response — but the response has to stay inside what calibration evidence can actually support. The trader flags the 85–95% range for review: checking which financial-event categories are driving the gap, whether the evidence threshold used to justify a probability above roughly 85% has been too permissive, and continuing to collect resolutions rather than treating the current sample as settled. What calibration evidence alone does not justify is a position-sizing decision — that would import a risk-management conclusion from a forecasting-accuracy finding, the same kind of category error the rest of this article argues against. Any change to sizing or exposure is a separate decision made through the trader’s own risk process, not a direct output of the calibration record. Confidence calibration for position sizing covers the additional checks — a sufficient sample, a predefined adjustment rule — a calibrated bucket like this one would still need to clear before it could influence a discretionary trade’s size.

Common calibration-review failures

Judging one resolution as proof the forecast was right or wrong. A single outcome cannot confirm or refute a probability claim; only a bucket of comparable forecasts can.

Logging the probability after the outcome is known. A number recorded with the resolution already visible is a rationalization, not a forecast.

Drawing a conclusion from a thin bucket. A handful of resolved contracts in one probability range is not enough evidence to call a pattern stable.

Treating calibration as proof of an edge. Accurate probabilities and a profitable trading edge are different claims; if the executable market price already reflects your probability estimate, that estimate alone provides no probability-based trading edge after applicable costs.

Pooling every category together. A trader can be well calibrated in one domain and poorly calibrated in another; combining them into one overall score hides exactly the pattern a category-level review is meant to surface.

Category segmentation itself carries a tradeoff worth naming. Splitting crypto, macro, rates, and commodities contracts into separate buckets can reveal domain-specific miscalibration that a pooled score would hide, but every additional split leaves fewer resolved forecasts in each cell, which weakens the kind of conclusion a calibration review is trying to reach — the same sample-size problem covered above, recreated one level down. Split by category when the distinction is substantively meaningful — a domain where your process or information genuinely differs — and when enough resolved observations exist in the resulting segment to interpret it; splitting further than that just produces more thin buckets.

Where Costante fits

Costante’s relevance here is adjacent: it concerns the review discipline around a decision, not the probability-calibration calculation itself. Costante’s core workflow is built for discretionary intraday traders: planning rules before the session, retaining trade context, checking execution against those rules, and reviewing behavioral drift afterward. A prediction-market trader applying the framework in this article would still need a separate record that preserves each ex-ante probability, timestamp, contract, and eventual resolution. Costante does not calculate or analyze forecast calibration. It does not provide a dedicated prediction-market probability record, calculate a Brier score or its reliability, resolution, or uncertainty components, generate reliability diagrams or calibration curves, estimate event probabilities, predict contract outcomes, recommend prediction-market contracts, connect to prediction-market exchanges, execute prediction-market trades, or infer position size from calibration. The probability record, bucket analysis, and every interpretive or sizing decision remain the trader’s own.

Frequently asked questions

How many resolved forecasts do you need before trusting a calibration bucket?

There is no universal number. A bucket’s observed frequency carries binomial sampling uncertainty that shrinks as more resolutions accumulate, so a handful of resolutions is rarely enough to distinguish a genuine gap from ordinary variance. Treat an early apparent gap as a reason to keep collecting evidence, and look for the pattern to hold across more than one review boundary before acting on it.

Does being well calibrated mean a trader has an edge in event contracts?

No. Calibration measures whether stated probabilities match realized frequency. A theoretical ex-ante advantage exists when the trader’s probability estimate implies positive expected value relative to the executable contract price after applicable fees and execution costs. Equivalently, it must clear the relevant cost-adjusted break-even probability. Confirming it as a repeatable, reliable edge is a separate, further test that needs a large out-of-sample resolved record, and belongs with a broader trading performance review, not with the calibration record on its own.

Should a probability estimate be logged before or after a contract resolves?

Before. A number recorded or reconstructed after the outcome is known reflects hindsight, not a forecast. Log the stated probability at a fixed point before resolution, and record any later revision as a separate, timestamped entry rather than overwriting the original.

Is a single 70% forecast that resolves NO evidence the forecast was wrong?

Not by itself. A forecaster stating 70% should expect that specific outcome to fail roughly three times in ten; only a pattern across many similarly stated forecasts can show whether the 70% label is actually accurate.

Sources

Costante provides educational workflow tools, not financial advice. Trading involves risk.

Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1), 1–3. ↩
Murphy, A. H. (1973). A New Vector Partition of the Probability Score. Journal of Applied Meteorology, 12(4), 595–600. ↩
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007). Probabilistic Forecasts, Calibration and Sharpness. Journal of the Royal Statistical Society: Series B, 69(2), 243–268. ↩
Murphy, A. H., & Winkler, R. L. (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Journal of the Royal Statistical Society: Series C, 26(1), 41–47. ↩
Page, L., & Clemen, R. T. (2013). Do Prediction Markets Produce Well-Calibrated Probability Forecasts? The Economic Journal, 123(568), 491–513. ↩
Kagan, N., & Baiocchi, R. (2026). Calibration in Prediction Markets: Theory and Evidence. Kalshi Research working paper — platform-affiliated, not independent peer-reviewed research. ↩