Trading Data Statistical Reliability: Read Your Journal Correctly

A pattern in a trading journal is statistically reliable only when several things line up together: enough evidence relative to the size of the effect being claimed, protection against picking the one split that happened to look good out of many tried, observations stable and comparable enough to count as the same kind of evidence, and — when the pattern was found by scanning the journal rather than checked in advance — confirmation on trades logged after it was noticed. A pattern that clears all of that still has to be large enough, after costs, to be worth acting on; statistical support and economic usefulness are separate questions. A journal can show a real pattern, a temporary artifact of the period it was logged in, or nothing but noise dressed up as a result — and the same win rate or expectancy figure can honestly belong to any of the three.

This article covers reading statistical reliability into data you have already logged: win rates, expectancy figures, and patterns pulled from your own trade or session history. It does not cover validating a systematic rule set through backtesting and forward testing before it is traded live — backtest vs. forward test owns that separate pipeline and its own overfitting checks. It also does not replace the layer-specific frameworks for what to measure in a full review (trading performance) or one setup’s own segmentation (trading setup performance); this is the reliability check that sits underneath both, applied to whichever number from either framework you are about to act on.

How many trades before a number stops being noise?

There is no fixed trade count that makes a result trustworthy. How much evidence you need depends on the size of the effect you’re judging relative to your actual baseline, how variable individual outcomes are, and how stable and independent the underlying trades are — not on trade count by itself.

A coin that lands heads 7 times in 10 flips tells you almost nothing about whether it’s fair, and a trading setup that wins 7 of 10 trades sits in the same statistical position: a 70% observed win rate from 10 trades carries wide uncertainty, even though it can feel more meaningful because you were watching it happen. That comparison illustrates how uncertainty around a proportion behaves in a small sample; it is not a claim that 50/50 is the relevant reference point for judging a trading setup, which is a separate question covered next.

Fifty percent is not the baseline that tells you whether a setup has an edge. A coin flip with an even payoff breaks even at a 50% win rate because a win and a loss are worth exactly the same amount. A trading setup usually isn’t that: its break-even win rate is implied by the size of an average winning trade versus an average losing trade, plus commissions, spread, and slippage. A setup where the average win is twice the average loss breaks even at roughly a 33% win rate before costs; a setup where losses run larger than wins needs a win rate above 50% just to avoid losing money. Comparing an observed win rate against a flat 50% instead of your own estimated economic break-even rate can make a genuinely losing setup look fine, or make a genuinely profitable one look unconvincing. Use the coin-flip illustration to understand sampling uncertainty around a proportion — not as a substitute for estimating your own break-even rate from your average win, average loss, and costs. If average win and average loss are themselves estimated from a limited sample, the implied break-even rate carries sampling uncertainty too.

Two things scale how much data you actually need once you’re comparing against the right baseline:

How close the observed win rate sits to your setup’s estimated break-even rate. All else equal, a setup winning 70% of the time against an estimated 33% break-even rate generally requires less evidence to distinguish the observed gap from ordinary sampling variation than a setup winning 55% against an estimated 50% break-even rate, because the smaller gap is easier for ordinary variance to produce on its own.
How variable individual outcomes are. A binary win/loss outcome with a fixed payoff is easier to judge from a small sample than a setup where occasional large wins or losses dominate the total, since a handful of tail outcomes can swing the average result by itself. Prediction-market probability calibration covers the related but distinct case of a position that resolves as one probability estimate meeting one outcome, where the calibration question is judged across many resolved contracts rather than through this win-rate framework at all.

What counts as enough evidence also depends on the payoff distribution, the precision the decision actually requires, whether the trades are independent of each other, how stable the setup definition and market regime have stayed, and what decision the number is meant to influence — the next section covers why trade count and independent information aren’t the same thing. As a cautious practical heuristic, not a statistical cutoff: a result built from a handful of trades is almost always too thin to act on regardless of how it looks, and a narrow edge over its own break-even rate takes substantially more evidence to confirm than an obvious one does. Treat that as a general caution, not a threshold — the factors above are what actually decide when a specific result stops looking like ordinary variance.

More trades doesn’t always mean more information

A larger trade count only adds proportionally more evidence if the additional trades behave like comparable, independent observations. Nominal sample size and effective information aren’t the same thing when trades are:

strongly clustered in one market regime or session type,
generated under a setup definition, entry trigger, or risk rule that changed partway through the sample,
serially dependent — one trade’s outcome shaped by conditions carried over from the trade before it, rather than each trade being a fresh draw,
concentrated in one unusually calm or volatile stretch that doesn’t represent the conditions the setup will actually be traded in going forward.

A hundred nominal trades can contain substantially less than a hundred independent observations when outcomes are clustered within the same market episodes. How much less depends on the strength and structure of that dependence, so trade count alone cannot tell you the effective amount of information in the sample. This article doesn’t attempt to back into an “effective sample size” for a discretionary trading journal — that requires modeling the dependence structure, which most journals don’t record the inputs for — but the practical response holds regardless of the exact number: keep the setup definition and risk rules stable across the sample, check the pattern across more than one market regime or session type where that’s relevant, and weight prospective replication — the pattern holding up on new trades collected later, under different conditions — more heavily than the same raw trade count collected all at once under one set of conditions. Event-contract records show a concrete version of the same problem: several contracts listed under one parent event resolve together, so a favorite-longshot bias test counts events rather than contracts when judging how much evidence a price band holds.

The multiple-comparisons trap: finding a pattern by looking hard enough

If you slice a trading journal enough ways — by day of week, session, an indicator reading, account size, even mood at entry — you should expect at least one slice to look statistically notable purely by chance, even if nothing in the underlying process actually changed. Ioannidis formalized why this happens in research generally: as the number of hypotheses tested against the same data grows, so does the expected number of “significant” results that are actually false positives, and the specific result that gets reported is disproportionately likely to be one of them, because it was selected for looking good rather than tested on its own.¹

The trading-journal version of this is familiar: noticing “I do better after 10am,” or “setups where RSI was above 60 outperform,” after scanning many possible splits rather than checking one you expected in advance. This is the same structural problem backtest vs. forward test covers for a systematic rule set searched across many parameter combinations — the discretionary-journal version just replaces “parameter combinations” with “ways of slicing your own trade log.”

Two disciplines handle this differently, and the difference matters more than any specific number of splits. Confirmatory analysis decides the comparison before looking at outcomes: pick the split, decide in advance what result would count as support, then evaluate it under that rule. Exploratory analysis runs the other direction — scan the journal, notice a candidate split, and label it explicitly as hypothesis-generating rather than proven. A split discovered this way is a reasonable candidate for further attention, but the same data that revealed it cannot by itself serve as independent confirmation, because it was already selected for looking good in the sample where it was found. Confirming a discovered split means freezing its definition and testing it against trades collected after the discovery — the same out-of-sample check covered below.

If you’re informally checking enough splits that you’re effectively running many comparisons against the same journal, the problem researchers formally address with family-wise-error-rate or false-discovery-rate control — different methods for controlling error rates across a family of simultaneous tests — is showing up in your trade log.² You don’t need to run a formal multiplicity correction on a discretionary journal to act on the lesson: a split noticed by scanning many possibilities needs independent confirmation before it should be treated as confirmatory evidence or given substantial decision weight, regardless of how many splits were checked to find it.

Distinguishing a real pattern from a good story

Once a pattern is visible, it is easy to construct a plausible-sounding explanation for it after the fact — that is what a pattern-seeking mind is built to do, and it works whether or not the pattern is real. Discovery and confirmation are different steps, and mixing them is the most common way a coincidence gets treated as an edge. Three things help you judge whether a visible pattern is worth acting on or just a good story:

A plausible mechanism — whenever it’s identified. A mechanism that existed before the pattern was noticed adds prior plausibility and rules out some false positives up front. A mechanism invented after seeing the result is a hypothesis about why the pattern might be real, not confirmation that it is — but it can still be worth testing on that basis. A pattern can also be found empirically before anyone understands why it happens; a missing mechanism doesn’t by itself disqualify a pattern, and a stated mechanism doesn’t by itself validate one.
Persistence on a genuinely new slice of trades. Trades logged after the pattern was first noticed, evaluated the same way, are the closest thing a discretionary trader has to an out-of-sample test. A pattern that fails to hold up on new trades weakens the original result substantially, but a single non-replication has more than one possible explanation: the original sample may have been noise, the new sample may be too small to detect a real but smaller effect, the market regime may have shifted between the two periods, the setup’s definition or labeling may have drifted, or the relationship may genuinely be less stable than it first looked. Treat non-replication as a serious mark against the pattern, not automatic proof it was never real.
An economically meaningful size, not just a statistically notable one. A small, reliable-looking edge can still be too small to matter once trading costs, slippage, and the effort of maintaining it are counted — statistical evidence and decision usefulness are different claims, and treating one as the other is a well-documented and common misreading of what a statistical result can support.³

A pattern that fails one or more of these checks is not necessarily false — it may simply not be established yet. The honest response is usually to keep collecting evidence under the same definition, not to declare the pattern confirmed or discard it outright.

Win-rate uncertainty, the Wilson interval, and why it isn’t an expectancy interval

A win rate calculated from a trade sample is an estimate of an underlying rate, not the rate itself, and the same point estimate can carry very different uncertainty depending on the sample size behind it. The Wilson score interval — a standard method for estimating a plausible range around an observed proportion, and one that performs more reliably than the simpler normal-approximation interval across many binomial settings⁴ — makes the shape of that uncertainty concrete.

Worked example. Seven wins out of 10 trades and 70 wins out of 100 trades are both a 70% observed win rate, but they carry very different uncertainty:

7 out of 10 (70%): 95% Wilson interval ≈ 39.7%–89.2%
70 out of 100 (70%): 95% Wilson interval ≈ 60.4%–78.1%

The same point estimate spans a roughly 50-percentage-point-wide interval in the first case and an 18-point-wide interval in the second. That interval describes a range of win rates consistent with the observed data at a stated confidence level, produced by a method that — applied repeatedly across many samples — contains the true win rate close to 95% of the time. It is not the probability that the true win rate falls inside this one specific observed interval; a computed interval either contains the true rate or it doesn’t, and the 95% figure describes the method’s long-run reliability, not a probability statement about this particular result.

These intervals assume the trades behave like a stable binomial proportion — each one a reasonably independent draw from a win rate that isn’t moving underneath you. Strong serial dependence between trades, a material shift in market regime partway through the sample, or a setup definition that changed along the way can all make the nominal interval overstate how much information the raw trade count actually contains, even though the Wilson arithmetic itself is unaffected. Wilson quantifies the uncertainty you’d have if the trades were independent draws from a fixed rate; it does not detect or correct for dependence or nonstationarity in the data feeding it.

A Wilson interval bounds win rate. It does not bound expectancy. Expectancy combines win rate with the size of wins and the size of losses, and win/loss magnitudes carry their own sampling variability on top of the win-rate uncertainty above — a handful of unusually large wins or losses can move expectancy by more than win-rate uncertainty alone would suggest, especially in a small sample. Do not report a Wilson interval as if it were a confidence interval on expectancy; it answers a narrower question about the win/loss split alone. A genuine uncertainty range around expectancy would need to account for the variance of the win and loss sizes themselves, not just how many trades were wins versus losses — and for most trading journals, the more honest response is to treat a small-sample expectancy figure as a rough compass while more data accumulates, rather than compute an interval that implies more precision than the sample supports.

This does not mean small-sample figures are useless — they are the only figures available early on — but it means a small-sample win-rate or expectancy number should be read as a rough compass, not a precise measurement. Mistake-adjusted expectancy and trading setup performance cover how to calculate expectancy and win rate correctly for a given sample; this section covers the separate question of how much weight those calculated figures deserve once you know how large, and how independent, the sample behind them actually is.

A reliability checklist before acting on a journal pattern

Check	Question to ask	Fails if
Sample uncertainty	Is the win-rate/expectancy uncertainty (e.g., a Wilson interval on win rate) narrow enough for the claim being made?	A wide plausible range would already change the conclusion
Economic break-even	Is the comparison baseline your setup’s estimated break-even rate — implied by average win, average loss, and costs — not a flat 50%?	The result is only being judged against 50% instead of the rate that would actually break even
Stable measurement	Has the setup definition, risk rules, and execution process stayed the same across the sample?	Definitions or rules shifted mid-sample without splitting the data at that boundary
Dependence / regime concentration	Are the trades reasonably independent and drawn from more than one relevant market condition?	The sample is really a small number of correlated episodes, or all from one unusual regime
Pre-specified vs. discovered	Was this split decided before looking, or found by scanning many splits?	The split was one of many tried and treated as confirmed rather than exploratory
Multiple comparisons	If many splits or hypotheses were checked, has that been accounted for?	A pattern is reported as if it were the only thing tested
Prospective confirmation	Does the pattern hold on trades logged after it was first noticed?	The pattern only appears in the sample that produced it
Economic relevance	Is the effect large enough, after costs, to actually change a decision?	The estimated edge is real but too small to matter in practice

A pattern that clears every row above is still an observation from your own trading, not a guarantee it continues — conditions can shift later even after a well-supported result. A pattern that fails several of them is not yet strong enough evidence to generalize confidently beyond the specific trades that produced it.

Where Costante fits

Costante’s session planning, guardrails, and low-friction logging produce the consistent setup labels, timestamps, and outcome records this kind of check depends on. Costante does not calculate statistical significance, does not flag which patterns in a trading log are real, and does not decide how many trades are enough before a pattern should be acted on. Those judgments belong to the trader, made from a record that stays consistent enough to support them.

Frequently asked questions

Is there a minimum number of trades before I can trust my win rate?

There is no universal minimum. It depends on how far the observed win rate sits from your setup’s estimated break-even rate — not a flat 50% — how variable individual trade outcomes are, and how independent and stable the trades behind it are. A handful of trades is a safe case to call provisional regardless of how strong the result looks; beyond that, the evidence required scales with how narrow the edge is and how much precision the decision needs, not with a specific trade count.

Is 50% the win rate I need to beat to know if my setup has an edge?

No. Fifty percent is only the relevant no-edge baseline when a win and a loss are worth the same amount before costs, like an even-payoff coin flip. A trading setup’s break-even win rate is implied by the size of its average win relative to its average loss, plus commissions, spread, and slippage — a setup with wins twice the size of losses can be profitable well under 50%, and a setup with losses larger than wins needs a win rate above 50% just to break even. Compare your win rate against your own estimated break-even rate, not against 50%.

I found a pattern by checking several different splits of my journal. Is it real?

Treat it as hypothesis-generating, not confirmed. Scanning many splits and reporting the one that looked best makes a false-positive result likely even when every individual check was performed correctly — the same data that revealed the pattern cannot by itself serve as independent confirmation, because it was selected for looking good in that sample. Freeze the definition and check whether it holds on trades logged after you noticed it before treating it as anything more than a candidate worth watching.

Does a Wilson interval tell me how confident to be in my expectancy figure?

Not directly. A Wilson interval bounds a win rate — a binomial proportion — not expectancy, which also depends on the size of your wins and losses. Two samples with the same win rate and the same Wilson interval can have very different expectancy uncertainty if their win and loss sizes vary differently. Use a Wilson interval to judge win-rate confidence specifically, and treat a small-sample expectancy figure as a rough estimate rather than a value with a precisely calculable margin of error.

How is this different from backtesting a trading strategy?

This article covers reading reliability into discretionary trades and journal patterns you have already logged. Backtest vs. forward test covers the separate question of validating a systematic rule set against historical and new data before it is traded live, including its own overfitting checks for a searched set of rule parameters. Replaying your own logged trades under a different stop or target sits between the two, and is covered in stop-loss and profit-target simulation.

My sample is small but the result looks strong. Can I act on it anyway?

You can treat it as a reason to keep trading the pattern under close review, not as a confirmed edge. A strong-looking result from a small sample can still be entirely explained by ordinary variance, and the uncertainty around a small-sample win rate or expectancy figure is often wider than it looks from the point estimate alone. Continuing to collect trades under the same definition, and checking whether the pattern holds as the sample grows, is what eventually separates a real pattern from a lucky run.

Sources

Wilson, E. B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association, 22(158), 209–212.
Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101–133.
Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124.
Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300.
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.

Costante provides educational workflow tools, not financial advice. Trading involves risk.

A 2022 correction to Ioannidis’s paper fixed a formula error in Table 2; it does not affect the multiple-testing point cited here. ↩
The Benjamini-Hochberg false-discovery-rate procedure is one standard way multiplicity is formally controlled when many hypotheses are tested at once; it’s referenced here for the underlying concept, not as a procedure this article is asking a discretionary trader to run by hand. ↩
The American Statistical Association’s statement on p-values explicitly warns against equating statistical significance with the importance, size, or practical relevance of an effect — the same distinction drawn here between statistical evidence and economic usefulness. ↩
Brown, Cai, and DasGupta showed that the standard normal-approximation (“Wald”) interval for a binomial proportion has poor and erratic coverage across much of the parameter range, and recommended the Wilson interval (or comparable alternatives) instead — the basis for using it here rather than the simpler normal approximation. ↩