Sample Size: Why a Small Sample Deceives

Sample size — the number of trades you use to evaluate a strategy — determines whether you can trust the conclusions at all. A small sample deceives: it reflects randomness, not a real edge. Let's look at why sample size matters so much and how many trades you need for meaningful conclusions.

Why Sample Size Matters

All strategy-evaluation metrics (win rate, expectancy, profit factor, drawdown) make sense only on a sufficient sample. On a small number of trades, these metrics reflect not the system's real properties but randomness — the luck or bad luck of one specific short series. A real edge shows up only over distance, where randomness averages out and the pattern emerges. Sample size determines whether you can tell signal (a real edge) from noise (randomness). So before drawing any conclusions about a strategy, you must ask: is the sample sufficient to trust them? Without that, the evaluation is meaningless.

How a Small Sample Deceives

A small sample deceives in several ways. A random lucky streak creates the illusion of an excellent system: a few wins in a row push up the win rate and profit factor, creating false confidence. A random unlucky streak, conversely, can make you reject a good system. Individual large trades on a small sample distort the metrics (one big win inflates the profit factor). Even a fully random system (with no edge) can show a profit on a short series — just by luck. The problem is that on a small sample it's impossible to tell whether the result is due to an edge or randomness: both look identical. That's exactly why conclusions from a few trades, a week, or even a month of trading are often deceptive.

How Many You Need for Conclusions

The exact number needed depends on the statistics, but the general principle is: the more, the more reliable, and you need dozens, better hundreds, of trades for meaningful conclusions. You can't judge by a few trades at all. A few dozen give a first, still rough, impression. Hundreds of trades give a fairly reliable estimate of the metrics. The lower the strategy's win rate and the greater the spread of results, the larger the sample needed for reliability. The quality of the sample matters too: it should span different market conditions (not just a favorable period), or even a large number of trades from one regime won't reveal real durability. The general rule is to distrust conclusions about a strategy until a sufficient sample is gathered, and to treat small-sample results as preliminary, not final.

How to Account for Sample Size in Practice

Accounting for sample size changes how you treat results and decisions. Don't draw conclusions about a strategy from a few trades, a day, or a week — that's noise; wait for a sufficient sample. Don't abandon a system over a short losing streak (it may be random) and don't glorify it over a short lucky streak (it may be luck). Evaluate metrics (win rate, expectancy, profit factor) only on a sufficient number of trades. When testing a strategy, ensure a large sample spanning different conditions. Remember that verifying a change to the system also needs a series of trades with it, not a couple of examples. Think in series and distance, not individual outcomes. Understanding that a small sample deceives protects you from two mistakes — believing in a randomly lucky system and rejecting a good one over a randomly unlucky streak — and trains you to trust only statistically significant results.

The Practical Takeaway

Sample size — the number of trades for evaluating a strategy — determines whether you can trust the conclusions: all metrics (win rate, expectancy, profit factor, drawdown) are meaningful only on a sufficient sample, and on a small number of trades they reflect randomness (luck or bad luck), not a real edge, which shows up only over distance. A small sample deceives: a random lucky streak creates the illusion of an excellent system (inflating metrics), an unlucky one makes you reject a good one, individual large trades distort the metrics, and even a random system with no edge can show a profit on a short series by luck — on a small sample you can't tell an edge from randomness. For conclusions you need dozens, better hundreds, of trades (the lower the win rate and greater the spread, the more), and the sample should span different market conditions. Account for this in practice: don't draw conclusions from a few trades, a day, or a week (that's noise), don't abandon a system over a short losing streak or glorify it over a lucky one, evaluate metrics only on a sufficient number of trades, ensure a large sample when testing, and think in series and distance. Understanding that a small sample deceives protects you from two mistakes (believing in a randomly lucky system and rejecting a good one over random bad luck) and trains you to trust only statistically significant results.

This material is for educational purposes and is not individual investment advice.