In-sample and out-of-sample: why split the data in a backtest

Splitting the sample into in-sample and out-of-sample is the basic way to check whether a system genuinely works or is merely fitted to the past. The idea is simple: tune the strategy on one part of the history and test it on another part it 'hasn't seen.' Let's look at why this is needed and how it catches overfitting.

The essence of the method

The history is split into two parts. In-sample (the in-sample data) is the stretch on which you develop and tune the system and select parameters. Out-of-sample (the out-of-sample data) is a stretch set aside in advance and not used during tuning; on it you test the finished system. The key condition: the out-of-sample data must remain 'untouched' until the final check. If you peek at it during tuning, the method loses its meaning — you're once again fitting the system to the whole dataset.

Why it's needed

The problem with a backtest is that on the data the system was tuned on it almost always looks good — because that's what it was fitted to. That says nothing about the future. Testing on out-of-sample data emulates meeting an unfamiliar market: if the system works on data it hasn't seen, it probably has a real edge. But if everything is beautiful in-sample while the out-of-sample result falls apart, the system was overfitted to a specific stretch.

How to read the result

The logic of the check is simple. The system works well both in-sample and out-of-sample — an encouraging sign: the edge appears real and stable. Excellent in-sample but poor out-of-sample — the classic sign of overfitting: the system memorized the noise of the training stretch. It's reasonable to expect the out-of-sample result to be somewhat more modest than in-sample (a small deterioration is normal), but it shouldn't fall apart. A sharp drop in quality on unfamiliar data is a signal that the system can't be trusted.

Limits of the method

Splitting the sample is a necessary but not absolute safeguard. A single out-of-sample stretch may happen to be lucky or unlucky. That's why the method is reinforced: several out-of-sample periods are used, along with walk-forward testing, and testing across different markets and regimes. Realistic accounting for costs also matters — without it, even an honestly tested system may not survive real spreads and slippage. Splitting the sample is the foundation of honest testing, but not its only element.

The practical takeaway

In-sample and out-of-sample means dividing the history into a stretch for tuning the system and a set-aside stretch for testing. Tune the strategy on the in-sample data and test on the untouched out-of-sample data: if the system works on data it hasn't seen, it probably has a real edge; if it's excellent on the training stretch and falls apart on the test stretch, that's overfitting. A small deterioration out-of-sample is normal; a sharp drop is a warning signal. Don't peek at the test data during tuning, or the method is useless. Reinforce the check with several periods, walk-forward testing, and realistic costs. Understanding and applying the split is the basic skill of honest testing, separating a system with a real edge from a pretty fit to the past.

This material is for educational purposes and is not individual investment advice.