Data Mining Bias: How Testing Enough Ideas Fools You

Test enough variants and the best of them will look excellent — even when none of them has an edge. This is not a rare anomaly but an unavoidable property of searching, and most of the impressive results you are shown are built on it.

Data mining bias in our own numbers

We ran ten classic rules across twelve pairs. The best result belongs to "a pullback into the 38-61% zone of a 20-bar range with a trend filter": a median return of 2.44% with 8 profitable pairs out of 12. It looks like a finding. But we obtained that figure by selecting the best of ten, and here is what that means: even if none of the ten had any edge, one of them would still have come out best, and its result would differ from zero purely through dispersion.

That is exactly why, in the article about that rule, we wrote that the result should not be filed as confirmation of the method. The number of variants tested is part of the result, and it cannot be left unmentioned.

How many variants are really being tested

Usually far more than it appears. Ten rules is the visible top layer. Beneath it sit parameter choices: changing a moving average from 20 to 21 periods is another test. Then instrument choice, timeframe choice, choice of history window. Someone who examined a hundred combinations and showed you the best has not lied about a single figure — they have simply not reported the denominator.

How this differs from overfitting

Overfitting is fitting the parameters of one system to history. Data mining bias is broader: it arises from searching across different systems, instruments and periods. You can optimise nothing at all and still fool yourself simply by testing enough ideas and reporting the one that worked.

What to do about it

First and most important, count and report the number of variants tested. Second, demand a larger margin from a result the more variants were examined. Third, keep part of the data untouched until the very end and test only the final candidate on it, once — splitting the sample only helps if the held-out part is used a single time. And fourth, ask of anyone else's result: how many variants were tested before this one? Without that answer a return figure cannot be interpreted.

This material is educational and is not individual investment advice. Backtested results do not guarantee similar results in the future. Trading forex carries the risk of losing capital.