A backtest is a rejection machine, not a proof. The arithmetic of trying too many ideas, the four biases that flatter every result, and the rule the desk uses — a configuration counts only if its neighbours work too.
A backtest replays a set of rules over history and reports what would have happened. That sounds like evidence, and it is — of one thing. A backtest can tell you, cheaply and reliably, that an idea would have lost. What it cannot do, no matter how good the number, is tell you that an idea will make money. The difference is not philosophical. It is arithmetic, and the arithmetic has been worked out.
The number of tries is the whole story
Suppose you test a strategy with no edge at all — true expected return zero — on two years of data, and then you test a second version with slightly different settings, then a third. Each test is a fresh roll of the dice, and the best of several rolls is better than average by construction. Bailey, Borwein, López de Prado and Zhu worked out how fast that gets out of hand: with two years of history, after trying only seven independent configurations the expected best in-sample Sharpe ratio is already 1.0 — a number that would get a strategy funded — while the true out-of-sample Sharpe of every one of them is zero. With five years of data they put the ceiling at about 45 independent trials before the best result is more likely noise than signal.
Nobody tries seven. A parameter sweep tries thousands. The desk's own August 2026 sweep tested 9,363 configurations on real ES history; 74 of them passed a serious robustness screen, and 74 is roughly the number pure noise produces at that search width. That sweep is the subject of the win-rate essay, and the lesson is the same one the mathematicians reached: the 75th finding is not a discovery.
Academic finance learned it the hard way. Harvey, Liu and Zhu counted hundreds of published factors claiming to explain stock returns and concluded that a new one should clear a t-statistic above 3.0 rather than the usual 2.0, and that "most claimed research findings in financial economics are likely false." McLean and Pontiff then measured what happens to 97 published predictors after they are published: returns were 26% lower out of sample and 58% lower after publication. An edge that everyone can read is an edge that is being arbitraged, and an edge that was never there decays to nothing the moment the data it was fitted to runs out.
The four biases that flatter every result
Even one honest test can lie, in four known ways.
Look-ahead bias — using information that was not available at the time. The classic case is accounting data restated after the fact; Banz and Breen showed in 1986 that building a test on a database of restated figures "may introduce a look-ahead bias." In a trading script the same bug hides in a rule that peeks at the bar's close before the bar has closed, or at an indicator computed on data the strategy would not yet have had. A result that uses tomorrow's number is not a result.
Survivorship bias — testing on the things that are still here. Brown, Goetzmann, Ibbotson and Ross showed in 1992 that studying only the funds that survived "gives rise to the appearance of predictability" that was never there. A stock strategy backtested on today's index members has quietly skipped every company that went to zero. Index futures are mercifully free of this one — the contract that was there in 2008 is the same contract — which is one reason the desk tests on them.
Data snooping — the trials problem again, at the level of a whole literature. Sullivan, Timmermann and White tested 7,846 technical trading rules on a century of Dow Jones data. The best rule looked excellent in sample. Its performance "is not repeated in the out-of-sample experiment" covering 1987 to 1996. Every rule you have read about was, in effect, selected by thousands of people trying thousands of rules.
Cost blindness — the quietest one. A backtest that assumes it is filled at the signal price, with no commission, will show a profit from a system that wins two ticks a trade. Real fills cost a tick or more each side and commission on top. The desk's engine charges every modeled trade two ticks of slippage per side and a commission before it reports anything; a result that was not computed that way should be read as gross, and gross is usually the whole edge.
What the honest tools look like
Two ideas carry most of the weight. The first is an out-of-sample period: a stretch of history the designer did not get to see while building, tested once, at the end. If the rule only works on the years it was tuned on, that is the answer. The second is to deflate the headline number for the number of tries: Bailey and López de Prado's deflated Sharpe ratio adjusts a strategy's Sharpe for how many configurations were tested and for the fact that trading returns are not normally distributed. A Sharpe of 1.5 found after 500 trials is a different object from a Sharpe of 1.5 found on the first try, and the deflated version says so.
The desk's own rule is a third thing, cruder and useful: a configuration counts only if its neighbours work too. A system whose stop is set at 8 points should still be profitable at 7 and 9; a system that works on the 15-minute chart should not fall apart on the 10 and the 20. A result that lives only at one exact setting is a result that has been fitted to the noise around that setting, and nudging every parameter is the cheapest test of that there is.
What this cost the desk
In August 2026 the system running as the desk's flagship looked good on the timeframe it had been selected on: 57% of trades profitable, profit factor 2.93. Tested on a full year of hourly data it had not been built on, it won 20% of its trades with a profit factor of 0.48 and finished only 3 of 12 months ahead. It had been fitted to its own window. It was retired rather than re-tuned, because re-tuning a curve fit produces a better-hidden curve fit, and the replacement was chosen by the neighbours rule instead of by the best score. That record, including the unflattering numbers, is on the demo desk.
Three questions that catch most of it
When someone shows you a backtest — including this desk — ask for three numbers and watch which ones they have.
- How many trades? Under about thirty, the result is noise whatever it looks like.
- Which years were out of sample, and when were they tested? "All of them" means none of them.
- How many versions were tried before this one? The honest answer is usually "a lot," and the right response to it is to deflate everything that follows.
A backtest answers one question well: would this have lost? Use it for that, every time, and treat a good result as permission to test harder rather than as a reason to stop.
- Pseudo-mathematics and financial charlatanism — the effects of backtest overfitting on out-of-sample performance — Bailey, Borwein, López de Prado and Zhu, Notices of the American Mathematical Society 61(5), May 2014 (authors' preprint)
- … and the cross-section of expected returns — Harvey, Liu and Zhu, Review of Financial Studies 29(1), 2016
- The deflated Sharpe ratio — Bailey and López de Prado, Journal of Portfolio Management 40(5), 2014
- Does academic research destroy stock return predictability? — McLean and Pontiff, Journal of Finance 71(1), 2016
- Data-snooping, technical trading rule performance, and the bootstrap — Sullivan, Timmermann and White, Journal of Finance, 1999 (LSE discussion paper version)
- Survivorship bias in performance studies — Brown, Goetzmann, Ibbotson and Ross, Review of Financial Studies 5(4), 1992
- Sample-dependent results using accounting and market data — Banz and Breen, Journal of Finance 41(4), 1986
- How to read a win rate — the desk's own 9,363-configuration sweep
Education, not advice. Nothing on this page recommends buying or selling anything, and nothing is a promise of results — every figure is cited above or hypothetical. TOTH does not provide investment advice. New to the vocabulary? The glossary has plain definitions.