How to read a track record (and the ten ways it lies)

A track record is an argument dressed as a table. Someone chose which account, which dates, which costs and which numbers to show, and they chose them because they wanted a reaction from you. That is not an accusation — this desk publishes a record and makes the same choices — it is the reason regulators wrote rules about it and academics spent careers measuring how far the published numbers sit from the real ones. This page is the rules, the measurements, and ten questions.

What the rules require, and why they exist

U.S. futures regulation has a specific view of a record that did not happen. CFTC Rule 4.41 requires anyone advertising simulated or hypothetical results to say, "prominently" and "in immediate proximity" to the numbers, that they "do not represent actual trading," that they "may have under- or over-compensated for the impact, if any, of certain market factors, such as lack of liquidity," that hypothetical programs "are designed with the benefit of hindsight," and that "no representation is being made that any account will or is likely to achieve profits or losses similar to these being shown." NFA Compliance Rule 2-29, which governs its members' promotional material, bars anything that "mentions the possibility of profit unless accompanied by an equally prominent discussion of the risk of loss," requires the familiar capitalised disclaimer — HYPOTHETICAL PERFORMANCE RESULTS HAVE MANY INHERENT LIMITATIONS — and adds a rule most people have never heard: once a firm has three months of actual trading results for a system, it may no longer show hypothetical results for it at all.

On the securities side the SEC's 2020 marketing rule for investment advisers forbids showing gross performance "unless the advertisement also presents net performance" with "at least equal prominence," and allows hypothetical performance only with policies to ensure it is relevant to the audience and with disclosure of "the criteria used" and "the risks and limitations." Its definition is broad on purpose: hypothetical performance "means performance results that were not actually achieved by any portfolio of the investment adviser," including model portfolios, backtests, and "targeted or projected performance returns."

Read those as a list of what the regulators have seen go wrong. Every clause is a scar.

The ten ways

1. It is hypothetical and does not say so loudly enough. Backtests, model accounts, "what the signals would have done" — all of it is a simulation until an account was traded. The CFTC's fraud advisory on trading systems puts the first question plainly: "Do you know if the advertised results are based on actual or 'hypothetical' trading?" Every figure on this desk's public pages is hypothetical and labelled so, and its rule for members is the NFA's: sim and live are tracked separately, and a sim figure is never shown as a live one.

2. It is gross. Commissions, exchange fees and slippage are not rounding errors in a system that trades often; they are frequently the whole edge. The SEC's equal-prominence rule exists because gross figures were routinely shown alone. Ask for net, and ask what costs were charged. (What a backtest can and cannot tell you has the desk's: two ticks of slippage and a commission per side, on every modeled trade.)

3. Its fills are perfect. A simulation fills you at the signal price, instantly, in any size. The CFTC's advisory: in real trading "a stop-loss order might be executed at a better or worse price, or not be executed at all," and execution "could impact the price paid, especially in less liquid or illiquid markets." Interactive Brokers' education page on moving from backtest to paper trading lists what paper trading still ignores — "market impact," "price slippage," "market microstructure" — and adds that paper trading "is not a substitute for live trading." A sim result is an upper bound.

4. The window was chosen. A record that starts the month after a drawdown ended, or stops the month before one began, is true and misleading at once. Ask for the full history from the first trade, and ask what happened in the months not shown. The desk's demo shows each hand's best month and worst month as an inseparable pair, for this reason.

5. The dead are not counted. Brown, Goetzmann, Ibbotson and Ross showed in 1992 that studying only the funds that survived "gives rise to the appearance of predictability" that was never there. A seller who tested ten systems and shows you the one still alive has shown you a survivor, not a system. Malkiel and Saha measured the effect in hedge fund databases: funds that stopped reporting — usually because they failed — pulled the surviving average up by several percentage points a year.

6. The history was backfilled. The same study found a second bias, larger than the first: when a fund joins a database it adds its past returns, and funds join when their past looks good. Malkiel and Saha's working paper put backfilled returns more than five percentage points a year above the returns reported live. A system that "was running since 2019" but published since last month has a backfilled record until proven otherwise.

7. There are not enough trades. A win rate is a proportion, and a proportion from a small sample has a wide error bar: its standard error is the square root of p(1 − p) over n. For a 50% system over 20 trades that is about 11 percentage points, so the true rate is somewhere between the high thirties and the low sixties; over 100 trades, about 5 points; over 1,000, under 2. Twenty trades cannot tell a coin from an edge, and most records that look remarkable are short.

The right-hand column is the rough 95% range — two standard errors each way. Read it before any win rate.

8. The headline is the win rate. It is the number most often shown and the one that says least; the desk's own sweep of 9,363 configurations found that win rate explained 1.5% of whether a system made money. How to read a win rate is that essay. Ask for profit factor, drawdown and trade count instead.

9. The risk measure was gamed. Goetzmann, Ingersoll, Spiegel and Welch showed in 2007 that the Sharpe ratio and its relatives can be manipulated by a dynamic strategy — "a simple strategy that accomplishes this has the fund sell an out of the money option" — producing years of smooth, high-ratio returns that are nothing but the premium collected for a disaster that has not arrived yet. A record that is too smooth is a question, not an answer. Ask what the strategy does in a crash, and whether the record contains one.

10. It was the best of many tries. A result selected from hundreds of configurations is better than average by construction, and the backtest cannot tell you how many were tried. Bailey and López de Prado's deflated Sharpe ratio corrects a record for the number of trials and the shape of its returns; the honest seller tells you the number of trials unprompted. The dishonest one does not know it either.

Two more that are not lies, just omissions

The drawdown is missing. The depth of the worst fall and the months spent below the previous high decide whether anyone could have held the system through its record; they are the two figures most often left out, because they do not sell. Drawdown decides whether you survive is what to do with them when you get them.

The period is annualised. Three good months, multiplied by four, is not a year's return; it is a guess with a confident face. Any figure described as "annualised" from a record shorter than a year deserves the question of how many actual years it contains.

The ten questions, in order

1. Actual account, or hypothetical? Which parts of the record are which?
2. Net of what costs, exactly — commission, fees, and how much slippage per trade?
3. Where were the fills taken — signal prices, or a broker's statement?
4. From the first trade to the last, what did every month look like?
5. How many systems were tested, and how many are still running?
6. When did public reporting start, and what was added from before?
7. How many trades, and what is the error bar on that win rate?
8. What are the profit factor, the worst drawdown, and the time underwater?
9. What does the record do in a crash, and does it contain one?
10. How many configurations were tried before this one?

A record that answers all ten is rare. This desk tries to answer them on its own pages — the record on the demo desk carries the hypothetical label, the modeled costs, the losing months beside the winning ones and the engine's trade counts — and where it cannot yet answer one, it says so instead of changing the subject. Hold everyone else to the same list, starting with us.

• Position sizing from the stop, not the account
• Drawdown is the number that decides whether you survive
• Leverage: the arithmetic of being wrong

← Risk, the common thread · All of Learn