Backtesting honestly: overfitting, lookahead and survivorship

You have an equity curve that goes up and to the right. Sharpe above 2, a maximum drawdown you could live with, a few hundred trades. It took a weekend of tuning and the result feels earned.

It is probably worthless, for mechanical reasons rather than any failure of skill. A backtest does not test a strategy. It measures how well a rule describes a dataset you have already seen. Those are different claims, and the second supports no conclusion about money. If you have a strategy you are proud of, read the rest with that strategy in mind.

Overfitting is not sloppiness, it is a counting problem

The standard framing is that overfitting happens when you add too many parameters. That is part of it. The larger part is how many times you looked.

Every test you run against the same dataset is a trial. Change a lookback and rerun: trial. Add a filter and rerun: trial. Try a different asset because the first was disappointing: trial. After 200 trials on the same history, the best-looking result is what you would expect from noise alone, whether or not any edge exists.

At a 5% significance threshold, 200 independent trials produce roughly 10 spuriously "significant" results by chance. You then select the best one, which is precisely the selection procedure that guarantees you pick a false positive.

The trial count almost never appears in anyone's writeup, and it should be the first number reported. Log every configuration you test, including abandoned ones, and treat the final result as one draw from that whole search.

And the count is bigger than your own experiments. If you are testing a moving average crossover on the S&P, you are not the first. Thousands of people have tested variants of that idea on the same history, and the ones that failed were never published. You inherited a search you did not run.

Lookahead bias hides in ordinary code

Lookahead is using information at time t that was not available at time t. It is easy to state and constantly violated.

Using the closing price as both signal and fill. Your rule triggers on today's close and you fill at today's close. In live trading you cannot know the close until it has happened. The gap is small on liquid daily bars and enormous on anything intraday or thin.

Restated data. Economic figures get revised. A database showing the final revised value on the original date lets your rule use a number that did not exist for weeks.

Indicators computed over the full series. Any normalization, ranking or z-score computed across the whole history injects future information into every early bar.

The symptom is a suspiciously smooth equity curve with a high win rate and shallow drawdowns. That shape usually means the system is being handed answers rather than making predictions.

Survivorship is a property of your data, not your code

If your universe is today's list of assets, you have already excluded everything that died. Delisted stocks, failed tokens, collapsed exchanges, pairs that stopped trading. Every one of them was in the investable universe at the time and is missing from your test.

In crypto this is severe. The set of tokens with a full multi-year price history is a heavily selected sample - the ones that survived. A strategy tested only on survivors looks robust because bankruptcy was removed from the sample.

Costs are not a rounding error

Frictions get added last, if at all, and they change conclusions more than any parameter. Include the spread you actually pay rather than the quoted mid, slippage that scales with volatility and size, and commissions, funding or swap on held positions.

A system that survives at zero cost and dies at realistic cost was never a system. It was a description of the mid-price. Double your assumed frictions and see what remains. If the edge disappears, it was inside the spread.

What an honest test looks like

Split the data before you start, and do not touch the holdout. Develop on one segment, then run once on data you have never examined. Once. If you go back and adjust after seeing that result, the holdout is contaminated and you no longer have one.

Prefer walk-forward. Fit on a window, trade the next window, roll forward. It approximates how you would really have operated, and it exposes strategies that only worked in one regime.

Report the trial count, the frictions, and the parameter sensitivity. A result that survives a 10% change in every parameter is worth more than a better result that does not.

Then expect live performance to be worse anyway. Some degradation is normal even with clean methodology, because markets change and because you cannot fully eliminate your own selection.

This is why forward records matter more than backtests. Indikora hashes every published signal with SHA-256 and chains it to the previous one, so the public track record cannot be edited or back-dated, and it publishes probability calibration - when it says 60%, you can check how often that was right. Neither is possible with a backtest, which is written after the fact by definition.

The strategy you tuned over a weekend may still be fine. But you do not know that yet, and the honest position is to say so out loud before sizing anything to it.