Backtests are hypotheses
A backtest answers one question: how would this rule set have behaved against this data under these modelling assumptions. That is a useful question. It is not the same question as how the rule set will behave against a live broker feed with real spread, real slippage and real rejection behaviour.
The honest way to use a backtest is as a falsification tool. If a strategy fails in simulation, it will certainly fail live. If it succeeds in simulation, you have learned only that it has not yet been ruled out.
The four sources of divergence
- —Spread modelling. Most tests use an average spread. Gold's spread widens exactly when your strategy is most active.
- —Slippage. Simulations fill at the requested price; live markets fill at the available one, and the difference is asymmetric against you.
- —Swap and commission drift. Financing costs change over time; historical tests apply today's figures to yesterday's conditions.
- —Data quality. Tick data from one source is not tick data from your broker, and gold in particular shows meaningful venue variation.
Overfitting has a smell
A strategy with fourteen parameters tuned to the last three years is not a strategy, it is a description of the last three years. You can usually detect overfitting without any statistical machinery: the equity curve is unnaturally smooth, performance collapses when a single parameter moves 10%, and the rules have oddly specific values that no one could justify from first principles.
What a realistic haircut looks like
In our own testing across gold, the honest expectation is that live results land somewhere between 60% and 85% of a properly modelled backtest once real execution costs are absorbed. If a vendor's live results match their backtest exactly, that is not reassurance — it is a strong signal that one of the two numbers is not what it claims to be.
Walk-forward analysis, and why in-sample results are marketing
Any parameter set can be made to look excellent on the data used to select it. The only structurally honest test is to fit on one window, evaluate on the next window the fitting never saw, then roll forward and repeat. The resulting out-of-sample curve is usually far less impressive than the optimised one — and it is the only curve worth showing anyone.
How to run it without fooling yourself
- —Split the history into consecutive in-sample and out-of-sample windows before you look at anything.
- —Re-fit only on the in-sample window, then evaluate untouched on the following window.
- —Aggregate every out-of-sample segment into one continuous curve.
- —If the out-of-sample degradation exceeds roughly a third, the parameters are fitted to noise.
Modelling costs that testers quietly ignore
Most retail backtests assume a fixed spread, zero commission, instant fills and no swap. Every one of those assumptions is generous, and together they can turn a modestly profitable system into a modestly unprofitable one. The correct approach is to be pessimistic on purpose: model the spread you see at the worst hour of your trading window, not the average.
A deliberately harsh cost model
If the strategy still clears a sensible margin under that model, you have something. If it only works with perfect fills and average spreads, you have a report, not a system.
Everything discussed here is applied on a public, third-party verified account — updated continuously, losing weeks included.