Free resource

The Backtest-to-Production Checklist

Twelve ways your research is lying to you about live performance, and how to catch each one before capital is at risk. The same checks I run before any strategy or model is trusted with real money.

1. 1

   Point-in-time data

   Confirm every feature was actually knowable at decision time. No lookahead, no future information leaking backward.

2. 2

   Walk-forward, not a single split

   Expanding or rolling windows with out-of-sample evaluation at every step, never one in-sample fit over the full history.

3. 3

   Regime splits

   Test across distinct volatility and market regimes, not one averaged period that hides where the edge dies.

4. 4

   Purged and embargoed validation

   Remove overlapping-label and adjacent samples around each test fold so train and test do not bleed into each other.

5. 5

   Multiple-testing correction

   Discount the best-of-N search. Track how many configurations you tried before the one that looked good.

6. 6

   Execution costs modeled from day one

   Slippage, spread, latency, and market impact belong in the backtest, not in a footnote added afterward.

7. 7

   Parameter stability

   Does the edge survive small parameter perturbations, or does it only exist at one precise point in the grid?

8. 8

   Feature-leakage audit

   Hunt target leakage, train/test contamination, and the classic scaler-fit-on-all-data bug.

9. 9

   Capacity and position sizing

   Does the strategy survive realistic size, and what is the risk of ruin under honest assumptions?

10. 10

    Documented failure modes

    Write down how and when it breaks before you trust it with capital.

11. 11

    A monitored deployment path

    Paper, then small size, then full allocation, with drift and regime monitoring at each stage.

12. 12

    A kill switch

    Pre-agreed conditions under which you stop, decided before going live, not in the middle of a drawdown.