Ask how the final rule was chosen.
A backtest can reveal losses, costs and sensitivities. When the same record helps select the rules, it also shapes the answer. Overfitting captures sample features that fail to generalise; even a static allocation can hide many rejected trials.
The search can be more complex than the resulting portfolio
Ask how many combinations, dates and assumptions were tried and when the rule became fixed. Bailey/co-authors’ linked February 2015 manuscript assesses backtest selection; understanding the evidence does not require implementing its statistical framework.
The later winner was unavailable to the earlier decision.
Choose A at the boundary using development results only.
← Choosing B after seeing both periods brings later evidence back into selection.
A carries its own later −10%, not C’s +8%. These assign no real strategy probabilities; one ranking reversal does not prove overfitting.
Compound both intervals for each candidate
A: 1.20 × 0.90 − 1 = 8%; B: 1.15 × 1.05 − 1 = 20.75%; C: 1.10 × 1.08 − 1 = 18.8%. A sound process can disappoint; an overfit one can succeed by chance.
A reserved file can still become design input.
Out-of-sample evidence depends on use, not its label. Looking, revising rules and testing again uses the reserved period for selection. Familiar historical events can influence design before any formal test.
Evidence after publicly fixed rules needs mandate changes, costs, time and conditions stated. Launch does not guarantee independence or representativeness. Other markets can share shocks, methods and research choices.
Account for rejected versions and practical assumptions.
The strongest result needs its search history across universes, dates and assumptions, including explanations invented afterward. Multiple testing complicates interpretation without proving every strategy useless.
Check historical members, release dates, income, financing, taxes and expenses. A statistical pattern can be difficult to own; costs alone need not explain later disappointment. Keep simulated/live boundaries explicit.
False and missed discoveries are different errors
Harvey/Liu distinguish false discoveries from real effects a test misses. The search changes how to interpret its strongest result; neither uncertainty nor an appealing retrospective explanation identifies the true effect. The multiple-test illustration shows that distinction without assigning a real strategy’s probability.
A better test does not supply a recovery date.
No universal return, score or number of years settles every proposal. State purpose, search, later evidence and implementation.
Compare defined exposure, losses, expenses and attention; record what remains assumed. Better evidence can identify experiences to prepare for, not promise future drawdowns, recovery or spending outcomes.