A published flag may hide twenty tests.
Search before publication
20 independent tests → inspect their results
If a flag occurs: select one for publication.
If none occurs: there is no flag to select.
Chance of at least one false flag
About 64.2% produce a flag; 35.8% produce none. Expected count: 1 flag per experiment, not one guaranteed flag.
This is not the probability that an actual factor is false. A 5% threshold does not mean a claim has a 95% chance of being true. Real tests have different error properties; weak evidence does not establish absence.
False flags, expected counts and missed discoveries
No flags: 0.95²⁰ ≈ 35.8%; at least one: 1 − 0.95²⁰ ≈ 64.2%. Expected count: 20 × 5% = 1. None identifies an individual false result.
Changing dates, definitions or companies can hide searches. Harvey/Liu study false and missed discoveries: genuine effects can go undetected. Real tests may be highly correlated.
The information must exist before the decision.
- 31 December 2019Company accounts close.
- 20 March 2020Accounts become public.
- 1 April 2020April’s portfolio can use these accounts; January’s could not.
- End of 2022Researcher fixes and publishes the rule after studying 2010–2022.
- 2023–2025Later data test that unchanged rule.
Retuning on 2023–2025 uses up that test: the revised rule needs fresh evidence.
Later revisions and a survivor list learned afterwards also introduce unavailable knowledge. Separate original-sample, other-period or market, post-publication and live-fund results.
Different providers can reuse history, methods and conditions. Several charts are not necessarily separate tests.
Inspect samples and later evidence separately
Check selection in the original sample, comparability across settings, and changed conditions and record length after publication. Live funds additionally reveal mandate, expenses, exposure and benchmark.
McLean and Pontiff distinguish weaker later and post-publication returns from original-sample results: reason to inspect, not a universal discount or fund forecast.
What could an investor actually keep?
Identify long-only, long–short or exposure-adjusted results. Fund securities, financing and tilt can differ. Include expenses, transactions and applicable tax; check index launch date, back-calculated history, currency and income treatment.
Illiquid or expensive securities can weaken implementation. Trading/concentration limits may reasonably produce a milder tilt, changing what is being tested. The SEC’s investor bulletin distinguishes hypothetical from actual performance.
Record what remains uncertain.
Record the definition, evidence, household risks and added exposure; research disagreements may persist.
A precise premium or promised recovery date remains an unsupported planning dependency. Apply the burden and simplification decision; choosing less complexity need not prove factors can never work.