The Lab

Where it comes from

• Prices that wander at random. Louis Bachelier (1900) first modelled prices as a random walk. Alfred Cowles (1933) checked the forecasts of professional market letters and found they did no better than chance. Maurice Kendall (1953) found stock prices moved with little memory.
• Why that would be. Paul Samuelson (1965) showed that if traders use what they know, a predictable move gets traded away before it happens.
• The "efficient market". Eugene Fama (1970) named three forms: prices reflect all past prices (weak), all public news (semi-strong), or all information, even private (strong). Burton Malkiel's A Random Walk Down Wall Street (1973) brought it to ordinary readers.
• The objections. Sanford Grossman and Joseph Stiglitz (1980) pointed out that if prices were perfectly efficient, nobody would be paid to gather information, so markets can't be perfectly efficient. Andrew Lo (2004) proposed "adaptive markets": edges appear, get found, and fade.
• Testing against chance. Ronald Fisher's The Design of Experiments (1935) set out the idea of comparing a result with what random rearrangements of the same data give.

The maths, step by step

1. Many tries. Test 20 rules that have no edge, each at the usual 5% bar. The chance at least one "passes":

1 − 0.95²⁰ = 64%. More likely than not.

2. Raising the bar. Ask each rule to pass at 5% ÷ 20 = 0.25% instead (the Bonferroni correction). The chance any of the 20 passes by luck falls to 4.9%.

3. A shuffle rank. Run a month's real prices and 20 shuffled copies. If the rule has no edge, the real run is equally likely to land anywhere among the 21.

• Beating all 20 by luck: 1 in 21, 4.8%.
• Beating 18 or more: 3 in 21, 14%.
• Beating 15 or more: 6 in 21, 29%. Run that on 14 coins with no edge, and about 14 × 6 ÷ 21 = 4 coins pass by luck.

4. Expected luck. Test 576 settings, each needing to beat 19 or 20 of 20 shuffles (a 2-in-21 chance by luck): about 55 pass with no edge at all.

5. The best of many. With no skill at all, the best of many tries still looks good. In units of how much the tries' scores spread:

• the best of 10 is about 1.57;
• the best of 112 is about 2.57.

A winner has to beat that, not zero.

What others found

This section is other people's work, not ours.

• Most professional forecasters don't beat chance, from Cowles (1933) on.
• Many published patterns are luck. Campbell Harvey, Yan Liu and Heqing Zhu (2016) counted hundreds of published stock "factors" and argued that a new one should clear a much higher bar (a t-statistic of about 3, not 2).
• Backtests flatter. David Bailey, Jonathan Borwein, Marcos López de Prado and Jim Zhu (2014) showed how trying many settings on one history produces winners that fail on new data. Halbert White (2000) gave a test that judges the best of many rules against what luck would give.
• But markets aren't perfectly efficient. Grossman and Stiglitz's argument, and documented patterns such as momentum, mean that some edges exist. They tend to be small, costly to trade, and to fade.

What we tested and found

This section is about how our own tests use this, with examples.

Our trading studies set random twins and a pass bar before the run. Here is what they caught:

• Buying sharp drops made +0.26% a trade on Bitcoin. Buying at the same hour on a nearby day made +0.12%: about half the "edge" was simply the market rising. A same-day twin we had also set would have passed it every time, because a random hour earlier that day has the crash in its next 24 hours. So the choice of twin matters.
• Momentum had a good score in its first years, but random books from the same coins often scored as well: it beat the best of 16 random books in only 67% of draws.
• A tuned stop-and-target bot did well in the three years it was tuned on, and lost on all three coins in the year held back.
• The time of day: the best hour scored 2.80 against a raised bar of 3.19 (step 2's correction for 35 tries).
• Grid bots beat at least 15 of their 20 shuffled twins on 2 to 4 of 14 coins. At that bar luck alone gives about 4 (step 3), so this is no more than luck.
• Selling slices of Bitcoin high beat holding, but beat only 87% of the same sales on random weeks, against a 95% bar.

Most of our studies found no edge once judged against their twins. That agrees with the published work: an honest test makes "no" the usual answer.

The catch

• A random twin can be unfair. It has to share everything with the real rule except the idea being tested, such as the same market, the same hours and the same costs. Get that wrong and the test is too easy or too hard.
• Shuffling breaks some things you want to keep. Shuffled prices lose their trends and their calm-and-wild spells. A rule that lives on those may look better or worse than it is.
• Passing isn't proof. A result that clears the bar can still be luck, just less likely. It needs a fresh test on data it hasn't seen.
• One long path. A market's multi-year shape happens once. No shuffle can tell whether that shape was luck.

Credits

• Louis Bachelier, "Théorie de la spéculation", Annales scientifiques de l'École Normale Supérieure, 1900.
• Alfred Cowles 3rd, "Can Stock Market Forecasters Forecast?", Econometrica, 1933.
• Maurice G. Kendall, "The Analysis of Economic Time-Series, Part I: Prices", Journal of the Royal Statistical Society, 1953.
• Paul A. Samuelson, "Proof That Properly Anticipated Prices Fluctuate Randomly", Industrial Management Review, 1965.
• Eugene F. Fama, "Efficient Capital Markets: A Review of Theory and Empirical Work", Journal of Finance, 1970.
• Harry V. Roberts, an unpublished paper given at the University of Chicago, 1967: the names of the three forms, as Fama 1970 credits.
• Burton G. Malkiel, A Random Walk Down Wall Street, W. W. Norton, 1973.
• Sanford J. Grossman and Joseph E. Stiglitz, "On the Impossibility of Informationally Efficient Markets", American Economic Review, 1980.
• Andrew W. Lo, "The Adaptive Markets Hypothesis", Journal of Portfolio Management, 2004.
• R. A. Fisher, The Design of Experiments, Oliver and Boyd, 1935.
• Carlo Emilio Bonferroni, Teoria statistica delle classi e calcolo delle probabilità, 1936; Olive Jean Dunn, "Multiple Comparisons Among Means", Journal of the American Statistical Association, 1961.
• Halbert White, "A Reality Check for Data Snooping", Econometrica, 2000.
• David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu, "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", Notices of the American Mathematical Society, 2014.
  • Overfitting is tuning a rule until it fits the past's noise rather than a real pattern, so it fails on new prices. For example, of 100 settings with no skill, the best on last year's prices will usually look good by luck, and do no better than the rest the year after.
• David H. Bailey and Marcos López de Prado, "The Deflated Sharpe Ratio", Journal of Portfolio Management, 2014: step 5's formula.
• Campbell R. Harvey, Yan Liu and Heqing Zhu, "… and the Cross-Section of Expected Returns", Review of Financial Studies, 2016.
• Our tests, as named above: "Buying a sharp drop: does the market bounce the next day?", "Momentum across coins: buy the strongest, sell the weakest", "Does a '2% stop, 4% target' bot beat every strategy?", "Do coins have a clock? Hours, weekdays, funding times, the US open", "Grid bots on 14 coins: what 30 months of prices say", and "Ending up with more Bitcoin, without adding money".
• PerpGremlin and Claude, Hard-to-beat markets, and why every test needs a random twin, TerminalGremlin, 2026: the tests, the code and the words.