The Lab

Where the idea comes from

• The problem. Try enough rules on one history and the best will look good by luck. Our theory page "Hard-to-beat markets, and why every test needs a random twin" works the arithmetic for tries that are independent of each other. Real rules are related (a 4-week and an 8-week rule see much the same prices), and then that arithmetic can't say how big the luck is.
• The test. Halbert White, "A Reality Check for Data Snooping" (Econometrica, 2000). Resample the history many times, and each time ask how good the best of all the rules would look if none had an edge. Because every rule is resampled on the same days, their relatedness is kept.
• On trading rules. Ryan Sullivan, Allan Timmermann and Halbert White (Journal of Finance, 1999) ran it on 7,846 technical trading rules on the Dow Jones over a century. The best rule passed on its own history, then did not hold up on the decade after it.
• The resampling keeps runs of days together, so calm and wild spells survive: Dimitris Politis and Joseph Romano's stationary bootstrap (1994).
• Part of our series "Published edges against the coin".

How we tested

• The three searches:
  • Session momentum (our page "Session momentum: do the first hours predict the last?"): all 648 rules, day by day, four years. Each day counts what the rule's side added over a coin-flip side, after the market-order fee.
  • Trend over months (our page "Trend over months: does a coin's own past point its way?"): the 10 futures rules, week by week, 2021 to 2026. Each week counts the rule's result less always long's, since the papers claim to beat simply holding.
  • Momentum across coins (our page "Momentum across coins: buy the strongest, sell the weakest"): its 16 books, week by week, 2021 to 2026, after fees and funding. These books hold as much long as short, so no edge means 0.
• Judged alone: the best rule's chance of doing that well by luck, as if it were the only rule tried.
• The reality check: 1,000 resamples of the days (in runs of about 10 days, or 4 weeks), the same resample for every rule. Each time, the best of all the rules if none had an edge. The chance is the share of resamples where that luck did as well as our best.
• The bar, fixed before the run: a search's best survives if the reality check gives a chance under 5%.

The maths, step by step

1. One rule. A rule averages +0.1% a day over 1,459 days, and days swing about 2%. The uncertainty of the average is 2% ÷ √1,459 = 0.052%, so it is 1.9 uncertainties above 0. Alone, luck does that about 3% of the time.

2. 648 rules. If 648 rules had no edge and were independent, the best of them would be expected around 3 uncertainties above 0 by luck alone. Our theory page's step 5 shows how that bar rises with the number of tries.

3. Related rules. Our 648 rules are not independent: many trade the same days on the same side. The real bar sits somewhere between 2 and 3, and only resampling all of them together finds it. That is White's test.

4. Resampling. Pick a random day, copy a run of about 10 days from there, and repeat until there are 1,459 days. Every rule gets the same days. Subtract each rule's true average, so that none has an edge, and see how good the best one looks.

What we found

The chance that luck does as well. Under 5% passes.

1. Session momentum: luck. Its best rule looked like a 1-in-200 shot on its own. Allowing for the 647 others, luck did as well 78% of the time.

2. Trend over months: luck. Against always long, even the best of the 10 was ordinary.

3. Momentum across coins: it passed, but only just, and only with its last year. The best of the 16 books was the one that page had chosen in advance (the top 20, ranked by 4 weeks). Allowing for all 16, luck did as well 2.5% of the time. We then checked two things, after the look:

• without the last year (only the weeks the setting was picked on): 13%, luck;
• without funding (price moves only): 14%, luck.

So it passes only with the last year and the funding together. That is the year when a few squeezed coins paid the book large funding while moving 90% or more a week, as that page says.

Credits

• Halbert White, "A Reality Check for Data Snooping", Econometrica, 2000: the test.
• Ryan Sullivan, Allan Timmermann and Halbert White, "Data-Snooping, Technical Trading Rule Performance, and the Bootstrap", Journal of Finance, 1999: the test on a century of trading rules.
• Dimitris N. Politis and Joseph P. Romano, "The Stationary Bootstrap", Journal of the American Statistical Association, 1994: the resampling.
• Our own: the three searches' series and benchmarks, the bar, and the two after-the-look checks.
• Data: as on the three pages: Bybit's 1-hour USDT perpetual prices; Binance's public futures prices and funding.
• PerpGremlin and Claude, The best of many tries: our own results through White's reality check, TerminalGremlin, 2026: the tests, the code and the words.