Overfitting in trading backtests: we tested 1,000 random rules
We picked the best of 1,000 random Bitcoin rules: +155.22% after fees in training, then −51.72% on later dates. Here is what overfitting looks like.
Overfitting in a trading backtest means you pick a rule because it fit old prices well, but much of that fit was luck. We made 1,000 random rules for hourly Bitcoin prices on Binance and kept the one with the best score on 2019 to 2022. It gained 155.22% after fees in that stretch, then lost 51.72% after fees from January 2023 to September 2026, with no changes to the rule. Its Sharpe score fell from 0.75 to −0.19.
A backtest checks how a trading rule would have done on past prices. We used Binance BTCUSDT perpetual futures, or perps: contracts that follow Bitcoin's price and never expire. Everything here is a simulation of the past.
For a method that tests on unseen dates one step at a time, see Walk-forward analysis explained.
How can a random rule look good on old prices?
Each rule looked back a random number of hours, from 1 to 240, and checked how much the price had moved. If the move was bigger than a random threshold (between 0.01% and 5.00%), the rule took a position. If not, it stayed out.
Each rule had an even chance of following momentum (betting the move would continue) or mean reversion (betting it would turn around). A long position gains when price rises. A short position gains when price falls.
We split the history in two:
- Training period: the dates used to pick a winner (September 2019 through 2022).
- Holdout period: dates hidden until the pick was final (January 2023 through September 2026).
We ranked the rules by Sharpe ratio in training. Sharpe compares the average hourly gain or loss after fees with how much those results swing, scaled to a year. Higher is better. It does not show the deepest loss.
When you try 1,000 rules, some will fit a lucky stretch just by chance. Showing only the winner hides the 999 other tries that made finding it likely.
What happened to the winning Bitcoin rule?
The winner looked at the price change over the last 179 hours (about a week) and followed it. It was in the market almost all the time, long or short. We froze it before looking at the holdout.
Training had 29,046 hours of data. The holdout had 32,856 hours. All times are in UTC, the world standard clock.
| Measure (Binance BTCUSDT perp, hourly) | Training, Sep 2019–2022 | Holdout, 2023–Sep 2026 |
|---|---|---|
| Sharpe ratio, after fees | 0.75 | −0.19 |
| Total return before fees | +544.51% | +42.96% |
| Total return after fees | +155.22% | −51.72% |
| Largest drop from a past high | −60.22% | −76.83% |
"Total return" here is compounded: each hour's gain or loss changes the amount used in the next hour. The largest drop from a past high is called the maximum drawdown; our drawdown study shows how deep they got for Bitcoin itself.
Three things stand out:
- Fees flipped the holdout. Before fees, the rule gained 42.96%. After fees, it lost 51.72%. It changed position so often that it paid the 0.05% fee 2,170 times, which adds up to 108.5% of the position size.
- Even the winner was painful. In training, the account fell 60.22% from a high at one point before ending up.
- The holdout was worse in every row. Lower Sharpe, lower return and a deeper drop.
What this test cannot tell us: how much of the drop came from luck in picking the rule and how much came from the market changing after 2022. Both can happen at once.
Did one year make the whole training gain?
Mostly, yes. Here is the same frozen rule, year by year. Each year is tested on its own, starting and ending out of the market.
| Year (UTC) | Hours | Return after fees |
|---|---|---|
| 2019 (from Sep 8) | 2,742 | −5.89% |
| 2020 | 8,784 | +135.72% |
| 2021 | 8,760 | +4.80% |
| 2022 | 8,760 | +9.44% |
| 2023 (holdout) | 8,760 | +2.37% |
| 2024 (holdout) | 8,784 | −18.26% |
| 2025 (holdout) | 8,760 | −50.57% |
| 2026 to Sep (holdout) | 6,552 | +16.38% |
In training, 2020 alone returned 135.72% after fees. The other training years were small. So the "best rule" was close to "the rule that fit 2020 best". The holdout had winning years too (2023 and 2026), but they did not make up for 2024 and 2025.
Do not add these yearly numbers together: gains and losses compound, and each yearly test pays its own entry and exit fees.
Was the winner unusual among the 1,000 rules?
Yes. Most random rules lost money, even in training:
| All 1,000 rules | Training | Holdout |
|---|---|---|
| Median return after fees | −90.23% | −83.26% |
| Best return after fees | +155.22% | +1.62% |
| Share with a positive Sharpe | 24.50% | 1.00% |
The median is the middle result: half the rules did better, half did worse. We did not split out the fees for every rule.
The best holdout return, +1.62%, came from a rule we could only spot after seeing the holdout. Picking it then would be cheating, so we did not.
Another sign of overfitting: the top 10 training rules were not 10 different ideas. All 10 followed momentum, and all looked back between 175 and 240 hours. They were close copies of one idea that worked in this stretch. All 10 lost between 32.08% and 57.47% after fees in the holdout.
One more detail. A rule's training Sharpe and its holdout Sharpe were linked (a correlation of 0.84 across the 1,000 rules), so better-ranked rules tended to stay better-ranked. But 83.60% of rules scored lower in the holdout than in training. Ranking well against other rules is not the same as making money.
This study does not prove every strategy must fail. It shows that, in this test, picking the best training result out of 1,000 did not give a gain on later dates after fees.
How we measured this
We used 61,903 hourly price bars of Binance BTCUSDT perpetual contracts, from September 8, 2019 to the end of September 2026, with no missing hours. Each decision used only hours that had already closed. The rule put the full account into each position, entering at the previous hour's closing price. We charged 0.05% each time a position opened or closed, including at the start and end of each test. We left out funding payments, spread, slippage and delays, so real results could differ. We tested only Bitcoin, a coin that still trades today.
How can you check your own backtest for overfitting?
A checklist for any rule, from any data source:
- Hide a holdout first. Pick a date. Do not look at anything after it until your rule, score, costs and settings are written down.
- Count your tries. Write down every version you tested, including changes to chart length and dates. More tries means more luck in the winner.
- Split the result by year. If one year makes most of the gain, like 2020 did here, your rule may just fit that year.
- Look at your top 10, not only your top 1. If they are near copies of each other, you found one idea, not ten.
- Show before-fee and after-fee results side by side.
- Test once. If you change the rule after seeing the holdout, those dates are used up. You need new untouched dates.
A separate mistake is letting a rule see prices it would not have had yet. Our look-ahead bias example shows how that one bug fakes a winner. The same care applies to chart matching: our chart look-alike test checks matches on later dates, and our chart-length test looks at one more setting that is easy to over-tune.
Try this in Comparand
You can run a fixed holdout test on Hyperliquid data. Choose your test times first, then ask your AI agent: "For these Bitcoin times on Hyperliquid, find the closest past cases in one batch, with what happened over the next hour. Include the source record for each answer. Then list the daily archive files for these dates and give me download links."
What comes back, per the docs:
- Batch search (hl_analogs_batch): up to 100 searches in one call. Each compares Bitcoin only with its own Hyperliquid past, with the outcome over the next 1 hour or 4 hours. It costs $0.01 per search that runs, from credits. Refused queries are not charged.
- Source record (the prov block): every answer lists the data days, digests (codes that identify the exact data) and code versions behind it. Save it with your results so you can check later what your test used.
- Whole files (hl_bulk_files and hl_bulk_link): a free list of daily archive files with sizes and sha256 codes, then download links that last 24 hours and allow 5 downloads each. Plans include 40, 200 or 800 GB of links a month (Starter, Pro, Desk). Pay-as-you-go pays $0.15 per GB.
These are past cases, not a forecast. You can try 3 coins free.
The short version
- The best of 1,000 random Bitcoin rules gained 155.22% after fees in training (Sep 2019–2022), then lost 51.72% on 2023–Sep 2026.
- Most of its training gain came from one year, 2020.
- Most random rules lost money even in training: the median rule lost 90.23% after fees.
- The top 10 rules were near copies of one idea, and all 10 lost in the holdout.
- Hide a holdout, count your tries, split by year, include fees, and test once.
Questions people ask
What is overfitting in a trading backtest?
Overfitting means picking a trading rule because it fit old prices well, when part of that fit was luck. When you try many versions and keep the best one, the winner often does worse on prices it has not seen.
How did the best of 1,000 random Bitcoin rules do on later dates?
On hourly Binance BTCUSDT perpetual prices, the rule with the best training score gained 155.22% after fees from September 2019 through 2022. Kept unchanged, it lost 51.72% after fees from January 2023 through September 2026.
Does a positive Sharpe score mean a strategy made money?
No. Sharpe compares the average hourly result with how much results swing. In our Bitcoin test, one rule had a slightly positive Sharpe on later dates but still lost 32.08% after fees, because losses and gains build on each other.
Can I change my rule after seeing the holdout results?
You can, but then those dates become part of building the rule. To test the new version fairly, you need another stretch of dates you have not looked at yet.
How can I tell if my backtest is overfit?
Hide a later stretch of dates before you start, pick your rule without looking at it, then test once. Also check whether one year made most of the gain, whether your top rules are near copies of each other, and whether the gain survives fees.