Backtest Overfitting: 7 Signs Your Strategy Only Works on the Past

You build a strategy. The backtest looks great. You run it forward and it falls apart in a week.

That's usually overfitting, also called curve fitting. The rules learned the noise in your old data instead of a real pattern. They fit the past like a key cut for one lock, and the next lock is different.

An overfit backtest looks exactly like a good one, so don't trust the chart. Look for the signs. Here are seven we watch for at Pickled Lizard, with a quick test for each.

For the step-by-step on running an honest backtest, see how to backtest a trading strategy without fooling yourself. This post is about reading the result.

Sign 1: The chart is too smooth

Real strategies have bad weeks, flat stretches, and losing streaks. If your equity curve climbs in a near-straight line with tiny dips, be suspicious, not happy.

Markets are noisy. A rule that trades a noisy market and gets a clean result has often been tuned until the noise lined up.

Quick test: look at the worst month and the longest losing streak. If both are tiny for how often it trades, the rules were probably tuned around every bad patch.

Sign 2: Lots of settings, few trades

Every setting you can tune, like an average length, a stop distance, or a time-of-day filter, is one more chance to fit noise.

Now compare that to how many trades you have. Eight settings and 40 trades is a lot of freedom for very little evidence.

Quick test: count your tunable settings. Count your trades. If you can't explain why each setting is there before you saw the results, it's probably fitting, not finding.

Sign 3: The best setting sits on a narrow peak

Say your strategy works best with a 21-period average. What happens at 19, 20, 22, 23?

If the result is decent across the whole range, you may have found something real. If 21 is great and its neighbors are bad, you found a lucky number. Real patterns are a broad hill. Overfit ones are a single spike.

Quick test: run a small grid around every main setting and look at the neighbors, not just the winner. Pick a setting from the middle of a good area, not the top of a spike.

Sign 4: It falls apart on data it hasn't seen

This is the clearest sign. You held back a later stretch of data, you ran the locked rules on it once, and the result collapsed.

There's a quieter version too. You checked the held-back data, it looked bad, you tweaked the rules, and you checked again. After a few rounds it passes. That data isn't held back anymore. You trained on it.

Quick test: be honest about how many times you looked at your test period. If the answer is more than once, treat it as part of the training data and get fresh data, or test forward on paper.

Sign 5: It only worked in one kind of market

A trend-following rule tested mostly in a strong trend will look brilliant. That's not a strategy yet. It's one market mood.

Quick test: split the results by market conditions, for example trending, choppy, and fast-moving. Then try the same rules on a similar market. If almost all the gains came from one stretch or one coin, you've tested a moment, not an idea.

Some strategies really do only work in certain conditions. That's fine, but then the rule for when to turn it on and off is part of the strategy, and it needs testing too.

Sign 6: The rules have oddly specific exceptions

"Buy the breakout, except on Mondays, unless volume is above the 30-day average, and skip the first hour."

Each exception made the backtest better. Together they often mean the rules were shaped to dodge specific bad trades in the past.

Quick test: for each filter or exception, ask whether you would have written it before seeing any results. Then remove it and rerun. If one exception carries most of the result, it's carrying luck.

Sign 7: It was the best of many tries, and you lost count

If you test 50 versions and keep the winner, the winner will look good even if none of them work at all. Something has to come out on top.

AI makes this easy to do by accident. Ask ChatGPT, Claude, or Grok for 20 variations, test them all, and you remember only the one that worked.

Quick test: write down every version you tried. Report the full count with the result, like "2 of 30 versions net positive (simulated)." If the count is big and the winners are few, the winners are probably noise.

What to do when you see the signs

Seeing one sign doesn't mean the idea is dead. Seeing three usually does. Here's what helps:

• Cut settings. Fewer knobs, set for a reason, beat many knobs set by the data.
• Pick from the middle. Use settings from a broad good area, not the single best spot.
• Make costs worse on purpose. Add a bit more fee and slippage than you expect. Weak ideas fall apart first.
• Get new data. A later period, a similar market, or a different time frame.
• Test forward on paper. Lock the rules, run them on live data they've never seen, and change nothing while it runs. See how to test an AI trading strategy before you use real money.
• Try to break it. The goal of testing isn't to make the idea pass. It's to find out if it fails.

How Pickled Lizard handles overfitting

At Pickled Lizard, a good backtest is a starting point, not proof. In practice:

• An idea goes through stages: discover, test, challenge, observe forward, prove. A strong backtest doesn't skip any of them.
• We can run many trade-management versions on the same entries in a shadow lab, and every version is counted, not just the winner.
• Paper trades use real fees, margin rules, and timestamps, and strategies only see data that existed at the time.
• Failed tests stay in the record next to the passing ones. We don't rewrite history.
• Results are shown one way: "X of N net positive (simulated)."

We're not building an AI that tells you what to buy. We're building the infrastructure that forces AI trading ideas to prove themselves.

A quick overfitting checklist

Before you trust a strategy, ask:

• Does the chart have normal bad patches, or is it suspiciously smooth?
• Do I have far more trades than settings?
• Do nearby settings work too, or only one exact number?
• Did I test held-back data once, and only once?
• Does it hold up in more than one kind of market?
• Would I have written every filter before seeing results?
• Do I know how many versions I tried?

If several answers are no, the strategy probably only works on the past.

Try it free

The beta is free. We're looking for our first 1,000 testers, the Founding 1,000. Bring a strategy, from you or your AI, and run it forward on paper with real fees and clean timestamps. You can see simulated results on our paper trading leaderboard.

Join the Founding 1,000

More plain guides: see all posts on Learn. Why I'm building this: read the About page.

Paper trading only. Results are simulated. Not financial advice.