How to Backtest a Trading Strategy Without Fooling Yourself

A backtest is easy to run. You take a set of trading rules, play them over past prices, and look at the result. Most tools will do it in a few clicks.

The hard part is trusting what you see. A backtest almost never lies on purpose. It lies quietly, through small choices that all lean the same way: toward a nicer chart.

This post is a plain list of the ways a backtest fools people, and the habits we use at Pickled Lizard to stop it. You can use them anywhere.

What a backtest can and can't tell you

A backtest answers one question: if I had followed these rules in the past, what would have happened?

That's useful. It kills bad ideas fast and for free. But it can't tell you what will happen next. Markets change. A rule that worked in one kind of market can fail in another.

So think of a backtest as a filter, not a promise. Its job is to throw out ideas, not to crown winners.

Mistake 1: Fuzzy rules

If your rules aren't exact, you will "help" the strategy without noticing. You'll skip a trade that looks silly, or take one that "obviously" fits.

Write every rule as a plain sentence before you test. When does it enter? When does it exit? How big is each trade? Which market and time frame? Then lock that version and give it a name or a date. If you change a rule later, that's a new version, not the same strategy.

Mistake 2: Peeking at the future

This is the most common way a backtest cheats. It's called look-ahead bias. The test uses information that wasn't known yet at the moment of the trade.

A simple example: a rule says "buy if today closes green," and the test buys at today's open. In real life you don't know the close until the day ends.

It also hides in data. Prices that were corrected later, indicators that use the full bar, or data stamped with the time it's "about" instead of the time it arrived.

The fix is what we call point-in-time truth. For each trade, ask: what did the strategy know, when did it know it, and can I prove it? If you can't answer, don't trust the trade.

Mistake 3: Leaving out the boring costs

A strategy that trades a lot can look great before costs and lose money after them. Put these in from the start:

• Fees you would really pay, on every entry and exit.
• Slippage, the gap between the price you wanted and the price you got.
• Funding, if you hold perpetual futures.
• Missed fills. A limit order in a backtest that "touched" the price may not have filled in real life.

If a strategy only works with zero costs, it doesn't work.

Mistake 4: Testing on all your data

If you build and tune your rules on all the history you have, you have nothing left to check them on. Every tweak quietly fits the past a little better.

Split your data before you start:

• In-sample: the part you use to build and tune the idea.
• Out-of-sample: a later part you don't touch until the rules are locked.

Run the locked rules on the out-of-sample part once. If the result falls apart, the idea probably fit noise. Don't go back and tweak until it passes. That turns your test data into training data.

Mistake 5: Forgetting how many things you tried

Say you test 100 versions of an idea and keep the best one. That best one will look good, even if every version is random. You picked the luckiest, not the best.

This is the trap AI tools make worse. ChatGPT, Claude, or Grok can write 50 strategies in a minute. If you test them all and share the top one, the chart means very little.

Keep a simple log of every version you tested, including the ones that failed. When you report results, use the full count. For example: "3 of 40 versions net positive (simulated)." The "of 40" is the honest part.

Mistake 6: Too few trades, one kind of market

Ten trades is a story, not a test. You need enough trades that a few lucky ones can't carry the result.

Also check when the trades happened. If almost all of them came from one strong trend, you've tested one market mood, not a strategy. Look at how it did in quiet markets, choppy markets, and sharp moves, separately.

Mistake 7: Rules that break when you nudge them

A real pattern should hold up, more or less, when you change small things. Try these:

• Move the start date by a few weeks.
• Change a setting a little, like a 20-period average to 18 or 22.
• Make costs a bit worse.
• Run it on a similar market.

If one small change wrecks the result, the original setting was probably luck.

After the backtest: test it forward

Even an honest backtest only shows what would have worked. The next step is to run the locked rules forward, on paper, on new data the idea has never seen. No changes allowed while it runs.

We go deeper on this in how to test an AI trading strategy before you use real money.

How Pickled Lizard handles this

We built Pickled Lizard because these mistakes are easy to make and hard to see. A few things we do by default:

• Paper trades use real fees, margin rules, and timestamps.
• Market data is stored with the time it actually arrived, so strategies only see the past.
• We never rewrite records. Failed tests stay in the log, next to the ones that passed.
• Ideas move up in stages, from Shadow to Unproven Paper to Verified. A nice backtest alone doesn't skip a step.
• Results are shown one way: "X of N net positive (simulated)."

We're not building an AI that tells you what to buy. We're building the infrastructure that forces AI trading ideas to prove themselves.

A quick checklist

Before you trust a backtest, ask:

• Are the rules written down and locked?
• Could any rule have used data that wasn't known yet?
• Are fees, slippage, funding, and missed fills included?
• Did I keep a held-back period and test it only once?
• Did I count every version I tried?
• Are there enough trades, across more than one kind of market?
• Does it survive small nudges?

If any answer is no, it's not ready for real money.

Try it free

The beta is free. We're looking for our first 1,000 testers, the Founding 1,000. Bring a strategy, from you or your AI, and see how it does on paper with real fees and clean timestamps. You can also check simulated results on our paper trading leaderboard.

Join the Founding 1,000

More plain guides: see all posts on Learn. Why I'm building this: read the About page.

Paper trading only. Results are simulated. Not financial advice.