Can You Trust an AI Trading Bot's Backtest? A 10-Point Checklist

AI trading bots love to show backtests. A smooth line going up and to the right. A win rate in the 70s. A drawdown that barely shows on the chart.

Some of those backtests are honest. Many are not, and usually not on purpose. A backtest is easy to get wrong and hard to check from the outside.

This post is a checklist for the moment someone, or some AI, shows you a backtest and says "look how well this did." It works for a bot you're thinking about paying for, a strategy a chat AI wrote for you, or your own code. If you want the full testing process instead, start with how to test an AI trading strategy before you use real money.

Why AI backtests need a second look

An AI model can write a strategy, write the code, and describe a backtest in under a minute. That speed is useful. It also means it can produce a lot of convincing-looking results that nobody checked.

Three things make AI backtests extra risky:

• The numbers may not come from a real test. A chat answer can describe results it never ran on real data.
• AI makes many variations fast. Show someone the best of 200 tries and it will look great by luck alone.
• Costs get skipped. Fees, slippage and funding are boring, so they're the first thing left out.

None of that means AI ideas are bad. It means the backtest is a claim, not proof. Here's how to check the claim.

The 10-point checklist

1. Where did the data come from?

Ask which market, which exchange, which dates, and which candle size. "Bitcoin, last two years" is not enough. If nobody can tell you where the prices came from, the result can't be checked.

Also ask if the data includes coins or markets that no longer trade. Testing only on assets that survived makes almost anything look better.

2. Are fees, slippage and funding included?

This one sinks the most strategies. Ask for the fee rate used per trade, how slippage was modeled, and whether funding costs on perpetual futures were counted.

A strategy that trades many times a day can look great without costs and lose money with them. If the answer is "we'll add costs later," treat the result as unfinished. We covered the full list of costs in paper trading vs live trading.

3. Could the strategy see the future?

This is called lookahead bias. It happens when a rule uses information that wasn't known yet at the moment of the trade. A common example: using a candle's closing price to decide to buy at that same candle's open.

We think of it as point-in-time truth. For every trade, ask what the strategy knew, when it knew it, and whether you can prove it. If the backtest can't answer that, it's not reliable.

4. How many versions were tried?

If 100 versions were tested and you're looking at the best one, the result is mostly selection, not skill. Ask how many ideas, settings and tweaks were tried before this one was picked.

An honest answer sounds like "we tested 40 settings and this is the median," not "this is our best."

5. Was there data it never saw?

A backtest that was built and judged on the same data will always look good. Ask whether part of the history was held back and only used once at the end. Better still, ask if the locked rules were run forward on new data as it came in.

A backtest shows what would have worked. A forward test shows if the idea survives.

6. How many trades is it based on?

Twelve trades can't tell you much. A great result on a small sample is often noise. Look for enough trades, across more than one kind of market (rising, falling, flat), to see a pattern.

7. What did the worst stretch look like?

Ask for the biggest drop from a high point (max drawdown) and the longest losing streak. Then ask yourself if you could sit through that with real money. A chart that hides the bad months is a sales chart, not a test.

8. Does it survive small changes?

Shift the start date. Change a setting by a little. Make fees slightly worse. A real pattern should still mostly hold. If the result falls apart when one number moves, it was likely fit to the past. Our post on 7 signs of backtest overfitting goes deeper on this.

9. Is the score the full count?

"Our best strategy did great" tells you nothing. The honest version counts every strategy tested. For example: "3 of 11 strategies net positive (simulated)." The "of 11" shows the failures too, and the "simulated" makes clear it's not real money.

If someone only shows winners, ask to see the rest.

10. Are the failures still on the record?

Good research keeps what didn't work. Bad research quietly deletes it and reruns until the numbers look nice. Ask whether old results, failed versions and bad records are kept and visible, or rewritten.

We once found that some of our own trade exit records, the ones showing where a trade hit its target or its stop, couldn't be trusted. We kept them, labeled them, and refused to count them as wins. That's the bar we hold ourselves to.

A quick way to score it

Go through the 10 questions and mark each one yes, no or unknown.

• Mostly yes: worth testing further, on paper, with your own limits.
• Several unknowns: ask for the missing details before you go further.
• Any no on costs, lookahead or the full count: keep it on paper. Those three alone can make a losing idea look like a winner.

Even a backtest that passes all 10 is only the first step. It earns the right to be tested forward. It doesn't earn real money yet.

How Pickled Lizard handles this

Pickled Lizard is built around these questions. You bring your own AI, like ChatGPT, Claude, Grok or Cursor. Ideas go through the Research Factory: discover, test, challenge, forward observe, prove. Paper trades use real fees, real margin rules and real timestamps.

Ideas move up in stages: Shadow, then Unproven Paper, then Verified. Each stage has to be earned on data the idea hasn't seen. A nice backtest doesn't skip a step.

AI thinks. Pickled Lizard proves and executes. We're not building an AI that tells you what to buy. We're building the infrastructure that forces AI trading ideas to prove themselves.

Test it free

The beta is free. Bring an idea from your AI and watch it face new data on paper. You can also see simulated results on our paper trading leaderboard.

Test it free at pickledlizardz.com/join

More on how we test ideas: see all posts on Learn. Why I'm building this: read the About page.

Paper trading only. Results are simulated. Not financial advice.