How to Test an AI Trading Strategy Before You Use Real Money

Ask ChatGPT, Claude, or Grok for a trading strategy and you'll get one in seconds. It will sound smart. It will have rules, numbers, maybe a stop loss. It might even come with a backtest that looks great.

That's the problem. It's easy for an AI to sound sure. It's hard for anyone to know if the idea really works.

I'm building Pickled Lizard because of this gap. Here's the simple process we use to test AI trading ideas. You can use it too, with or without us.

Why AI strategies need extra checking

An AI model is good at writing ideas. It is not good at knowing if those ideas hold up.

Three things go wrong a lot:

1. It can't see real data unless you give it data. When a chat answer tells you "how this strategy did," check where the numbers came from. If the AI didn't run the rules on real market data, there are no real numbers.
2. It makes too many ideas. If you ask for 50 strategies and keep the best-looking one, you probably picked the luckiest one, not the best one.
3. It forgets the boring costs. Fees, slippage, funding, and missed fills often get left out. A strategy that looks fine without costs can lose money once you pay to trade.

None of this means AI is useless. It means AI ideas need a test, just like ideas from people.

Step 1: Write the rules down, plainly

Before you test anything, turn the AI's answer into clear rules. What makes it enter a trade? What makes it exit? How big is each trade? Which market, and which time frame?

If you can't write a rule as a simple sentence, you can't test it. Ask the AI to rewrite the fuzzy parts until every rule is clear. "Buy when momentum looks strong" is not a rule. "Buy when the 20-period average crosses above the 50-period average on the 4-hour chart" is.

Then save that version. That's your locked rule set.

Step 2: Backtest it, honestly

Now run the rules on past data. This is a backtest. It answers one question: would this have worked before?

A few honest habits make a big difference:

• Include real fees. Use the fees you would really pay, on every trade.
• No peeking. The strategy can only use data it would have had at that moment. If a rule uses a price that wasn't known yet, the test is broken. We call this point-in-time truth: what did it know, and when did it know it?
• Count every try. If you tested 20 versions, write down 20, not just the best one.

Most ideas fail here. That's good news. A failed backtest costs nothing.

Step 3: Try to break it

If it passed, attack it. Run it on a different time period. Try another market. Make fees a bit worse. Move the start date.

A real pattern should still hold up, more or less, when you change small things. If the result falls apart when you shift one number, it was probably luck.

Step 4: Forward test it on paper

This is the step most people skip, and the one that matters most.

Take the locked rules and run them on new data as it comes in, with paper money. No changes allowed. The idea can't have "seen" this data, so nobody could have tuned it to fit.

A backtest shows what would have worked. A forward test shows if the idea survives.

Give it time. One good week means very little. You want enough trades to see a pattern, in more than one kind of market.

Step 5: Keep score the honest way

When you look at results, count all of them. Not "my best strategy did great." Instead, use the full count. For example, if you tested 11 strategies and 3 came out ahead on paper, you'd write "3 of 11 strategies net positive (simulated)."

The "of 11" part keeps you honest, because it shows the ideas that failed too. The "simulated" part reminds everyone it's paper. If you share results with anyone, share both.

Step 6: Set your risk limits yourself

An AI can suggest how much to risk. It should not decide. Set your own limits first: the most you'll put on one trade, the most you're willing to lose in a day, and when everything stops.

At Pickled Lizard we call this the Risk Passport. Your limits belong to you. The AI can only propose trades inside them, and it can't change them.

How Pickled Lizard does this

We built Pickled Lizard around this process. You bring your own AI: ChatGPT, Claude, Grok, Cursor, any of them. Your idea goes through what we call the Research Factory: discover, test, challenge, forward observe, prove. Paper trades use real fees, real margin rules, and real timestamps.

Ideas also move up in stages: Shadow, then Unproven Paper, then Verified. Each step has to be earned. A nice backtest alone doesn't skip anything.

We say it this way: AI thinks. Pickled Lizard proves and executes.

We're not building an AI that tells you what to buy. We're building the infrastructure that forces AI trading ideas to prove themselves.

A quick checklist

Before you trust any AI trading idea, ask:

• Are the rules written down and locked?
• Did the backtest include real fees and no peeking?
• Did I count every version I tried?
• Has it run forward, on paper, on data it never saw?
• Are my risk limits set by me, not the AI?

If any answer is no, keep it on paper.

Try it free

The beta is free. We're looking for our first 1,000 testers, the Founding 1,000. Bring an idea from your AI and watch it face new data on paper. You can also see simulated paper results on our paper trading leaderboard.

Join the Founding 1,000

More on how we test ideas: see all posts on Learn. Want to know why I'm building this? Read the story on our About page.

Paper trading only. Results are simulated. Not financial advice.