Backtest vs Forward Observation: Why a Nice Chart Is Not Proof

Most trading ideas start the same way. Someone has a thought. They run a backtest. The chart goes up and to the right. They feel sure. Then they trade it, and it falls apart.

I've seen this happen a lot. I've done it myself. So when we started building Pickled Lizard, we made one rule early: a backtest is where an idea starts, not where it ends.

This post explains why, and what we do instead.

What a backtest really tells you

A backtest takes a set of rules and runs them on old market data. It asks: "If I had done this in the past, what would have happened?"

That's a useful question. It can kill bad ideas fast. If a strategy loses money on the past, it probably won't do better in the future.

But a good backtest only tells you one thing: this idea would have worked on data you already had. It does not tell you the idea will work next week.

Here's the problem. When you build a strategy, you look at the past. You tweak a number. You run it again. You tweak another number. After 50 tries, you find something that looks great. But you didn't find a pattern in the market. You found a pattern in your 50 tries. People call this overfitting. It's the most common way smart people fool themselves.

AI makes this easier, not harder. An AI model can make 1,000 strategies in an hour. Some of them will look amazing on old data just by luck. If you can't tell luck from skill, more ideas just means more ways to be wrong.

What forward observation does

Forward observation is simple. You take the idea, freeze the rules, and watch it on new data that did not exist when you made it.

No tweaking. No "let me just change this one thing." The rules are locked. The market does what it does. You write down what happens.

A backtest shows what would have worked. Forward observation shows if the idea survives.

It's slower. It's boring. You have to wait. But it's the only honest test, because the idea can't have "seen" the future data. Nobody could have tuned it to fit.

Our process: the Research Factory

At Pickled Lizard we call our process the Research Factory. It goes like this:

1. Discover. An idea comes in. It might come from a person, or from their AI (ChatGPT, Claude, Grok, Cursor, any of them).
2. Test. We run it on past data. This is the backtest. Most ideas die here, and that's fine.
3. Challenge. We try to break it. Different time periods. Different markets. Real fees. If it only works in one narrow window, we want to know.
4. Forward observe. We lock the rules and watch it on new data, in paper trading.
5. Prove. We only call it proven when it has a real track record on data it never saw.
6. Deploy. Only after that does it become eligible for more.

The common path most people take is: backtest, nice chart, trade. We put three steps in the middle on purpose.

Paper trading as evidence, not a toy

A lot of people think of paper trading as practice mode. We use it as an evidence system.

Every paper trade uses real fees, real margin rules, and real timestamps. That matters. A strategy that looks fine with zero fees can be a loser once you pay to trade.

Strategies also move through stages: Shadow, then Unproven Paper, then Verified, then Live eligible. An idea has to earn each step. It can't skip ahead because the backtest looked good.

How we talk about results

Because of all this, we're careful about how we share results. You won't see us post a big percent gain. You'll see the full count instead. For example, if 3 of 11 strategies came out ahead on paper, we'd write "3 of 11 strategies net positive (simulated)."

That line has two parts that matter. The first is "of 11." We always show the full count, including the ideas that failed. If you only show winners, you're hiding the most useful data. The second is "simulated." It's paper. We say so every time.

Why this matters for AI trading

Here's the big idea behind Pickled Lizard. We're not building an AI that tells you what to buy. We're building the infrastructure that forces AI trading ideas to prove themselves.

Your AI can think. It can come up with ideas all day. What it can't do on its own is check if those ideas hold up in the real world, with time passing and no peeking. That's the gap we're trying to fill. AI thinks. Pickled Lizard proves and executes.

A simple checklist you can use today

Even if you never use our platform, here's a quick test for any strategy:

• Did you change the rules after looking at the results? If yes, your backtest is too hopeful.
• Did you include real fees? If no, add them and run it again.
• Has it run on data it has never seen, with rules locked? If no, it's not tested yet.
• Are you showing all your tries, or only the best one? Count them all.

If an idea passes all four, it's worth watching. If not, keep it on paper.

Try it

The beta is free. We're looking for our first 1,000 testers, the Founding 1,000. Bring your own AI, give it an idea, and watch it face new data on paper.

Join the Founding 1,000

Want to know why I'm building this? Read the story on our About page.

Paper trading only. Results are simulated. Not financial advice.