The Deflated Sharpe Ratio: Why 1,000 Backtests Need a Higher Bar

Here's a trap almost everyone falls into, me included.

You test a trading idea. It looks so-so. You tweak it and test again. Then again. After a few hundred tries, one version looks amazing. Smooth curve, high Sharpe ratio, small drawdowns. You feel like you found something.

Maybe you did. Or maybe you just tried enough times that luck handed you a winner.

The deflated Sharpe ratio is a way to tell those two apart. This post explains the idea without the math wall, shows a quick simulation, and covers what you can do about it.

First, what's a Sharpe ratio?

The Sharpe ratio is a simple score for a strategy. It's the average return divided by how bumpy the returns were. Higher means you got more return for each unit of bumpiness.

People love it because it fits in one number. A Sharpe of 1 is often called decent. A Sharpe of 2 or 3 in a backtest makes people excited.

The problem is that one number says nothing about how many other numbers you threw away to find it.

The problem: the best of many tries is always flattering

Think about flipping coins. Ask 1,000 people to flip a coin 10 times. Someone will get 9 or 10 heads. That person isn't a great coin flipper. You just asked a lot of people.

Backtesting works the same way. Every time you change a setting, a time frame, a filter, or a coin, you run a new test. Each test is another person flipping coins. If you pick the best one at the end and look only at its score, you're looking at the luckiest flipper.

Researchers call this multiple testing, or selection bias. It's one of the main reasons backtests fall apart later. We wrote about other warning signs in 7 signs your strategy only works on the past.

A quick simulation (random data, no real edge)

To show how big this effect is, we ran a simple test. We made 1,000 fake strategies. Each one got a year of random daily returns with no edge at all. On average, every one of them should go nowhere.

Then we looked at the results. We repeated the whole thing 200 times and averaged it.

• About 498 of 1,000 strategies ended net positive (simulated). That's what you'd expect from coin flips.
• About 156 of 1,000 had a Sharpe ratio above 1.0. None of them had any real skill.
• The best strategy out of each 1,000 had an average Sharpe of about 3.3.

Read that last line again. Pure noise, no edge, and the top pick still showed a Sharpe that would look great on a chart. If you'd tested 1,000 ideas and shown only the winner, it would look like a breakthrough.

These are simulated results on random data. They are not trading results.

So what does "deflated" mean?

The deflated Sharpe ratio comes from a 2014 paper by David Bailey and Marcos López de Prado. The idea is simple, even if the formula isn't.

Instead of asking, "Is this Sharpe better than zero?", it asks, "Is this Sharpe better than what the luckiest of my tries would show by chance?"

It adjusts for a few things:

1. How many things you tried. The more ideas you tested, the higher the bar. As a rough guide, the luck you'd expect from the best of 10 tries is about 1.5 standard errors above zero. For 100 tries it's about 2.5. For 1,000 it's about 3.2. The bar keeps rising, just more slowly.

2. How long the test was. A short backtest has a lot of noise in its Sharpe. A longer one gives a steadier estimate, so the bar can come down a bit.

3. How strange the returns are. Crypto returns often have fat tails and lopsided days. A strategy that wins small most days and then loses big once in a while can show a high Sharpe right up until it doesn't. The deflated Sharpe ratio adjusts for that kind of shape.

The output is a probability. Roughly: how likely is it that this strategy has a real edge, after counting all your tries? A high Sharpe that came from trial 812 of 1,000 might come out much lower than you hoped.

The hard part: being honest about the count

The math is the easy part. The hard part is counting how many things you really tried.

Most people undercount. They forget the versions they deleted. They forget the coins they tested and dropped. They forget the "quick check" that changed the stop loss. Every one of those was a try.

This is why we care so much about keeping records of everything, including what failed. If you only keep the winners, you can't know how high your bar should be. We explain why in point-in-time truth and in our AI trading bot backtest checklist.

What you can do, even without the formula

You don't need to run the full math to use the idea:

• Write down every test. A plain list is fine. Count it before you get excited about any one result.
• Raise your bar as the count grows. If you tried 5 ideas, a Sharpe of 1.5 might mean something. If you tried 500, it probably doesn't.
• Be extra careful with short tests. A few months of data plus many tries is the perfect recipe for a lucky winner.
• Stress the winner. Shuffle its trades and see how bad the drawdown could get. We show how in Monte Carlo simulation, in plain English.
• Make it prove itself on new data. This is the strongest check. The future hasn't seen your tweaks.

How this fits Pickled Lizard

Pickled Lizard is built to test lots of trading ideas, from people and from their AIs. That means multiple testing is our problem too. The more ideas go in, the more lucky winners will show up.

So we don't treat a good backtest as the finish line. A strategy runs its backtest on the full data set we have. Then it has to hold up in forward Paper trading, on data it has never seen. It's marked Verified only when it reaches 80% of its own backtest's trade rate and 80% of its backtest win rate. Until it has made enough trades to judge, it shows as still collecting.

Forward testing is how a lucky backtest gets caught. We cover the difference in backtest vs forward observation. The more we trade, the smarter it gets, but only if we're honest about what we tried.

Try it

The beta is free. Bring a strategy idea, or one your AI came up with, and watch it run on Paper against real market data.

Test it free at pickledlizardz.com/join

Paper trading only. Results are simulated. Not financial advice.