In-Sample vs Out-of-Sample Testing: What a Fair Split Looks Like

If you build a trading strategy and test it on the same data, the test can't fail. Not really. You shaped the rules to fit that data, so of course they fit.

That's why people split their data. One part is for building. The other part is for checking. It's one of the oldest ideas in testing, and one of the easiest to get wrong without noticing.

This post covers what in-sample and out-of-sample mean, what a fair split looks like, and the habits that keep the split honest.

The two words, in plain English

In-sample is the data you use to build the strategy. You look at it, try ideas, tune settings, and pick a version. Everything you learn about the strategy comes from here.

Out-of-sample is data you hide from yourself while you build. You don't look at it, you don't tune on it, and you don't peek "just once." When the strategy is finished and locked, you run it on this data one time and see what happens.

The in-sample result tells you how well you fit the past. The out-of-sample result is the first real hint of whether the idea works at all.

Why a strategy always looks good in-sample

Say you test 30 versions of a breakout strategy on two years of data and keep the best one. That winner looks great. But some of its result is the real pattern, and some of it is luck that happened to line up with those two years.

You can't tell those apart in-sample. Both look like edge. The only way to separate them is to show the strategy data it has never seen. Real patterns tend to carry over, at least partly. Luck usually doesn't.

We wrote more about the warning signs in 7 signs your strategy only works on the past.

What a fair split looks like

There's no single perfect ratio, but a few rules matter more than the exact numbers.

1. Split by time, not at random. Markets change over time, and trades near each other are related. If you shuffle bars randomly into two buckets, the out-of-sample set ends up full of days that sit right next to days you trained on. The test looks better than it should. Keep the out-of-sample period after the in-sample period, in one clean block.

2. Make the out-of-sample part big enough to mean something. A common starting point is roughly 70% in-sample and 30% out-of-sample. What really matters is the trade count. If your strategy trades twice a month, a three-month out-of-sample window gives you about six trades. That can't tell you much either way. Size the window by how many trades it will contain, not by how long it sounds.

3. Try to include more than one kind of market. If the in-sample period was mostly a rising market and the out-of-sample period is too, you've only tested one condition twice. Check whether your out-of-sample block includes some choppy or falling stretches.

4. Lock everything before you look. Rules, settings, fees, position size, and the date ranges themselves. Write them down. Then run the out-of-sample test once.

A simple example

Here's a made-up example to show the shape of it. The numbers are invented for this post.

You have three years of 4-hour data. You keep the first two years as in-sample and the final year as out-of-sample. You build and tune a trend strategy on the first two years, then lock it.

In-sample, it looks strong. You run it once on the final year. It's weaker, with fewer winning months, but it isn't a collapse. Split into months, maybe 7 of 12 months net positive (simulated).

That's a normal, healthy result. Expect a drop out-of-sample. A drop is fine. A collapse, or a result that flips sign, tells you the in-sample edge was mostly fit.

If the out-of-sample result is suspiciously close to the in-sample one, check your setup. That can mean information leaked across the split.

The mistakes that quietly ruin the split

Peeking and re-tuning. You run the out-of-sample test, it looks bad, so you tweak a setting and run it again. Now the out-of-sample data is part of your building process. It's in-sample in disguise. If you do this a few times, you've used up your clean data.

Choosing the split after seeing results. Moving the cut-off date until the out-of-sample period looks good is the same mistake in a different shape.

Leaking information. An indicator that uses the full candle before it closes, a value that got revised later, or a setting computed over the whole data set can all sneak future knowledge into the test. We call the fix point-in-time truth: only use what the strategy could have known at that moment. More in what did the AI know, and when?

Leaving out costs. Fees, slippage, and funding belong in both halves. A strategy that only works before costs doesn't work.

Stopping at one split. One out-of-sample period is one sample. Walk-forward testing repeats the split many times as it slides through history, which gives you a much better picture. See walk-forward analysis, explained.

Out-of-sample still isn't the finish line

Even a clean out-of-sample test uses old data. You chose what data to download and when to stop, and you've probably read about those years. Some bias always creeps in.

The strongest check is forward testing: run the locked strategy on new data as it arrives, on paper, with no changes. Nobody could have tuned it to data that didn't exist yet. We explain the difference in backtest vs forward observation.

How we use this at Pickled Lizard

At Pickled Lizard, ideas move through a Research Factory: discover, test, challenge, forward observe, prove. The out-of-sample check is part of "challenge." Passing it doesn't make a strategy Verified. It earns the right to run forward on paper, with real fees and real timestamps, where it has to keep up with its own backtest.

We also keep every result, including the ones that failed. If 11 versions were tried, the record says 11, not 1.

A quick checklist

• Is the out-of-sample block after the in-sample block in time?
• Does it hold enough trades to mean something?
• Were the rules, settings, and dates locked before you looked?
• Did you run it once, without re-tuning?
• Are costs included in both halves?
• Is the next step a forward test on paper?

Try it free

The beta is free. Bring a strategy idea from your own AI, ChatGPT, Claude, Grok, or Cursor, and watch it face new data on paper. You can also see simulated paper results on our paper trading leaderboard.

Test it free at pickledlizardz.com/join

More on testing ideas: see all posts on Learn. Why I'm building this: read the story on our About page.

Paper trading only. Results are simulated. Not financial advice.