In This Episode
- When small numbers of backtesting trades are enough
- Our thoughts on Monte Carlo simulations
- Why big sample sizes can actually be bad
What's the Least Number of Trades to Trust a System
Many traders online talk about having a minimum number of completed or backtested trades for you to trust a trading system.
While there is some validity to that idea, it's not always true.
There are some trading strategies that can be validated with just a few trades. So listen in as we talk about the idea of sample size and what we've discovered in our own experiences.
Read the Transcript:
Walter: Hi, Hugh. How is it going?
Hugh: Hi! Good, Walter.
Walter: So I got this question from a trader today and she's doing really well. I recommended that she get a Forex Tester and start simulating. She's sort of new to trading and tripled her account. It's what I normally say, “Try and triple your account” so she did.
She was trading I think a dollar a pip on a thousand dollar demo or not demo before, excess of her account. She tripled it in thirty-two trades trading the daily charts. I think she was trading kangaroo tails on the Euro then she said, “You know, I'm pretty excited blah, blah, blah. I'm going to start moving on to other currency pairs and do more, get more kangaroo tail under my belt” and that sort of thing.
I said, “Yeah, that's great.” There's a couple of things that I thought of when this came up and one of them was that you hear people say, “You don't have enough data”. Normally if someone had thirty trades, people would say things like, “Oh that's not enough data. You don't know if it's significant” or whatever and that can be true.
That's why for example when you see companies and they do like drug tests. Where they have a placebo and then the real drug, they'll have thousands of people in the study. One of the reasons why that is because when you have thousands of people in the study you have the ability to zoom in and see really little tiny effects.
The placebo effect is well known to help like thirty-something percent of people just by taking a sugar pill. So they need a big sample because if they had a small sample of you know fifty people in each group, they'd probably find nothing.
They spend all that money creating that drug for nothing. They need big samples so they can zoom in and see really tiny small effects. So I said, “It's great that it worked out to be thirty-two trades and you're able to triple your account. But, you might have been risking too much because in the future you might not be that lucky”. Actually end up with a lower win rate, more losing trades in a row and get into a deeper draw down. Throw up your hands and say, “Ah, I quit.” So that's one thing, obvious.
The other thing is, what people don't talk about small samples is, if you get an effect for a really small sample of in this case you know thirty-two trades, where she had a fifty-nine percent win rate; a one-point-two reward to risk ratio, that can tell you that you have a really strong edge.
If you only take a few trades and you find like this really good effect. Even if you you take out the impact of the risk percentage that she was using per trade, even if you pull that out just like in terms of R, — the amount of risk versus the amount of return you know — you see that this is a really strong effect that could mean that you're dealing with a really strong edge. Your interpretation of the kangaroo tail, that the way that you trade it is viable.
Hugh: Hey there! I hope you find this episode useful. I just want to let you know that Walter and I give away something valuable every month that helps traders improve their skills. You can enter to win by simply leaving an iTunes review and leaving a comment on our YouTube videos.
At the end of each month, we'll look at the comments and reviews from the month and we'll pick a winner at random. Each comment and each review counts for one entry during the month that it's pitted.
So, if you're interested in that, be sure to enter after this podcast is over. Alright, back to the episode.
Walter: So that's something that no one ever talks about. People just say, they just assume that small sample equals maybe you just got a freaky lucky you know thing but actually it also could be that you actually don't need a really strong telescope to see the moon. You can take them in with your bare eyes and that's the kind of thing that could be happening with that.
So that's something I just want to point out for traders. When you're testing is that you know, it's really easy. Everyone knows a story but you don't have enough data. Fine. The other side of that is if you have too much data, you're going to find a really small edge.
Maybe you don't want to trade that. Maybe you don't want to trade the really small edge. Maybe you want to trade those systems that after fifty trades, it's pretty obvious that there's something going on here. So that's something that I wanted to point out and I think can be useful to the other traders out there.
I know like in the past, you've traded systems. Presumably I know that you've tested because you're really good at that and then later on you might decide, “Well, this isn't really for me”. Is that because you think that you were trading something with a small edge or are there other reasons that came into that?
Hugh: I think in the past for sure like the small sample size kind of messed with my head a little bit. Now I understand that some of these systems are only going to have a few trades. Like what I'm testing right now. Actually, it only does not even one trade a month but when it does fire off, it's like a ninety percent probability of winning.
So it's one of those things where you just have to keep it in the toolbox and then if it comes up then just track the metrics. Track the results versus the back testing and they kind of go from there but definitely the small sample size thing did mess with me before.
Walter: It's a bit tricky. It's kind of like systems where everyone focuses on the entry and it's similar with testing. Everyone looks at the win rate and they look at the sample size. They all work together sort of thing. Some traders I know they won't even use money management.
Actually technically, she didn't either because she risked a constant lot size on all of the trades even when the account doubled and you know got to the tripling point; that's kind of cool because when you use fixed lot sizes you can look at R. So that's a way of looking at R. Your R return and I know lots of traders, usually they're programmers.
They have a real analytical mind. They won't even trade something unless they run the test in R you know on the basis of risk and see what kind of R did it return. Did it return 15R or 50R or 30R you know after five hundred trades? How many R am I up? So that's another way to look at it and that's actually probably one of the best ways to look at it as a trader.
Hugh: You can actually tune things when you're looking at R instead of just the per percent or whatever you're doing, fixed percentage.
Walter: Exactly because if you use some really hyper aggressive algorithm with your risk, you know that can make it look really good when really there's just a small edge though.
Hugh: Yeah and what do you think about Monte Carlo testing?
Walter: Oh, I'm a big fan. The reason why I like it and the reason why I pay people to create spreadsheets — because I'm not smart enough to make them — is because like if you use Forex Tester, you can get sucked into this idea. For example, that woman that was back testing the kangaroo tail. She posted in the forum and she said, “Hey, you know this is what I got.”
You cannot as a Forex Tester trader, run through the same number of simulations that you can with Monte Carlo. You're just going to be on one path and so you can think, “Okay, that's going to be pretty close to what I'm going to trade if I go live.” The reality is that might not be true.
The great news is with Monte Carlo, people that don't know Monte Carlo is just when you run a whole bunch of simulations with your trading strategy — and there's lots of tools that will do it for you and then — what you get is kind of a spread of equity curve. You can see like, “Oh, okay you know the average path is here but it could be as bad as this” or as good as this.
So that kind of gives you a range of what to expect. You can do really cool things like eighty percent of the draw downs were you know less than thirty R or whatever you know those sorts of things that you can do. You just don't get that richness of data when you use Forex Tester.
Forex Tester is really cool for trying to figure out if you have an edge but Monte Carlo is important if you're trying to figure out what kind of risk rules to use because the starting point for that I think should be: What am I trying to avoid? What sort of draw down am I trying to avoid?
Hugh: Worst case scenario.
Walter: Yeah, worst case scenario. So that's really where Monte Carlo gets the best, I think.
Hugh: We'll try to link to some Monte Carlo tools in the show notes. Thanks a lot, Walter.
Walter: Okay, thanks. See you!
Hugh: All the information in this podcast is for educational and informational purposes only and is not trading or investment advice.
SHOW NOTES:
Enter the Monthly Contest
We give away a trading prize every month, so be sure to enter to win.
You can win by doing one or more of the following:
Each action counts for one entry in the month that it was posted. We will pick a winner at random from the entries that month.
The giveaways can include books, coaching sessions, trading tools, or surprise gifts.
Listen to the Audio Version
Click the play button below to hear the audio-only version. You can also download the mp3 file below.
Podcast: Play in new window | Download
How to Get New Episodes of The Think Profit Podcast
You can get notified of new episodes of the podcast by subscribing to our email list, or subscribing via any of the major podcast platforms that can be found here.
If you enjoyed this episode, a 5-star review on your favorite podcast platform is always greatly appreciated!
Thanks for listening and we hope that your trading is going well!