- Joined
- 9 May 2009
- Posts
- 118
- Reactions
- 2
Thought I would post this in the beginners area as quite often I get questions about building or perfecting strategies and it is important for those new to the backtesting world to have some understanding of the tips and tricks so to speak.
Lets review Curve Fitting, Survivorship Bias, and setting realistic parameters with regard to funds available to trade.
Curve Fitting
Really this applies when traders are looking at past data in order to build a strategy. (which is offcourse backtesting)In other words, lets say over the last 10 years you were trading the Australian Stock Index and you noticed that over that time, a 31 day and a 46 day moving average crossover system yielded nice returns and seemed to always get you in at the right moments and get out before or as a trend is turning negative. Maybe you fiddled around with other time frames and it seemed that out of all of them this worked the best over these years. The challenge is that you have just built a system that is optimized for the PAST 10 years not nessacarily the future. This is whats refered to as CURVE FITTING where one has arranged the criteria in such a way to work best. While I am a believer in selective curve fitting (save for another thread) generally this won't be the best way to go about things.
Suvivorship or Inclusion Bias-
Recently someone was evaluating a long/short trading system and was doing so by testing 10 years worth of price data on the S&P 500 index. They couldn't understand why it did so much better on the long side than the short side despite some serious sell offs and down markets over that time.
The problem was that their trading system was testing 10 years worth of data on CURRENT index members. Those that have merged, gone bankrupt, etc etc were not included as they were no longer among the companies that make up the index. Put another way, this means that on the short side they didn't in testing see the benefit for their system on stocks like Enron, Pets.com, bear stearns, lehman brothers and more because they are no longer in today's S&P Index. For this reason their system was underperforming in testing but in real world probably would have had some pretty nice short positions since all of those went to zero. In Bear Stearns case, it would have been short at $80.
Now the other side of this is what about long positions? Hear again is where say a stock that is in TODAY's s/p index and is therefore included in this backtest might have shown up as being bought 5 years ago and maybe at the beginning of a huge uptrend and yielded a huge profit in testing. But would it have been? See, sometimes stocks that have run higher and are added to the s&P index which is a market cap index might have been added much later and would not have been bought by the trader at all or at that price. So in this example, the traders backtest would see a benefit in testing when in reality they would not have purchased it.
Now, this doesn't mean that backtesting is all bad. But just keep those things in mind.
Realistic Money Allocation
Lastly, I've seen traders test a trading strategy on say 5000 stocks where they set the backtesting software to hypothetically purchase $10,000 of each anytime their long signal triggered. Now while unlikely that buy or short sales would have simaltaneously happened in all 5000 companies, this trader would have needed say $50million in their account to hold all of these at once. One, as we can see this probably wouldn't be possible. But secondly, since with realistic money allocation, they might have been only able to trade the first X number of signals. What if the very best companies tested were signals 4999 and 5000. The traders wouldn't have had the $$$ left to buy at this point.
So when creating your first backtest- try to remember these things. Backtesting can be a great way to cross off strategies that don't work at all. To see how big losses might hypothetically be with a strategy, and to help tweak criteria. Test strategies during different types of markets. For example, if you have a range bound strategy designed to buy at support and sell at resistance, test it on things that have trended and vice versa. Try to be realistic on the money side. If possible, some backtesting has a monte carlo feature which simply means it re shuffles WHEN the order of when positions would have been opened. This is helpful say if your account couldn't sustain a number of losers in a row. Its important to know rather than getting a single backtest report where you just happened to have your first trades be winners.
I'll end it there but hope this helps as you start to think about building strategies and perhaps testing to see how they hypothetically might have done. Didn't want to delve too deep into things but feel free to reply to the thread with questions.
cheers
Derek
Lets review Curve Fitting, Survivorship Bias, and setting realistic parameters with regard to funds available to trade.
Curve Fitting
Really this applies when traders are looking at past data in order to build a strategy. (which is offcourse backtesting)In other words, lets say over the last 10 years you were trading the Australian Stock Index and you noticed that over that time, a 31 day and a 46 day moving average crossover system yielded nice returns and seemed to always get you in at the right moments and get out before or as a trend is turning negative. Maybe you fiddled around with other time frames and it seemed that out of all of them this worked the best over these years. The challenge is that you have just built a system that is optimized for the PAST 10 years not nessacarily the future. This is whats refered to as CURVE FITTING where one has arranged the criteria in such a way to work best. While I am a believer in selective curve fitting (save for another thread) generally this won't be the best way to go about things.
Suvivorship or Inclusion Bias-
Recently someone was evaluating a long/short trading system and was doing so by testing 10 years worth of price data on the S&P 500 index. They couldn't understand why it did so much better on the long side than the short side despite some serious sell offs and down markets over that time.
The problem was that their trading system was testing 10 years worth of data on CURRENT index members. Those that have merged, gone bankrupt, etc etc were not included as they were no longer among the companies that make up the index. Put another way, this means that on the short side they didn't in testing see the benefit for their system on stocks like Enron, Pets.com, bear stearns, lehman brothers and more because they are no longer in today's S&P Index. For this reason their system was underperforming in testing but in real world probably would have had some pretty nice short positions since all of those went to zero. In Bear Stearns case, it would have been short at $80.
Now the other side of this is what about long positions? Hear again is where say a stock that is in TODAY's s/p index and is therefore included in this backtest might have shown up as being bought 5 years ago and maybe at the beginning of a huge uptrend and yielded a huge profit in testing. But would it have been? See, sometimes stocks that have run higher and are added to the s&P index which is a market cap index might have been added much later and would not have been bought by the trader at all or at that price. So in this example, the traders backtest would see a benefit in testing when in reality they would not have purchased it.
Now, this doesn't mean that backtesting is all bad. But just keep those things in mind.
Realistic Money Allocation
Lastly, I've seen traders test a trading strategy on say 5000 stocks where they set the backtesting software to hypothetically purchase $10,000 of each anytime their long signal triggered. Now while unlikely that buy or short sales would have simaltaneously happened in all 5000 companies, this trader would have needed say $50million in their account to hold all of these at once. One, as we can see this probably wouldn't be possible. But secondly, since with realistic money allocation, they might have been only able to trade the first X number of signals. What if the very best companies tested were signals 4999 and 5000. The traders wouldn't have had the $$$ left to buy at this point.
So when creating your first backtest- try to remember these things. Backtesting can be a great way to cross off strategies that don't work at all. To see how big losses might hypothetically be with a strategy, and to help tweak criteria. Test strategies during different types of markets. For example, if you have a range bound strategy designed to buy at support and sell at resistance, test it on things that have trended and vice versa. Try to be realistic on the money side. If possible, some backtesting has a monte carlo feature which simply means it re shuffles WHEN the order of when positions would have been opened. This is helpful say if your account couldn't sustain a number of losers in a row. Its important to know rather than getting a single backtest report where you just happened to have your first trades be winners.
I'll end it there but hope this helps as you start to think about building strategies and perhaps testing to see how they hypothetically might have done. Didn't want to delve too deep into things but feel free to reply to the thread with questions.
cheers
Derek
