Is there a point to backtesting?

[ducati916]

Duc,

Do you believe that there i no positive bias in the indexes??

Do you believe that if you had bought the basket of stocks that represented the indexes in the '20's or '30's and held until now, your performance would have been the same as the index over that time???

If I'm so wrong, could you please give examples using real numbers??

brty

brty

I didn't say there was no positive bias in an index. Rather, that a powerful bias exists for equity that precedes any bias you may realise in an index. Hence why I said have a think about it.

The equity bias will drive any index bias that exists, thus is quantitatively more important.

jog on
duc

 

[MichaelD]

I have a system with a simple index filter, that doesn't allow new entries when the market filter is "off". It also tightens stops considerably when the filter is "off" on already open positions.

The difference in MaxDD and Absolute return is huge.

Care to elaborate / be more specific? The tightening of the stop in "unfavourable" conditions is a particularly interesting idea which I'd love to backtest in detail.

Try this code Metastock

(Mov(C,220,S)- Ref(Mov(C,220,S),- 55))/ATR(25)>3
LLV(Mov(C,50,S)*Mov(V,50,S),50)>500000
ATR(10)/Mov(C,10,S)>0.01 AND
ATR(10)/Mov(C,10,S)<0.1

Will do and report back the results.

 

[beamstas]

Care to elaborate / be more specific? The tightening of the stop in "unfavourable" conditions is a particularly interesting idea which I'd love to backtest in detail.

It's basic, simple, non curve fitted and works.

The index filter is a simple moving average filter, where trades will be taken or rejected depending on whether the market is trading above a certain moving average. The moving average you use doesn't matter, as long as you use the same for all stocks.

Rejecting trades when the market is trending down (or just starting a move up) seems to increase performance for the particular system. The average holding time is 7 days so i want the market to be moving upwards when i take trades. I don't need to "pick bottoms" or "turns" in the market because of the short holding time.

I had the idea of tightening stops when the market moves under the same moving average filter because it seemed strange that i was not accepting entries, but still leaving the stop fairly wide on open positions. Instead of using an X day lowest low trailing stop, i tighten it to a 1 day lowest low. This means that i can potentially stay in a trade while the market is moving down, as long as the stock is moving upwards. Using a 1 day lowest low will tend to force positions closed very fast, but what i'm doing is instead of liquidating all positions when the filter goes "off", i'm letting the market take me out instead.

You can use any moving average you like. I use a 40 day simple moving average, but i don't think it would matter too much, since holding time is so short.

Brad

 

[Nick Radge]

Michael,
Concur with Brad. Makes a big difference. Worthwhile looking into.

Nick

 

[Chorlton]

It's basic, simple, non curve fitted and works.

The index filter is a simple moving average filter, where trades will be taken or rejected depending on whether the market is trading above a certain moving average. The moving average you use doesn't matter, as long as you use the same for all stocks.

Rejecting trades when the market is trending down (or just starting a move up) seems to increase performance for the particular system. The average holding time is 7 days so i want the market to be moving upwards when i take trades. I don't need to "pick bottoms" or "turns" in the market because of the short holding time.

I had the idea of tightening stops when the market moves under the same moving average filter because it seemed strange that i was not accepting entries, but still leaving the stop fairly wide on open positions. Instead of using an X day lowest low trailing stop, i tighten it to a 1 day lowest low. This means that i can potentially stay in a trade while the market is moving down, as long as the stock is moving upwards. Using a 1 day lowest low will tend to force positions closed very fast, but what i'm doing is instead of liquidating all positions when the filter goes "off", i'm letting the market take me out instead.

You can use any moving average you like. I use a 40 day simple moving average, but i don't think it would matter too much, since holding time is so short.

Brad

I use a very similar strategy and based on my own testing, I agree with Brad that it works very well.

 

[MichaelD]

Try this code Metastock

(Mov(C,220,S)- Ref(Mov(C,220,S),- 55))/ATR(25)>3
AND LLV(Mov(C,50,S)*Mov(V,50,S),50)>500000
AND ATR(10)/Mov(C,10,S)>0.01
AND ATR(10)/Mov(C,10,S)<0.1

Interesting results.

Baseline system;
1340 trades
377% profit
9% drawdown

with the above filter added in
567 trades
190% profit
8.6% drawdown

Normalizing these results shows a much higher relative drawdown for the given return - almost double the drawdown for the same profit - but conversely only 1/3rd of the number of trades taken to get the result.

 

[MichaelD]

I have a system with a simple index filter, that doesn't allow new entries when the market filter is "off". It also tightens stops considerably when the filter is "off" on already open positions.

The difference in MaxDD and Absolute return is huge.

Baseline
1340 trades
377% profit
9.2% drawdown

As described above (tighten stop from 6.5 to 3.5 ATR when XAO below 60 day SMA, no new entries when XAO below 60 day SMA)
1105 trades
275% profit
11% drawdown

i.e. hurts my particular *long term* system

 

[fapturbo]

I would have to say backtesting and forward testing is an integral part of any trading system.

Automated System I'm working on, still needs a bit of work.

11% Max Drawdown.

2003 to 2009 Tested Period.

 

[TradeSim]

I would have to say backtesting and forward testing is an integral part of any trading system.

Automated System I'm working on, still needs a bit of work.

11% Max Drawdown.

2003 to 2009 Tested Period.

How do you find the fapturbo ?? Do you trade that using a forex robot ??

regards
david

 

[PolarBear]

I'm glad this has turned into such as interesting discussion.

I'd still like to get back to one of my original questions and see what everyone has to say about, how to establish what kind of time frame you would test back to into the past to verify a strategy that you want to start live trading in the market. Assume also that you can test the strategy live on current data also for a period of time. Just wanting to try to get to the bottom of people's opinions on this.

I think the 2 different opinions i've heard here so far are:
1) test on all the data - strategy should be able to cope with the different market conditions
2) test on only the past *** months of data as too far back is not representative of the current market (but how far back???)

what does everyone think?
cheers
-daniel

 

[howardbandy]

Hi Daniel --

The amount of data to use for backtesting depends on the model (the algorithm) and the data it is intended to model (the price series). Together, these comprise a trading system. The idea is to backtest over some period of time during which the model adjusts its logic and parameter values and gets itself "in sync" with the data. This period and this data is the "in-sample" data. Using the values for logic and parameters discovered in the in-sample runs, test a period of time that follows the in-sample data. The later test period is called the "out-of-sample" data. If the model has "learned" the signal portion of the in-sample data, the out-of-sample test will be profitable. If the model has learned the noise portion of the in-sample data, the out-of-sample test will not be profitable.

One of the important steps in trading system design is determination of the length of the in-sample period. That length varies with the model and the data. My recommendation is to keep the in-sample period as short as possible, consistent with profitable out-of-sample performance.

In-sample performance is always good -- we do not stop fooling with the system until it is good. In-sample results have no value in estimating the future performance of the system. Only the out-of-sample results are useful in estimating future performance.

Thanks for listening,
Howard

 

[Trembling Hand]

I think the 2 different opinions i've heard here so far are:
1) test on all the data - strategy should be able to cope with the different market conditions
2) test on only the past *** months of data as too far back is not representative of the current market (but how far back???)

what does everyone think?
cheers
-daniel

Polar Bear I think you are trying to make a short term system? If so you may be on a search for something that will break as soon as it works.

I cannot see how something that works on a short term basis in the equity futs 5 years ago would work now. Same something that worked in 08 just doesn't work now. People say yeah yeah markets are always the same but that is BS. The micro patterns, ranges, and trends on small time frames 1- 5 min are constantly changing in nature. Some times month to month but certainly year to year.

I've trader equity futs on a discretionary basis fulltime for the last 5 years. And have had to cycle through 5 different markets because of the continually changing conditions.

You are on a tough task, especially if you have a need to be right

 

[tech/a]

Polar Bear I think you are trying to make a short term system? If so you may be on a search for something that will break as soon as it works.

I cannot see how something that works on a short term basis in the equity futs 5 years ago would work now. Same something that worked in 08 just doesn't work now. People say yeah yeah markets are always the same but that is BS. The micro patterns, ranges, and trends on small time frames 1- 5 min are constantly changing in nature. Some times month to month but certainly year to year.

I've trader equity futs on a discretionary basis fulltime for the last 5 years. And have had to cycle through 5 different markets because of the continually changing conditions.

You are on a tough task, especially if you have a need to be right

Accepting you scalp.
Those that don't and trade for more than 10 secs to 1 min have some truisms to follow.

Buy low sell high---sell high buy back low.
Cut losses--let profits run.
Find a trend and hop on.
Test your plan to ensure its profitable.
Understand Risk and Positive expectancy.

 

[Trembling Hand]

Accepting you scalp.
Those that don't and trade for more than 10 secs to 1 min have some truisms to follow.

Buy low sell high---sell high buy back low.
Cut losses--let profits run.
Find a trend and hop on.
Test your plan to ensure its profitable.
Understand Risk and Positive expectancy.

Nah that's besides the point. The OP from memory is trying to find a short term fut trading system. Anything that trades more than 5 to 10 times a day will constantly be aiming at a moving target.

The dynamics of that time frame are always moving. You don't have to be a scalper to find that hard.

 

[PolarBear]

thanks for the replies.

Howard, i think you hit the nail on the head with what i was after. I've been trying to work out how far back to backtest and then test the algorithm on (as you say) out-of-sample data. BUT, the key point you made was to train the system on in-sample data, and then test how good it is on out-of-sample data that is after (MORE RECENT data) than the in-sample data. This does sound obvious now, but something i overlooked. thanks.

Polar Bear I think you are trying to make a short term system? I

TH, yes i'm trying to make a short term system. My goal is something that trades maybe once or twice a day that can consistantly make around 4 points profit per week.... that's the dream anyway.

cheers,
Daniel

 

[Tradesurfer]

how to establish what kind of time frame you would test back to into the past to verify a strategy that you want to start live trading in the market.

Polar Bear,

Exact time vs conditions presented are two different things. To explain, say I tested a long/short weekly trend based system.(actually used one) The system showed a short entry December 2007 on the s&p 500 and then closed out the short a few months ago and went long. Someone looking at the system would think it's the holy grail because it picked up the huge downturn profit and then went long and still long having picked up the move after the bottom was put in. If the market instead had gone sideways it would have seen little gains and little losses and pretty much be a flatline.

The point I hope I've made is that in developing this type of system, it was tested using quite a bit of data that incorporated trending, non trending, downtrending, sideways, volatile, non volatile, useless chop etc. What this did was provide useful information as to what results would be when it didn't get what it wanted. What hypothetically would have been the drawdowns. Also understand, there are conditions that have not yet happened an therefore have no way of testing.

Some have mentioned that why would I test a trending strategy during non trending periods? First I never predict what the market will do, only react and manage risk. Quite often the start of a huge trend might look the same as another that might have started only to fail and move down. This testing also helped develop filters to stay OUT of trades when the conditions weren't quite there and perhaps use a different strategy or simply do something else until things got better.

I think I also mentioned in my original post about survivorship bias.

I think the point was made about different stocks being in indexes over the years. Recently I was helping someone develop a short entry/exit strategy. They were running a backtest using the current S&P Index going back a number of years. They were a little disappointed in the results. The problem was that their backtest did not get credit for shorting Bear Stearns at $80 and riding down to $0. Didn't get Lehman Brothers, Enron. The reason was that those stocks are NO LONGER in the index and thus were not included in the test. This is why it is referred to as survivorship bias as only stocks still in the current index are tested on most platforms.

Monte Carlo was also mentioned and important. Say you have a system that tested extremely well especially on the equity curve. But what if all the losses occurred first instead of some wins? The curve might not have been all it seemed it was cracked up to be. For those unfamiliar with this feature in backtesting, it shuffles the order of the trades.

Before you start designing the perfect system , think about what type of trader you are. How often do you want to trade? Do you want to get signals every couple of minutes or once or twice a year. In other words, what time frame should you pick? What is your money management plan?

System development and testing for many traders can be more fun than actually trading. So have fun with it and good luck!

 

[Chorlton]

how to establish what kind of time frame you would test back to into the past to verify a strategy that you want to start live trading in the market.

Polar Bear,

Exact time vs conditions presented are two different things. To explain, say I tested a long/short weekly trend based system.(actually used one) The system showed a short entry December 2007 on the s&p 500 and then closed out the short a few months ago and went long. Someone looking at the system would think it's the holy grail because it picked up the huge downturn profit and then went long and still long having picked up the move after the bottom was put in. If the market instead had gone sideways it would have seen little gains and little losses and pretty much be a flatline.

The point I hope I've made is that in developing this type of system, it was tested using quite a bit of data that incorporated trending, non trending, downtrending, sideways, volatile, non volatile, useless chop etc. What this did was provide useful information as to what results would be when it didn't get what it wanted. What hypothetically would have been the drawdowns. Also understand, there are conditions that have not yet happened an therefore have no way of testing.

Some have mentioned that why would I test a trending strategy during non trending periods? First I never predict what the market will do, only react and manage risk. Quite often the start of a huge trend might look the same as another that might have started only to fail and move down. This testing also helped develop filters to stay OUT of trades when the conditions weren't quite there and perhaps use a different strategy or simply do something else until things got better.

I think I also mentioned in my original post about survivorship bias.

I think the point was made about different stocks being in indexes over the years. Recently I was helping someone develop a short entry/exit strategy. They were running a backtest using the current S&P Index going back a number of years. They were a little disappointed in the results. The problem was that their backtest did not get credit for shorting Bear Stearns at $80 and riding down to $0. Didn't get Lehman Brothers, Enron. The reason was that those stocks are NO LONGER in the index and thus were not included in the test. This is why it is referred to as survivorship bias as only stocks still in the current index are tested on most platforms.

Monte Carlo was also mentioned and important. Say you have a system that tested extremely well especially on the equity curve. But what if all the losses occurred first instead of some wins? The curve might not have been all it seemed it was cracked up to be. For those unfamiliar with this feature in backtesting, it shuffles the order of the trades.

Before you start designing the perfect system , think about what type of trader you are. How often do you want to trade? Do you want to get signals every couple of minutes or once or twice a year. In other words, what time frame should you pick? What is your money management plan?

System development and testing for many traders can be more fun than actually trading. So have fun with it and good luck!

Tradesurfer,

Excellent Post.

One question though, Is your definition of Monte Carlo totally correct? I wasn't aware that trades are actually shuffled. Instead, if more trades occur on a particular day than can be taken due to funds available, a different selection of those trades are chosen on each Monte Carlo run (ie some of them are ignored). If enough funds are available on that particular trading day, then all trades are taken. No shuffling as such takes place.

 

[weird]

Monte Carlo was also mentioned and important. For those unfamiliar with this feature in backtesting, it shuffles the order of the trades.

Is this the correct defintion ?

 

[Tradesurfer]

Thank you for the questions. If your interested in a detailed definition of monte carlo check out the link http://www.investopedia.com/articles/financial-theory/08/monte-carlo-multivariate-model.asp from investopedia.

One of the random things I've used backtesting programs for is the sequence of the buys and sells. Or, the sequence of when the data occurred.

The other thing that was mentioned in one of the posts was did the trader have the equity to actually do the trades?

This is a nice feature say if one were testing the entire S&P 500 index. If your test had you buying $1000 dollars of each stock when their trading rule = yes than in theory one might need $500,000 to be able to enter or carry each of the 500 issues at once. If the trader only had $50,000 instead, if stock number 51 came up as an entry, the trade would not be able to be put on since the $$$ would be exhausted. What if trade number 51 would have been buying Apple Computer and went on to a 5000% gain?

In this regard- it is important to account for what would have been reasonably put on and then re-shuffle the order of the trades. Maybe in the first one the trader bought apple and the strategy looked exceptional. In the second the apple trade would have been position # 68 and not able to buy.

You get the picture.

Now if your only trading say Aussie Dollar/USA Dollar on an intraday basis then the money available for x number of positions doesn't quite matter, while the number of contracts able to purchase or sell does. Plus the sequence of when the winning trades happened and when the losses happened might from an equity curve perspective help to understand what the account would look like.

Backtesting is a great tool to help see do i have something or don't I? Does my system get me out with smaller losses than wins? How does it perform. And then understand that in the future, conditions might develop that we have not been able to test. Its all about playing the probabilities.

Emotions is something though that can't be tested. How will you react when a winning trade is on? Can you resist the urger to take a profit even though your system still signals buy? Do you have the discipline to take small losses and accept that THIS is not going to work out THIS time?

Good discussion thread...

cheers

Derek

 

[Wysiwyg]

I am very much in the learning stage of this but the idea of taking a ten year block (or longer) and breaking that up into

1) full ten year backtest
2) two, five year backtests
3) three, three and a third year backtests
4) four, two and a half year backtests
and
5) five, two year backtests

All the results could be assessed as to performance over varied durations.

I`m still skeptical about testing what has already gone and could really only use results as a guide to follow as opposed to discretionary execution, based on the here and now.