# Is there a point to backtesting?



## PolarBear (23 August 2009)

Is there a point to backtesting and if so for how long? 

This is my question.  It may seem like a silly question, but i'll explain how i've come to this dilemma.

I've been backtesting automated strategies for about a year now on ES (emini s&p 500) 1 minute daily data.  I'm using ninja trader, which is good for me as it uses c#.net as it's programming environment, which i've had a lots of experience with so i've been able to test hundreds if not thousands of approaches in the time.... 

At the start i began just using a few years worth of data, however what i found was that i was able to optimise a particular strategy for that time frame, however when i let it run over the full 10 years worth of ES data the result would be overall negative over the time frame. 
Intuitively, I put this down to the problem of overfitting the test data - ie: the algorithm learns how to run on the training set data, but is not good for data unseen (in this case over the different timeframes).

I then began trying to develop algorithms that would work over the entire 10 years of the data.

Recently, i contacted a developer of a "system" which i'll let remain nameless, which i backtested and found that it didn't seem to work.  After contacting him I got a bunch of fairly abrupt responses, but his view was mainly summarised by the following:
"First of all going back to 1998 is really not the proper way to test any system. its anti productive.  Markets were different, volatility was different and you will never find anything that will work into the future based on a great past track history,"

This seems interesting to me.  I would like to know what everyone thinks about this.  
My thoughts are :
1) Backtesting is productive, allowing for development and refinement of algorithms based on real data
2) I agree that markets are different in the past, however I was trying to work towards a strategy that would work accross these changing markets, for exmple, using volatility... - is this not a good approach to aim for? 

So in essence i think backtesting is valuable, but i'm now questioning whether it's a waste of time to find an approach that works on old historical data that maybe is useless.....   If backtesting is good to do, what timeframe should be used - ie: just use say the last 2 years worth of ES data??

thanks.
-d


----------



## Timmy (23 August 2009)

Not really into backtesting but I know plenty are.  However, I trade the ES and there are many, many algos and bots active in it.  These automated strategies must be effective, so hang in there with it.


----------



## Mr J (23 August 2009)

> At the start i began just using a few years worth of data, however what i found was that i was able to optimise a particular strategy for that time frame, however when i let it run over the full 10 years worth of ES data the result would be overall negative over the time frame.




Of course, they're unprofitable strategies. A system being profitable over the data on which it was develop is to be expected. That data should also then be ignored for testing.



> Intuitively, I put this down to the problem of overfitting the test data




Maybe, but I'd just go with the strategy not being a profitable one.



> I then began trying to develop algorithms that would work over the entire 10 years of the data.




So, fitting it over those 10 years of data, but then you have no data on which to test it?



> Markets were different, volatility was different and you will never find anything that will work into the future based on a great past track history,"




I would think it depends on the strategy. I'm sure mine would have worked 10 years ago, and will work 10 years from now.


----------



## skyQuake (23 August 2009)

Do some research into system backtesting in general. 

A system lasting a year on the S&P would be fantastic. Edges are gained and lost all the time. What seems to be key now is a _*System that develops profitable systems*_, and can phase old stuff out and new stuff in.


----------



## Nick Radge (23 August 2009)

If you have over optimized or used too many parameter sets you will usually fall short of successful strategy trading. Usually 5 parameter sets is considered the upper limit, where less is better. 

Remember the goal is to find a system that works reasonably well all of the time rather than a system that works brilliant some of the time.

Also as a rule of thumb, if you can identify WHY the system should make money then you will (a) tend to need less backtesting and (b) be reasonably certain that it will make money.

Nick


----------



## PolarBear (23 August 2009)

Hi all, thanks for the quick interesting responses....  No-one's really got to the heart of my question though, which was: is backtesting worthwhile, and if so how long into the past is the data valid.  
Or another way of saying it, as an example, how long into the past would you backtest a strategy on say the ES, so that you can be confident (that word is going to cause some backlash...)  of trading that strategy profitably on the real market.

thanks again.
-daniel


----------



## motorway (23 August 2009)

PolarBear said:


> Is there a point to backtesting and if so for how long?
> 
> 2) I agree that markets are different in the past, however *I was trying to work towards a strategy that would work accross these changing markets, for exmple, using volatility...* - is this not a good approach to aim for?
> thanks.
> -d




Suspect this is a very wrong road to go down
When you filter with (or correct for ) volatility
You end up with just so much bath water
( ie the very volatile baby is chucked out as if it was never there)

Just an observation...
  what ever you use
make sure that the screens are objective

what is signal what is noise ?
often the noise is the signal

IE If you correct for volatility
You are simply filtering the signal out
and amplifying NOISE

motorway


----------



## howardbandy (23 August 2009)

PolarBear said:


> Hi all, thanks for the quick interesting responses....  No-one's really got to the heart of my question though, which was: is backtesting worthwhile, and if so how long into the past is the data valid.
> Or another way of saying it, as an example, how long into the past would you backtest a strategy on say the ES, so that you can be confident (that word is going to cause some backlash...)  of trading that strategy profitably on the real market.
> 
> thanks again.
> -daniel




Hi Daniel --

Backtesting is worth doing.  By backtesting, you find out whether your system worked sometime in the past.  Most people want to get some idea of how it might work in the future.  

So it is the testing on data that was not used in backtesting that is important.  Testing on data that was not used before is called out-of-sample testing.  

The data and time period used for the backtesting is called in-sample data.  Results over the in-sample period are always good -- we do not stop fooling with the system until they are good.  But in-sample results have no value in predicting future performance.

The goal is to gain confidence that trading the system tomorrow, with real money, is likely to be profitable.  But the best way I know how to gain that confidence is to practice.  The walk forward testing procedures give the practice.  Each step forward shows a transition from backtesting in-sample to forward testing out-of-sample.  Even using walk forward testing gives no guarantee, only an estimate based on statistical procedures.  

There are extensive discussions of the topic of backtesting in this forum.  Search on my name, howardbandy, and you will find several threads.



Thanks for listening,
Howard


----------



## Tradesurfer (25 August 2009)

When I backtest a new system I make sure and test during periods opposite to its design. In other words, I test trend based systems during periods I know not trending but sideways and vice versa. 

If you are testing a system that will profit when conditions occur and stay out when the are not occurring that it would seem a valid exercise. I would argue as a trend trader that trends happen again and again- in up and down markets and trading trends will ensure that one is on the right side of the market. 

One thing to be careful of is the survivorship bias - this simply refers to say testing the s&p 500 index on a particular strategy. If you are testing a short strategy on the current index, you wouldn't get the benefit of being short say Lehman brothers, bear stearns ,enron, etc etc.


----------



## daveM (25 August 2009)

One thing that is derived from backtesting is the practice of looking at the trading system and the charts very carefully to find entries and exits and the performance of the price.

That study alone is well worth the effort, it strengthens a person's discretionary entries.

I am a novice regarding trading system development.... I have found that I am scrutinizing the chart a lot more to find answers... surely that will help me as time passes...

That is just one small benefit of the exercise..


----------



## theasxgorilla (25 August 2009)

Tradesurfer said:


> When I backtest a new system I make sure and test during periods opposite to its design. In other words, I test trend based systems during periods I know not trending but sideways and vice versa.




I also do this, but the system I operate is designed to survive these periods.  Others I know of turn their system on and off at their own discretion when they identify that market conditions are not ideal.

As Nick has mentioned, the fewer parameters the better, but if you want the system to survive on it's own you probably need more parameters to keep the system out of unsuitable market conditions.


----------



## Cartman (25 August 2009)

Tradesurfer said:


> When I backtest a new system I make sure and *test during periods opposite to its design*. In other words, I test trend based systems during periods I know not trending but sideways and vice versa.




well i dont backtest anything as such .... always test systems with real money in a real market ... BUT .... your advice above T/Surfer (for punters who like to test stuff) is about as logical advice as i've seen (and exactly what i would do  ...

if u r gona see if something works, test it in unfavorable market conditions ... if it goes close to break even under those conditions ... might be time to mortgage the house


----------



## daveM (25 August 2009)

theasxgorilla said:


> I also do this, but the system I operate is designed to survive these periods.  Others I know of turn their system on and off at their own discretion when they identify that market conditions are not ideal.
> 
> As Nick has mentioned, the fewer parameters the better, but if you want the system to survive on it's own you probably need more parameters to keep the system out of unsuitable market conditions.





Quick question....... when you mention 'fewer parameters'........ how many would you start with before fine tuning begins.......?

Thanks


----------



## tech/a (25 August 2009)

> When I backtest a new system I make sure and test during periods opposite to its design. In other words, I test trend based systems during periods I know not trending but sideways and vice versa.




While this seems totally logical to some its totally foreign to me. 
Why would I try and fly a car or row a bus.
Why on earth would I expect something to perform well in an environment in which it wasnt designed?

Of course there is a point in backtesting.
This is the equity curve for a system which has been traded for many years live on the net.Note the flat period of late where it has been switched off!


----------



## MichaelD (25 August 2009)

theasxgorilla said:


> Others I know of turn their system on and off at their own discretion when they identify that market conditions are not ideal.






tech/a said:


> This is the equity curve for a system which has been traded for many years live on the net.Note the flat period of late where it has been switched off!




Something that occurred to me a while ago (and which is borne out by backtesting) is that switching systems on and off has severely adverse effects on their performance.

It may look and feel good to have locked in the returns during the good times, but the next equity peak has been limited as a direct result of these actions.


----------



## tech/a (25 August 2009)

MichaelD said:


> Something that occurred to me a while ago (and which is borne out by backtesting) is that switching systems on and off has severely adverse effects on their performance.
> 
> It may look and feel good to have locked in the returns during the good times, but the next equity peak has been limited as a direct result of these actions.




Yes your right.
With $300+K at risk I just couldn't do it.
Back into it now and without the 27% draw down the testing has subsequently shown.

I prefer my psychological damage to be seemingly controlled.


----------



## theasxgorilla (26 August 2009)

MichaelD said:


> Something that occurred to me a while ago (and which is borne out by backtesting) is that switching systems on and off has severely adverse effects on their performance.
> 
> It may look and feel good to have locked in the returns during the good times, but the next equity peak has been limited as a direct result of these actions.




I doubt this is correct.  Given a system that produces enough entries to use more than all the available capital, and thereby generate multiple future trading paths, luck will determine whether you get on one that has an equity peak which exceeds any other.


----------



## theasxgorilla (26 August 2009)

daveM said:


> Quick question....... when you mention 'fewer parameters'........ how many would you start with before fine tuning begins.......?
> 
> Thanks




I think I started with 5 or 6 and ended up with 11.  Parameters added included a volatility filter which prevents the system jumping on extreme breakouts, low-side entry price filters which prevents the system trading stocks priced below a certain level and rate of change filters, to frame entry into stocks that have gone up enough, but not too much in given period of time.

It's all in aid of smoothing the equity curve and making the system easier to live with, which ultimately comes from exposing the system to the right market conditions (it's a trend following system). And when your universe is the ASX you don't need to jump on everything that moves.


----------



## theasxgorilla (26 August 2009)

tech/a said:


> While this seems totally logical to some its totally foreign to me.
> *Why would I try and fly a car or row a bus.*
> Why on earth would I expect something to perform well in an environment in which it wasnt designed?




It's an extreme analogy, but I get it.  A better one might have been to say that a bus wasn't designed to drive off-road in the same manner as a 4x4, but it can, provided the conditions aren't too extreme, albeit with degraded performance.  Drive an F1 off-road and you'll destroy it pronto.

So it really depends on the kind of system we're talking about I think.  Some are more forgiving than others, by design.


----------



## MichaelD (26 August 2009)

theasxgorilla said:


> Luck will determine whether you get on one that has an equity peak which exceeds any other.



Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.

Your return is proportional to your risk. Take more risk, get more return. Smoothing one's equity curve with filters, limiting portfolio heat, stepping aside when conditions are "unfavourable" and all other such mechanisms decrease absolute system return in the long run.

You trade absolute gain for less psychological pain.

eg. Let's make a presumption - that we are seeing the beginnings of a new bull market. It may be, it may not be. Eventually, the type of scenario we have seen played out will turn into the next bull market.

Many traders are still equivocating. Various have posted here that they have begun to take entries, tentatively, or will be soon.

I have followed my system's signal all along. I have many fully pyramided positions, mostly starting near the "bottom" of this current market, and quite a few of them have trailing stops now way above the entries. I'm way ahead of those that remain on the sidelines or are dipping their toes in.

This is not because I cleverly picked the bottom of the market. This is because optimal performance of long term trend following does not presume to pick a market bottom, but by its natural operation it will be participating at the bottom.

Q: When do you get the lowest risk entries for a long term trend following system?
A: At the bottom of the market.

Q: How do you know you've reached the bottom of the market?
A: You can't know this until after the fact, when it's too late to take advantage of it.


----------



## theasxgorilla (26 August 2009)

MichaelD said:


> Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.




I see we have an interesting discussion on our hands. 

So let's deal with a system whose signals are generated by the same mechanism with the same money management/risk control parameters.  Make one instance of the system take all signals, let the other take 95% of signals by skipping a proportion of signals at random, and let another instance of the system be turned off and on by a trader according to their gut feel about where the market is at, top, bottom, good or bad conditions etc.

Are you saying that there is no chance the later two instances can beat the former, in terms of absolute gains, over any given period of time?


----------



## wayneL (26 August 2009)

MichaelD said:


> Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.




I'm with G. MM and risk control is of course paramount, but never underestimate the role of luck.

A simple illustration of this can be conducted in Excel and a random number generator.

The system is a simple zero expectancy system (50% chance of win - 1:1 risk/reward ratio) risking 10% fixed fractional risking 100 times

Results ranged from 90% loss of capital to 700% gain over 100 iterations.

A positive expectancy system might virtually guarantee a return over the long haul, but the real time results of two traders faithfully using the same system will diverge markedly.

Hell!* Even Monte Carlo testing shows this to be true also.*

What is the difference? Plain old dumb luck.

Denying this is so in face of overwhelming evidence, even from the backtesting system traders rely on seems ludicrous.


----------



## ducati916 (26 August 2009)

MichaelD said:


> Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.
> 
> Your return is proportional to your risk. Take more risk, get more return. Smoothing one's equity curve with filters, limiting portfolio heat, stepping aside when conditions are "unfavourable" and all other such mechanisms decrease absolute system return in the long run.
> 
> ...




And herein lies the legitimate fear of systems traders. You assume a bull market will eventuate. What if - it doesn't?

There are reasons, macroeconomic, why since the 1920's we have an upside bias. However, if those conditions were removed, would the bias [upside] remain?

jog on
duc


----------



## theasxgorilla (26 August 2009)

ducati916 said:


> And herein lies the legitimate fear of systems traders. You assume a bull market will eventuate. *What if - it doesn't?*
> 
> There are reasons, macroeconomic, why since the 1920's we have an upside bias. However, if those conditions were removed, would the bias [upside] remain?




System or not, the first question ought to be, should I be in equities?  You must have a view on this before you even invest (well-informed or otherwise)... beyond that point you're as committed as you are.  It's the same for anyone holding positions in equities, isn't it?


----------



## ducati916 (26 August 2009)

theasxgorilla said:


> System or not, the first question ought to be, should I be in equities?  You must have a view on this before you even invest (well-informed or otherwise)... beyond that point you're as committed as you are.  It's the same for anyone holding positions in equities, isn't it?




*asx*

The prevailing bias, as exemplified by MichaelD, seems to be over time, there is an upside bias to equities. Thus a *long* based system, with appropriate MM, should have a positive expectancy.

What if - that underlying macroeconomic factor driving the [agreed] current bias were removed?

I think that unconsiously many systems traders, unaware of the factor consiously, fear this potential variable.

jog on
duc


----------



## bingk6 (26 August 2009)

MichaelD said:


> Smoothing one's equity curve with filters, limiting portfolio heat, stepping aside when conditions are "unfavourable" and all other such mechanisms decrease absolute system return in the long run.




In terms of absolute system returns, I agree with you. However, the theoritical benefit of doing so is the reduction in drawdowns, at a greater % than the % reduction in absolute returns. So for somebody who uses some form of leverage, it is quite possible to achieve superior absolute returns and lower drawdowns than an unfiltered system.


----------



## Grep (26 August 2009)

MichaelD said:


> Your return is proportional to your risk. Take more risk, get more return.




This is not true.The relationship between risk and reward is not linear.
There is an efficient frontier beyond which increasing risk guarantees failure.


----------



## Aussiest (26 August 2009)

PolarBear said:


> 1) Backtesting is productive, allowing for development and refinement of algorithms based on real data
> 2) I agree that markets are different in the past, however I was trying to work towards a strategy that would work accross these changing markets, for exmple, using volatility... - is this not a good approach to aim for?
> 
> So in essence i think backtesting is valuable, but i'm now questioning whether it's a waste of time to find an approach that works on old historical data that maybe is useless.....   If backtesting is good to do, what timeframe should be used - ie: just use say the last 2 years worth of ES data??
> ...




You only have to study the market for 1+ year to see that there are significantly different time chunks with different variables, which would affect trading.

Eg, Around 2007 - into 2008: market was very volatile. Earlier this year (feb, march), volatility slowed down and market begun a bull run (bull run in bear market), and while the market rose, volatility slowed down. I have noticed over passed 2 weeks that volatility has increased again, thereby creating a better environment for day traders.

I would assume you would develop different strategies for different timeframes, esp. if trading intraday (upward bias is lost in a bear market).


----------



## brty (26 August 2009)

Duc,



> there is an upside bias to equities.




This bias is in the index, not individual stocks. There are precious few stocks that remain from the indexes of the '20's or ''30's. As the losers get dumped out of indexes, and the up-commers get included, this is where the bias lives.

The classic example of this is the ETF STW based on the ASX200. When this was first listed in late April '08 opening price was ~$55.20 while the ASX200 was at ~5,520. Today however STW is at ~$41.57 while the ASX200 is ~4,416.

In other words the fund that "mimicks" the index has managed to have a performance nearly 6% below the index in 15 months.

brty


----------



## Boggo (26 August 2009)

This is a chart of a TradeSim backtest of my breakout system.

It is a good example of how a breakout system cannot handle a negative market and proof that you need to park the Porsche when the market goes cross country which is what tech/a is saying.

I stopped trading this system in Dec 2007 but it is well and truly back up and running at the moment.

In reality though when a market turns down you really don't need to be Einstein to work out that a breakout long system is not going to work.

(the start date and the start equity on the right are the actual starting points and the chart was produced in May 09)

(click to expand)


----------



## tech/a (26 August 2009)

I agree with *Michael.*
If you turn discretionary on your system then your not trading your system.

But then I agree with myself.

"I'm sitting on the plane and have been for many many flights it goes up and down like clock work no problems."
In June 07 I notice the pilot walking around the plane with a worried look on his face.
Whats wrong?
This plane doesnt feel right infact it feels bloody terrible,I'm not happy flying it.
So he calls in all the experts.
We all agree this plane looks like it should be garaged until all is fixed.
So I have a look.
I agree it looks bloody un safe--so I get off the plane.
However the plane gets a new pilot and away it goes.
It crashes.Most survive,some die,I survive.

I hop on another plane with the same positive expectancy I had before.

I dont agree with *Duc*
Because I just dont agree with him.
Disagreement with Duc usually has an effect.


----------



## motorway (26 August 2009)

ducati916 said:


> *asx*
> 
> What if - that underlying macroeconomic factor driving the [agreed] current bias were removed?
> 
> ...




Free Markets MUST TREND

just like little streams MUST create Grand Canyons

Participants will Self organize 
so that trends always occur
prices will always adjust
such that  there will always be  bases and breakouts into trends ..

What If the underlying factor*s* changed ?

eg the little stream was stopped from flowing by Soviet Command ?

Then in the fullness of time NO trends ( NO Bias )
and No GrandCanyons

No 







> Matthew Effects







			
				tech/a said:
			
		

> All true Motorway
> but the point of developing a system is a far cry from trading in a discretionary manner.




My post in that thread does not contain the word _Discretionary_
My post was about the fact that the universe of what is useful to test
is much smaller than the universe of things that are tested.

hence why "things" seem to be so context driven
yet markets always do just the same things..

the real versus artefacts of the real...

motorway


----------



## motorway (26 August 2009)

Boggo said:


> This is a chart of a TradeSim backtest of my breakout system.
> 
> It is a good example of how a breakout system cannot handle a negative market and proof that you need to park the Porsche when the market goes cross country which is what tech/a is saying.
> 
> ...





Ok change to this

 This system stopped Trading in Dec 2007 but it is well and truly back up and running at the moment.

Of course--->

To be true it didn't stop at all.  It just had no entries triggered..

The system never stops even when it is doing nothing..

motorway


----------



## MichaelD (26 August 2009)

I won't multi-quote in this reply as I find this tends to get hard to read, but in reply to various points made herein;

The bell curves of possible equity returns between filtered and unfiltered trend following may overlap at the extreme ends, so I concede that some very lucky discretionary turn on/off traders may outperform some very unlucky mechanical traders, but that does not negate the fact that the median return is higher, to a very high degree of certainty.


----------



## wayneL (26 August 2009)

MichaelD said:


> I won't multi-quote in this reply as I find this tends to get hard to read, but in reply to various points made herein;
> 
> The bell curves of possible equity returns between filtered and unfiltered trend following may overlap at the extreme ends, so I concede that some very lucky discretionary turn on/off traders may outperform some very unlucky mechanical traders, but that does not negate the fact that the median return is higher, to a very high degree of certainty.




It might depend on the system. I recall Jose Silva had a filter that switched entries on/off that (allegedly) improved perfotmance. http://www.metastocktools.com/ADT/ADT-ASX.htm


----------



## beamstas (26 August 2009)

I have a system with a simple index filter, that doesn't allow new entries when the market filter is "off". It also tightens stops considerably when the filter is "off" on already open positions.

The difference in MaxDD and Absolute return is huge.


----------



## daveM (26 August 2009)

theasxgorilla

Thanks for that reply.....


----------



## fapturbo (26 August 2009)

Try this code Metastock

(Mov(C,220,S)- Ref(Mov(C,220,S),- 55))/ATR(25)>3
LLV(Mov(C,50,S)*Mov(V,50,S),50)>500000
ATR(10)/Mov(C,10,S)>0.01 AND
ATR(10)/Mov(C,10,S)<0.1


----------



## ducati916 (27 August 2009)

brty said:


> Duc,
> 
> 
> 
> ...




*brty*

Incorrect.

There exists a very powerful bias for equity [common stocks] This bias will favour different types of equity depending on the business cycle. Sentiment, fads, innovation can add to this already existing bias.

As to your indices hypothesis, you need to think that one through more carefully.

jog on
duc


----------



## brty (27 August 2009)

Duc,

Do you believe that there i no positive bias in the indexes??

Do you believe that if you had bought the basket of stocks that represented the indexes in the '20's or '30's and held until now, your performance would have been the same as the index over that time???

If I'm so wrong, could you please give examples using real numbers??

brty


----------



## ducati916 (27 August 2009)

brty said:


> Duc,
> 
> Do you believe that there i no positive bias in the indexes??
> 
> ...




*brty*

I didn't say there was no positive bias in an index. Rather, that a powerful bias exists for equity that precedes any bias you may realise in an index. Hence why I said have a think about it.

The equity bias will drive any index bias that exists, thus is quantitatively more important.

jog on
duc


----------



## MichaelD (27 August 2009)

beamstas said:


> I have a system with a simple index filter, that doesn't allow new entries when the market filter is "off". It also tightens stops considerably when the filter is "off" on already open positions.
> 
> The difference in MaxDD and Absolute return is huge.




Care to elaborate / be more specific? The tightening of the stop in "unfavourable" conditions is a particularly interesting idea which I'd love to backtest in detail.



fapturbo said:


> Try this code Metastock
> 
> (Mov(C,220,S)- Ref(Mov(C,220,S),- 55))/ATR(25)>3
> LLV(Mov(C,50,S)*Mov(V,50,S),50)>500000
> ...




Will do and report back the results.


----------



## beamstas (28 August 2009)

MichaelD said:


> Care to elaborate / be more specific? The tightening of the stop in "unfavourable" conditions is a particularly interesting idea which I'd love to backtest in detail.




It's basic, simple, non curve fitted and works.

The index filter is a simple moving average filter, where trades will be taken or rejected depending on whether the market is trading above a certain moving average. The moving average you use doesn't matter, as long as you use the same for all stocks. 

Rejecting trades when the market is trending down (or just starting a move up) seems to increase performance for the particular system. The average holding time is 7 days so i want the market to be moving upwards when i take trades. I don't need to "pick bottoms" or "turns" in the market because of the short holding time.

I had the idea of tightening stops when the market moves under the same moving average filter because it seemed strange that i was not accepting entries, but still leaving the stop fairly wide on open positions. Instead of using an X day lowest low trailing stop, i tighten it to a 1 day lowest low. This means that i can potentially stay in a trade while the market is moving down, as long as the stock is moving upwards. Using a 1 day lowest low will tend to force positions closed very fast, but what i'm doing is instead of liquidating all positions when the filter goes "off", i'm letting the market take me out instead.

You can use any moving average you like. I use a 40 day simple moving average, but i don't think it would matter too much, since holding time is so short. 

Brad


----------



## Nick Radge (28 August 2009)

Michael,
Concur with Brad. Makes a big difference. Worthwhile looking into.

Nick


----------



## Chorlton (28 August 2009)

beamstas said:


> It's basic, simple, non curve fitted and works.
> 
> The index filter is a simple moving average filter, where trades will be taken or rejected depending on whether the market is trading above a certain moving average. The moving average you use doesn't matter, as long as you use the same for all stocks.
> 
> ...





I use a very similar strategy and based on my own testing, I agree with Brad that it works very well.


----------



## MichaelD (3 September 2009)

fapturbo said:


> Try this code Metastock
> 
> (Mov(C,220,S)- Ref(Mov(C,220,S),- 55))/ATR(25)>3
> AND LLV(Mov(C,50,S)*Mov(V,50,S),50)>500000
> ...




Interesting results.

Baseline system;
1340 trades
377% profit
9% drawdown

with the above filter added in
567 trades
190% profit
8.6% drawdown

Normalizing these results shows a much higher relative drawdown for the given return - almost double the drawdown for the same profit - but conversely only 1/3rd of the number of trades taken to get the result.


----------



## MichaelD (3 September 2009)

beamstas said:


> I have a system with a simple index filter, that doesn't allow new entries when the market filter is "off". It also tightens stops considerably when the filter is "off" on already open positions.
> 
> The difference in MaxDD and Absolute return is huge.




Baseline
1340 trades
377% profit
9.2% drawdown

As described above (tighten stop from 6.5 to 3.5 ATR when XAO below 60 day SMA, no new entries when XAO below 60 day SMA)
1105 trades
275% profit
11% drawdown

i.e. hurts my particular *long term* system


----------



## fapturbo (3 September 2009)

I would have to say backtesting and forward testing is an integral part of any trading system.

Automated System I'm working on, still needs a bit of work.

11% Max Drawdown.

2003 to 2009 Tested Period.


----------



## TradeSim (4 September 2009)

fapturbo said:


> I would have to say backtesting and forward testing is an integral part of any trading system.
> 
> Automated System I'm working on, still needs a bit of work.
> 
> ...




How do you find the fapturbo ?? Do you trade that using a forex robot ??

regards
david


----------



## PolarBear (8 September 2009)

I'm glad this has turned into such as interesting discussion.

I'd still like to get back to one of my original questions and see what everyone has to say about, how to establish what kind of time frame you would test back to into the past to verify a strategy that you want to start live trading in the market.  Assume also that you can test the strategy live on current data also for a period of time.  Just wanting to try to get to the bottom of people's opinions on this. 

I think the 2 different opinions i've heard here so far are: 
1) test on all the data - strategy should be able to cope with the different market conditions
2) test on only the past *** months of data as too far back is not representative of the current market  (but how far back???) 

what does everyone think?
cheers
-daniel


----------



## howardbandy (9 September 2009)

Hi Daniel --

The amount of data to use for backtesting depends on the model (the algorithm) and the data it is intended to model (the price series).  Together, these comprise a trading system.  The idea is to backtest over some period of time during which the model adjusts its logic and parameter values and gets itself "in sync" with the data.  This period and this data is the "in-sample" data. Using the values for logic and parameters discovered in the in-sample runs, test a period of time that follows the in-sample data.  The later test period is called the "out-of-sample" data.  If the model has "learned" the signal portion of the in-sample data, the out-of-sample test will be profitable.  If the model has learned the noise portion of the in-sample data, the out-of-sample test will not be profitable.

One of the important steps in trading system design is determination of the length of the in-sample period.  That length varies with the model and the data.  My recommendation is to keep the in-sample period as short as possible, consistent with profitable out-of-sample performance.

In-sample performance is always good -- we do not stop fooling with the system until it is good.  In-sample results have no value in estimating the future performance of the system.  Only the out-of-sample results are useful in estimating future performance.

Thanks for listening,
Howard


----------



## Trembling Hand (9 September 2009)

PolarBear said:


> I think the 2 different opinions i've heard here so far are:
> 1) test on all the data - strategy should be able to cope with the different market conditions
> 2) test on only the past *** months of data as too far back is not representative of the current market  (but how far back???)
> 
> ...




Polar Bear I think you are trying to make a short term system? If so you may be on a search for something that will break as soon as it works.

I cannot see how something that works on a short term basis in the equity futs 5 years ago would work now. Same something that worked in 08 just doesn't work now. People say yeah yeah markets are always the same but that is BS. The micro patterns, ranges, and trends on small time frames 1- 5 min are constantly changing in nature. Some times month to month but certainly year to year.

I've trader equity futs on a discretionary basis fulltime for the last 5 years. And have had to cycle through 5 different markets because of the continually changing conditions.

You are on a tough task, especially if you have a need to be right


----------



## tech/a (9 September 2009)

Trembling Hand said:


> Polar Bear I think you are trying to make a short term system? If so you may be on a search for something that will break as soon as it works.
> 
> I cannot see how something that works on a short term basis in the equity futs 5 years ago would work now. Same something that worked in 08 just doesn't work now. People say yeah yeah markets are always the same but that is BS. The micro patterns, ranges, and trends on small time frames 1- 5 min are constantly changing in nature. Some times month to month but certainly year to year.
> 
> ...




Accepting you scalp.
Those that don't and trade for more than 10 secs to 1 min have some truisms to follow.

Buy low sell high---sell high buy back low.
Cut losses--let profits run.
Find a trend and hop on.
Test your plan to ensure its profitable.
Understand Risk and Positive expectancy.


----------



## Trembling Hand (9 September 2009)

tech/a said:


> Accepting you scalp.
> Those that don't and trade for more than 10 secs to 1 min have some truisms to follow.
> 
> Buy low sell high---sell high buy back low.
> ...



 Nah that's besides the point. The OP from memory is trying to find a short term fut trading system. Anything that trades more than 5 to 10 times a day will constantly be aiming at a moving target.

The dynamics of that time frame are always moving. You don't have to be a scalper to find that hard.


----------



## PolarBear (11 September 2009)

thanks for the replies.

Howard, i think you hit the nail on the head with what i was after.  I've been trying to work out how far back to backtest and then test the algorithm on (as you say) out-of-sample data.  BUT, the key point you made was to train the system on in-sample data, and then test how good it is on out-of-sample data   that  is after (MORE RECENT data)  than the in-sample data.  This does sound obvious now, but something i overlooked.  thanks.



> Polar Bear I think you are trying to make a short term system? I



TH, yes i'm trying to make a short term system.  My goal is something that trades maybe once or twice a day that can consistantly make around 4 points profit per week.... that's the dream anyway.

cheers,
Daniel


----------



## Tradesurfer (11 September 2009)

> how to establish what kind of time frame you would test back to into the past to verify a strategy that you want to start live trading in the market.




Polar Bear,

Exact time vs conditions presented are two different things. To explain, say I tested a long/short weekly trend based system.(actually used one) The system showed a short entry December 2007 on the s&p 500 and then closed out the short a few months ago and went long. Someone looking at the system would think it's the holy grail because it picked up the huge downturn profit and then went long and still long having picked up the move after the bottom was put in. If the market instead had gone sideways it would have seen little gains and little losses and pretty much be a flatline.

The point I hope I've made is that in developing this type of system, it was tested using quite a bit of data that incorporated trending, non trending, downtrending, sideways, volatile, non volatile, useless chop etc. What this did was provide useful information as to what results would be when it didn't get what it wanted. What hypothetically would have been the drawdowns. Also understand, there are conditions that have not yet happened an therefore have no way of testing.

Some have mentioned that why would I test a trending strategy during non trending periods? First I never predict what the market will do, only react and manage risk. Quite often the start of a huge trend might look the same as another that might have started only to fail and move down. This testing also helped develop filters to stay OUT of trades when the conditions weren't quite there and perhaps use a different strategy or simply do something else until things got better.

I think I also mentioned in my original post about survivorship bias.

I think the point was made about different stocks being in indexes over the years. Recently I was helping someone develop a short entry/exit strategy. They were running a backtest using the current S&P Index going back a number of years. They were a little disappointed in the results. The problem was that their backtest did not get credit for shorting Bear Stearns at $80 and riding down to $0. Didn't get Lehman Brothers, Enron. The reason was that those stocks are NO LONGER in the index and thus were not included in the test. This is why it is referred to as survivorship bias as only stocks still in the current index are tested on most platforms.

Monte Carlo was also mentioned and important. Say you have a system that tested extremely well especially on the equity curve. But what if all the losses occurred first instead of some wins? The curve might not have been all it seemed it was cracked up to be. For those unfamiliar with this feature in backtesting, it shuffles the order of the trades. 

Before you start designing the perfect system , think about what type of trader you are. How often do you want to trade? Do you want to get signals every couple of minutes or once or twice a year. In other words, what time frame should you pick? What is your money management plan?

System development and testing for many traders can be more fun than actually trading. So have fun with it and good luck!


----------



## Chorlton (11 September 2009)

Tradesurfer said:


> > how to establish what kind of time frame you would test back to into the past to verify a strategy that you want to start live trading in the market.
> 
> 
> 
> ...




Tradesurfer,

Excellent Post.

One question though, Is your definition of Monte Carlo totally correct?  I wasn't aware that trades are actually shuffled. Instead, if more trades occur on a particular day than can be taken due to funds available, a different selection of those trades are chosen on each Monte Carlo run (ie some of them are ignored). If enough funds are available on that particular trading day, then all trades are taken. No shuffling as such takes place.


----------



## weird (11 September 2009)

Tradesurfer said:


> Monte Carlo was also mentioned and important.  For those unfamiliar with this feature in backtesting, *it shuffles the order of the trades*.




Is this the correct defintion ?


----------



## Tradesurfer (12 September 2009)

Thank you for the questions. If your interested in a detailed definition of monte carlo check out the link http://www.investopedia.com/articles/financial-theory/08/monte-carlo-multivariate-model.asp from investopedia.

One of the random things I've used backtesting programs for is the sequence of the buys and sells. Or, the sequence of when the data occurred. 

The other thing that was mentioned in one of the posts was did the trader have the equity to actually do the trades?

This is a nice feature say if one were testing the entire S&P 500 index. If your test had you buying $1000 dollars of each stock when their trading rule = yes than in theory one might need $500,000 to be able to enter or carry each of the 500 issues at once. If the trader only had $50,000 instead, if stock number 51 came up as an entry, the trade would not be able to be put on since the $$$ would be exhausted. What if trade number 51 would have been buying Apple Computer and went on to a 5000% gain? 

In this regard- it is important to account for what would have been reasonably put on and then re-shuffle the order of the trades. Maybe in the first one the trader bought apple and the strategy looked exceptional. In the second the apple trade would have been position # 68 and not able to buy.

You get the picture.

Now if your only trading say Aussie Dollar/USA Dollar on an intraday basis then the money available for x number of positions doesn't quite matter, while the number of contracts able to purchase or sell does. Plus the sequence of when the winning trades happened and when the losses happened might from an equity curve perspective help to understand what the account would look like.

Backtesting is a great tool to help see do i have something or don't I? Does my system get me out with smaller losses than wins? How does it perform. And then understand that in the future, conditions might develop that we have not been able to test. Its all about playing the probabilities.

Emotions is something though that can't be tested. How will you react when a winning trade is on? Can you resist the urger to take a profit even though your system still signals buy? Do you have the discipline to take small losses and accept that THIS is not going to work out THIS time?

Good discussion thread...

cheers

Derek


----------



## Wysiwyg (12 September 2009)

I am very much in the learning stage of this but the idea of taking a ten year block (or longer) and breaking that up into 

1) full ten year backtest
2) two, five year backtests
3) three, three and a third year backtests
4) four, two and a half year backtests
and 
5) five, two year backtests 

All the results could be assessed as to performance over varied durations. 

I`m still skeptical about testing what has already gone and could really only use results as a guide to follow as opposed to discretionary execution, based on the here and now.


----------



## Chorlton (12 September 2009)

Wysiwyg said:


> I am very much in the learning stage of this but the idea of taking a ten year block (or longer) and breaking that up into
> 
> 1) full ten year backtest
> 2) two, five year backtests
> ...





Just make sure that if the purpose of backtesting is to assess the performance of the System (and to give you confidence in it going forward) make sure that the data you use is Out-of-Sample. 

Otherwise, the results are meaningless.


----------



## Wysiwyg (12 September 2009)

Chorlton said:


> Just make sure that if the purpose of backtesting is to assess the performance of the System (and to give you confidence in it going forward) make sure that the data you use is Out-of-Sample.
> 
> Otherwise, the results are meaningless.




Yes it would be worth doing the same  test procedure on another 10 year block outside the first. Unlimited alternatives to these.


----------



## tech/a (13 September 2009)

There are plenty of bourses around if designing a system for stock.
I have 7 data sets.
You can of course test those as well.
In sample/out of sample/Montecarlo.
Should work in all.


----------



## howardbandy (13 September 2009)

PolarBear said:


> thanks for the replies.
> 
> Howard, i think you hit the nail on the head with what i was after.  I've been trying to work out how far back to backtest and then test the algorithm on (as you say) out-of-sample data.  BUT, the key point you made was to train the system on in-sample data, and then test how good it is on out-of-sample data   that  is after (MORE RECENT data)  than the in-sample data.  This does sound obvious now, but something i overlooked.  thanks.
> 
> ...




Hi Daniel -- If you are looking to trade once or twice a day, you will get 300 or so trades per year.  Depending on your system, and what patterns it is looking for, and how much back data it needs to get synchronized, you probably only need two or years or so of intra-day bars.  You can determine how much data is too little and how much is too much by some single backtest runs.  If your system is stable, the length of the in-sample period should remain fairly constant, but you can check several time periods a few months apart to see if it appears to be the same.  Once you know how much in-sample data you need, run automated walk forward tests to see whether the system trades well out-of-sample.  Be sure the objective function you use is one you understand and like.

Thanks,
Howard


----------



## howardbandy (13 September 2009)

Tradesurfer said:


> Polar Bear,
> 
> Exact time vs conditions presented are two different things. To explain, say I tested a long/short weekly trend based system.(actually used one) The system showed a short entry December 2007 on the s&p 500 and then closed out the short a few months ago and went long.




Hi Derek --

That was an excellent trade.  One comment I would make related to systems like yours, backtesting, and validation is that it is almost impossible to get adequate out-of-sample data to statistically validate a trading system that holds positions for 18 months.  So tread very cautiously.

Thanks,
Howard


----------



## howardbandy (13 September 2009)

Wysiwyg said:


> I am very much in the learning stage of this but the idea of taking a ten year block (or longer) and breaking that up into
> 
> 1) full ten year backtest
> 2) two, five year backtests
> ...




Hi Wysiwyg --

All the backtests you are describing are in-sample.  In-sample testing is important in order to determine the best logic and parameter values.  But in-sample results have no value in estimating the future performance of the system.  For that you need out-of-sample tests.  

A better use of ten years of data is seven runs.  Each uses three years of data to pick parameters, then tests the following one year.  Step the whole thing forward one year and do it again.  Seven tests, each giving one year out-of-sample results.  Analyze the concatenated out-of-sample results to see if the model learned the data and the system is tradable.

Thanks for listening,
Howard


----------



## daveM (20 September 2009)

Howard......

please...

You mention doing 3 years and overlapping...... and I assume that this is the approach that you have developed in your experience.

My question is..... is the three years to allow for the one year overlap.....?  and. if you had data for 20 years. would you also use the three year method..............?

I am very new to this and am attempting to grasp the 'in sample/walk forward etc' concepts.

Thanks...


----------



## howardbandy (20 September 2009)

daveM said:


> Howard......
> 
> please...
> 
> ...




Hi Dave --

I used three years as the length of the in-sample period and one year out-of-sample as examples.  The amount of data and the length of time for your system is something you will have to determine yourself -- there is no single guideline.  

When I say system, I am describing a model -- the mathematical algorithm or pattern recognizer -- in combination with the data -- the price and volume of the issue you want to get your buy and sell signals from.

The in-sample period and data is used so that the system can get itself synchronized -- the model "learns" to recognize the pattern you have coded into the trading system program.  But that data consists of both signal -- the part that you want it to learn -- and noise -- everything in the data that you are not explicitly looking for.  Your hope is that the model will recognize the signal and adjust its parameters while processing the in-sample data, and that the signal portion of the data continues to act the same way for a long enough period of time for you to make money trading in the out-of-sample period.  If you do one in-sample / out-of-sample test, you get one data point -- where the data point is the transition from development to trading.  When you run a walk forward test, you set up a sequence of IS / OOS runs, each one of which gives you an example of what happens when you move from development to trading -- each of those transitions is a data point.  The more data points you have, the better able you are to estimate the robustness and profitability of your system.  

So -- it is up to you to describe your idea to yourself, code it into your trading system development platform (AmiBroker, MetaStock, ...), work with it to determine how long the ideal in-sample period is.  After you have figured that part out, run an automated walk forward test and evaluate the likelihood that your system will be profitable when you trade it based on the out-of-sample results.  Walk forward testing is the equivalent of "blind" testing in pharmaceutical development.  I know of no other way to gain the confidence needed to make real trades with real money.

Thanks for listening,
Howard


----------



## daveM (21 September 2009)

Thanks for that, Howard.


----------



## fapturbo (27 September 2009)

Past Performance does not guarantee future performance... guess there's only one way to find out.


----------



## tech/a (27 September 2009)

I think we all have one of those.


----------



## Wysiwyg (27 September 2009)

tech/a said:


> I think we all have one of those.




Smokin`   






.


----------

