Is there a point to backtesting?

[PolarBear]

Is there a point to backtesting and if so for how long?

This is my question. It may seem like a silly question, but i'll explain how i've come to this dilemma.

I've been backtesting automated strategies for about a year now on ES (emini s&p 500) 1 minute daily data. I'm using ninja trader, which is good for me as it uses c#.net as it's programming environment, which i've had a lots of experience with so i've been able to test hundreds if not thousands of approaches in the time....

At the start i began just using a few years worth of data, however what i found was that i was able to optimise a particular strategy for that time frame, however when i let it run over the full 10 years worth of ES data the result would be overall negative over the time frame.
Intuitively, I put this down to the problem of overfitting the test data - ie: the algorithm learns how to run on the training set data, but is not good for data unseen (in this case over the different timeframes).

I then began trying to develop algorithms that would work over the entire 10 years of the data.

Recently, i contacted a developer of a "system" which i'll let remain nameless, which i backtested and found that it didn't seem to work. After contacting him I got a bunch of fairly abrupt responses, but his view was mainly summarised by the following:
"First of all going back to 1998 is really not the proper way to test any system. its anti productive. Markets were different, volatility was different and you will never find anything that will work into the future based on a great past track history,"

This seems interesting to me. I would like to know what everyone thinks about this.
My thoughts are :
1) Backtesting is productive, allowing for development and refinement of algorithms based on real data
2) I agree that markets are different in the past, however I was trying to work towards a strategy that would work accross these changing markets, for exmple, using volatility... - is this not a good approach to aim for?

So in essence i think backtesting is valuable, but i'm now questioning whether it's a waste of time to find an approach that works on old historical data that maybe is useless..... If backtesting is good to do, what timeframe should be used - ie: just use say the last 2 years worth of ES data??

thanks.
-d

 

[Timmy]

Not really into backtesting but I know plenty are. However, I trade the ES and there are many, many algos and bots active in it. These automated strategies must be effective, so hang in there with it.

 

[Mr J]

At the start i began just using a few years worth of data, however what i found was that i was able to optimise a particular strategy for that time frame, however when i let it run over the full 10 years worth of ES data the result would be overall negative over the time frame.

Of course, they're unprofitable strategies. A system being profitable over the data on which it was develop is to be expected. That data should also then be ignored for testing.

Intuitively, I put this down to the problem of overfitting the test data

Maybe, but I'd just go with the strategy not being a profitable one.

I then began trying to develop algorithms that would work over the entire 10 years of the data.

So, fitting it over those 10 years of data, but then you have no data on which to test it?

Markets were different, volatility was different and you will never find anything that will work into the future based on a great past track history,"

I would think it depends on the strategy. I'm sure mine would have worked 10 years ago, and will work 10 years from now.

 

[skyQuake]

Do some research into system backtesting in general.

A system lasting a year on the S&P would be fantastic. Edges are gained and lost all the time. What seems to be key now is a System that develops profitable systems, and can phase old stuff out and new stuff in.

 

[Nick Radge]

If you have over optimized or used too many parameter sets you will usually fall short of successful strategy trading. Usually 5 parameter sets is considered the upper limit, where less is better.

Remember the goal is to find a system that works reasonably well all of the time rather than a system that works brilliant some of the time.

Also as a rule of thumb, if you can identify WHY the system should make money then you will (a) tend to need less backtesting and (b) be reasonably certain that it will make money.

Nick

 

[PolarBear]

Hi all, thanks for the quick interesting responses.... No-one's really got to the heart of my question though, which was: is backtesting worthwhile, and if so how long into the past is the data valid.
Or another way of saying it, as an example, how long into the past would you backtest a strategy on say the ES, so that you can be confident (that word is going to cause some backlash...) of trading that strategy profitably on the real market.

thanks again.
-daniel

 

[motorway]

Is there a point to backtesting and if so for how long?

2) I agree that markets are different in the past, however I was trying to work towards a strategy that would work accross these changing markets, for exmple, using volatility... - is this not a good approach to aim for?
thanks.
-d

Suspect this is a very wrong road to go down
When you filter with (or correct for ) volatility
You end up with just so much bath water
( ie the very volatile baby is chucked out as if it was never there)

Just an observation...
what ever you use
make sure that the screens are objective

what is signal what is noise ?
often the noise is the signal

IE If you correct for volatility
You are simply filtering the signal out
and amplifying NOISE

motorway

 

[howardbandy]

Hi all, thanks for the quick interesting responses.... No-one's really got to the heart of my question though, which was: is backtesting worthwhile, and if so how long into the past is the data valid.
Or another way of saying it, as an example, how long into the past would you backtest a strategy on say the ES, so that you can be confident (that word is going to cause some backlash...) of trading that strategy profitably on the real market.

thanks again.
-daniel

Hi Daniel --

Backtesting is worth doing. By backtesting, you find out whether your system worked sometime in the past. Most people want to get some idea of how it might work in the future.

So it is the testing on data that was not used in backtesting that is important. Testing on data that was not used before is called out-of-sample testing.

The data and time period used for the backtesting is called in-sample data. Results over the in-sample period are always good -- we do not stop fooling with the system until they are good. But in-sample results have no value in predicting future performance.

The goal is to gain confidence that trading the system tomorrow, with real money, is likely to be profitable. But the best way I know how to gain that confidence is to practice. The walk forward testing procedures give the practice. Each step forward shows a transition from backtesting in-sample to forward testing out-of-sample. Even using walk forward testing gives no guarantee, only an estimate based on statistical procedures.

There are extensive discussions of the topic of backtesting in this forum. Search on my name, howardbandy, and you will find several threads.



Thanks for listening,
Howard

 

[Tradesurfer]

When I backtest a new system I make sure and test during periods opposite to its design. In other words, I test trend based systems during periods I know not trending but sideways and vice versa.

If you are testing a system that will profit when conditions occur and stay out when the are not occurring that it would seem a valid exercise. I would argue as a trend trader that trends happen again and again- in up and down markets and trading trends will ensure that one is on the right side of the market.

One thing to be careful of is the survivorship bias - this simply refers to say testing the s&p 500 index on a particular strategy. If you are testing a short strategy on the current index, you wouldn't get the benefit of being short say Lehman brothers, bear stearns ,enron, etc etc.

 

[daveM]

One thing that is derived from backtesting is the practice of looking at the trading system and the charts very carefully to find entries and exits and the performance of the price.

That study alone is well worth the effort, it strengthens a person's discretionary entries.

I am a novice regarding trading system development.... I have found that I am scrutinizing the chart a lot more to find answers... surely that will help me as time passes...

That is just one small benefit of the exercise..

 

[theasxgorilla]

When I backtest a new system I make sure and test during periods opposite to its design. In other words, I test trend based systems during periods I know not trending but sideways and vice versa.

I also do this, but the system I operate is designed to survive these periods. Others I know of turn their system on and off at their own discretion when they identify that market conditions are not ideal.

As Nick has mentioned, the fewer parameters the better, but if you want the system to survive on it's own you probably need more parameters to keep the system out of unsuitable market conditions.

 

[Cartman]

When I backtest a new system I make sure and test during periods opposite to its design. In other words, I test trend based systems during periods I know not trending but sideways and vice versa.

well i dont backtest anything as such .... always test systems with real money in a real market ... BUT .... your advice above T/Surfer (for punters who like to test stuff) is about as logical advice as i've seen (and exactly what i would do ...

if u r gona see if something works, test it in unfavorable market conditions ... if it goes close to break even under those conditions ... might be time to mortgage the house

 

[daveM]

I also do this, but the system I operate is designed to survive these periods. Others I know of turn their system on and off at their own discretion when they identify that market conditions are not ideal.

As Nick has mentioned, the fewer parameters the better, but if you want the system to survive on it's own you probably need more parameters to keep the system out of unsuitable market conditions.

Quick question....... when you mention 'fewer parameters'........ how many would you start with before fine tuning begins.......?

Thanks

 

[tech/a]

When I backtest a new system I make sure and test during periods opposite to its design. In other words, I test trend based systems during periods I know not trending but sideways and vice versa.

While this seems totally logical to some its totally foreign to me.
Why would I try and fly a car or row a bus.
Why on earth would I expect something to perform well in an environment in which it wasnt designed?

Of course there is a point in backtesting.
This is the equity curve for a system which has been traded for many years live on the net.Note the flat period of late where it has been switched off!

 

[MichaelD]

Others I know of turn their system on and off at their own discretion when they identify that market conditions are not ideal.

This is the equity curve for a system which has been traded for many years live on the net.Note the flat period of late where it has been switched off!

Something that occurred to me a while ago (and which is borne out by backtesting) is that switching systems on and off has severely adverse effects on their performance.

It may look and feel good to have locked in the returns during the good times, but the next equity peak has been limited as a direct result of these actions.

 

[tech/a]

Something that occurred to me a while ago (and which is borne out by backtesting) is that switching systems on and off has severely adverse effects on their performance.

It may look and feel good to have locked in the returns during the good times, but the next equity peak has been limited as a direct result of these actions.

Yes your right.
With $300+K at risk I just couldn't do it.
Back into it now and without the 27% draw down the testing has subsequently shown.

I prefer my psychological damage to be seemingly controlled.

 

[theasxgorilla]

Something that occurred to me a while ago (and which is borne out by backtesting) is that switching systems on and off has severely adverse effects on their performance.

It may look and feel good to have locked in the returns during the good times, but the next equity peak has been limited as a direct result of these actions.

I doubt this is correct. Given a system that produces enough entries to use more than all the available capital, and thereby generate multiple future trading paths, luck will determine whether you get on one that has an equity peak which exceeds any other.

 

[theasxgorilla]

Quick question....... when you mention 'fewer parameters'........ how many would you start with before fine tuning begins.......?

Thanks

I think I started with 5 or 6 and ended up with 11. Parameters added included a volatility filter which prevents the system jumping on extreme breakouts, low-side entry price filters which prevents the system trading stocks priced below a certain level and rate of change filters, to frame entry into stocks that have gone up enough, but not too much in given period of time.

It's all in aid of smoothing the equity curve and making the system easier to live with, which ultimately comes from exposing the system to the right market conditions (it's a trend following system). And when your universe is the ASX you don't need to jump on everything that moves.

 

[theasxgorilla]

While this seems totally logical to some its totally foreign to me.
Why would I try and fly a car or row a bus.
Why on earth would I expect something to perform well in an environment in which it wasnt designed?

It's an extreme analogy, but I get it. A better one might have been to say that a bus wasn't designed to drive off-road in the same manner as a 4x4, but it can, provided the conditions aren't too extreme, albeit with degraded performance. Drive an F1 off-road and you'll destroy it pronto.

So it really depends on the kind of system we're talking about I think. Some are more forgiving than others, by design.

 

[MichaelD]

Luck will determine whether you get on one that has an equity peak which exceeds any other.

Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.

Your return is proportional to your risk. Take more risk, get more return. Smoothing one's equity curve with filters, limiting portfolio heat, stepping aside when conditions are "unfavourable" and all other such mechanisms decrease absolute system return in the long run.

You trade absolute gain for less psychological pain.

eg. Let's make a presumption - that we are seeing the beginnings of a new bull market. It may be, it may not be. Eventually, the type of scenario we have seen played out will turn into the next bull market.

Many traders are still equivocating. Various have posted here that they have begun to take entries, tentatively, or will be soon.

I have followed my system's signal all along. I have many fully pyramided positions, mostly starting near the "bottom" of this current market, and quite a few of them have trailing stops now way above the entries. I'm way ahead of those that remain on the sidelines or are dipping their toes in.

This is not because I cleverly picked the bottom of the market. This is because optimal performance of long term trend following does not presume to pick a market bottom, but by its natural operation it will be participating at the bottom.

Q: When do you get the lowest risk entries for a long term trend following system?
A: At the bottom of the market.

Q: How do you know you've reached the bottom of the market?
A: You can't know this until after the fact, when it's too late to take advantage of it.