Is there a point to backtesting?

theasxgorilla · 26 August 2009

MichaelD said:
Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.

I see we have an interesting discussion on our hands.

So let's deal with a system whose signals are generated by the same mechanism with the same money management/risk control parameters. Make one instance of the system take all signals, let the other take 95% of signals by skipping a proportion of signals at random, and let another instance of the system be turned off and on by a trader according to their gut feel about where the market is at, top, bottom, good or bad conditions etc.

Are you saying that there is no chance the later two instances can beat the former, in terms of absolute gains, over any given period of time?

wayneL · 26 August 2009

MichaelD said:
Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.

I'm with G. MM and risk control is of course paramount, but never underestimate the role of luck.

A simple illustration of this can be conducted in Excel and a random number generator.

The system is a simple zero expectancy system (50% chance of win - 1:1 risk/reward ratio) risking 10% fixed fractional risking 100 times

Results ranged from 90% loss of capital to 700% gain over 100 iterations.

A positive expectancy system might virtually guarantee a return over the long haul, but the real time results of two traders faithfully using the same system will diverge markedly.

Hell! Even Monte Carlo testing shows this to be true also.

What is the difference? Plain old dumb luck.

Denying this is so in face of overwhelming evidence, even from the backtesting system traders rely on seems ludicrous.

ducati916 · 26 August 2009

MichaelD said:
Quite untrue. Luck has nothing to do with it. Money management and risk control has everything to do with it.

Your return is proportional to your risk. Take more risk, get more return. Smoothing one's equity curve with filters, limiting portfolio heat, stepping aside when conditions are "unfavourable" and all other such mechanisms decrease absolute system return in the long run.

You trade absolute gain for less psychological pain.

eg. Let's make a presumption - that we are seeing the beginnings of a new bull market. It may be, it may not be. Eventually, the type of scenario we have seen played out will turn into the next bull market.

Many traders are still equivocating. Various have posted here that they have begun to take entries, tentatively, or will be soon.

I have followed my system's signal all along. I have many fully pyramided positions, mostly starting near the "bottom" of this current market, and quite a few of them have trailing stops now way above the entries. I'm way ahead of those that remain on the sidelines or are dipping their toes in.

This is not because I cleverly picked the bottom of the market. This is because optimal performance of long term trend following does not presume to pick a market bottom, but by its natural operation it will be participating at the bottom.

Q: When do you get the lowest risk entries for a long term trend following system?
A: At the bottom of the market.

Q: How do you know you've reached the bottom of the market?
A: You can't know this until after the fact, when it's too late to take advantage of it.

And herein lies the legitimate fear of systems traders. You assume a bull market will eventuate. What if - it doesn't?

There are reasons, macroeconomic, why since the 1920's we have an upside bias. However, if those conditions were removed, would the bias [upside] remain?

jog on
duc

theasxgorilla · 26 August 2009

ducati916 said:
And herein lies the legitimate fear of systems traders. You assume a bull market will eventuate. What if - it doesn't?

There are reasons, macroeconomic, why since the 1920's we have an upside bias. However, if those conditions were removed, would the bias [upside] remain?

System or not, the first question ought to be, should I be in equities? You must have a view on this before you even invest (well-informed or otherwise)... beyond that point you're as committed as you are. It's the same for anyone holding positions in equities, isn't it?

ducati916 · 26 August 2009

theasxgorilla said:
System or not, the first question ought to be, should I be in equities? You must have a view on this before you even invest (well-informed or otherwise)... beyond that point you're as committed as you are. It's the same for anyone holding positions in equities, isn't it?

asx

The prevailing bias, as exemplified by MichaelD, seems to be over time, there is an upside bias to equities. Thus a long based system, with appropriate MM, should have a positive expectancy.

What if - that underlying macroeconomic factor driving the [agreed] current bias were removed?

I think that unconsiously many systems traders, unaware of the factor consiously, fear this potential variable.

jog on
duc

bingk6 · 26 August 2009

MichaelD said:
Smoothing one's equity curve with filters, limiting portfolio heat, stepping aside when conditions are "unfavourable" and all other such mechanisms decrease absolute system return in the long run.

In terms of absolute system returns, I agree with you. However, the theoritical benefit of doing so is the reduction in drawdowns, at a greater % than the % reduction in absolute returns. So for somebody who uses some form of leverage, it is quite possible to achieve superior absolute returns and lower drawdowns than an unfiltered system.

Grep · 26 August 2009

MichaelD said:
Your return is proportional to your risk. Take more risk, get more return.

This is not true.The relationship between risk and reward is not linear.
There is an efficient frontier beyond which increasing risk guarantees failure.

Aussiest · 26 August 2009

PolarBear said:
1) Backtesting is productive, allowing for development and refinement of algorithms based on real data
2) I agree that markets are different in the past, however I was trying to work towards a strategy that would work accross these changing markets, for exmple, using volatility... - is this not a good approach to aim for?

So in essence i think backtesting is valuable, but i'm now questioning whether it's a waste of time to find an approach that works on old historical data that maybe is useless..... If backtesting is good to do, what timeframe should be used - ie: just use say the last 2 years worth of ES data??

thanks.
-d

You only have to study the market for 1+ year to see that there are significantly different time chunks with different variables, which would affect trading.

Eg, Around 2007 - into 2008: market was very volatile. Earlier this year (feb, march), volatility slowed down and market begun a bull run (bull run in bear market), and while the market rose, volatility slowed down. I have noticed over passed 2 weeks that volatility has increased again, thereby creating a better environment for day traders.

I would assume you would develop different strategies for different timeframes, esp. if trading intraday (upward bias is lost in a bear market).

brty · 26 August 2009

Duc,

there is an upside bias to equities.

This bias is in the index, not individual stocks. There are precious few stocks that remain from the indexes of the '20's or ''30's. As the losers get dumped out of indexes, and the up-commers get included, this is where the bias lives.

The classic example of this is the ETF STW based on the ASX200. When this was first listed in late April '08 opening price was ~$55.20 while the ASX200 was at ~5,520. Today however STW is at ~$41.57 while the ASX200 is ~4,416.

In other words the fund that "mimicks" the index has managed to have a performance nearly 6% below the index in 15 months.

brty

Boggo · 26 August 2009

This is a chart of a TradeSim backtest of my breakout system.

It is a good example of how a breakout system cannot handle a negative market and proof that you need to park the Porsche when the market goes cross country which is what tech/a is saying.

I stopped trading this system in Dec 2007 but it is well and truly back up and running at the moment.

In reality though when a market turns down you really don't need to be Einstein to work out that a breakout long system is not going to work.

(the start date and the start equity on the right are the actual starting points and the chart was produced in May 09)

(click to expand)

tech/a · 26 August 2009

I agree with Michael.
If you turn discretionary on your system then your not trading your system.

But then I agree with myself.

"I'm sitting on the plane and have been for many many flights it goes up and down like clock work no problems."
In June 07 I notice the pilot walking around the plane with a worried look on his face.
Whats wrong?
This plane doesnt feel right infact it feels bloody terrible,I'm not happy flying it.
So he calls in all the experts.
We all agree this plane looks like it should be garaged until all is fixed.
So I have a look.
I agree it looks bloody un safe--so I get off the plane.
However the plane gets a new pilot and away it goes.
It crashes.Most survive,some die,I survive.

I hop on another plane with the same positive expectancy I had before.

I dont agree with Duc
Because I just dont agree with him.
Disagreement with Duc usually has an effect.

motorway · 26 August 2009

ducati916 said:
asx

What if - that underlying macroeconomic factor driving the [agreed] current bias were removed?

jog on
duc

Free Markets MUST TREND

just like little streams MUST create Grand Canyons

Participants will Self organize
so that trends always occur
prices will always adjust
such that there will always be bases and breakouts into trends ..

What If the underlying factors changed ?

eg the little stream was stopped from flowing by Soviet Command ?

Then in the fullness of time NO trends ( NO Bias )
and No GrandCanyons

No

Matthew Effects

tech/a said:
All true Motorway
but the point of developing a system is a far cry from trading in a discretionary manner.

My post in that thread does not contain the word Discretionary
My post was about the fact that the universe of what is useful to test
is much smaller than the universe of things that are tested.

hence why "things" seem to be so context driven
yet markets always do just the same things..

the real versus artefacts of the real...

motorway

motorway · 26 August 2009

Boggo said:
This is a chart of a TradeSim backtest of my breakout system.

It is a good example of how a breakout system cannot handle a negative market and proof that you need to park the Porsche when the market goes cross country which is what tech/a is saying.

I stopped trading this system in Dec 2007 but it is well and truly back up and running at the moment.

In reality though when a market turns down you really don't need to be Einstein to work out that a breakout long system is not going to work.

(the start date and the start equity on the right are the actual starting points and the chart was produced in May 09)

(click to expand)

Ok change to this

This system stopped Trading in Dec 2007 but it is well and truly back up and running at the moment.

Of course--->

To be true it didn't stop at all. It just had no entries triggered..

The system never stops even when it is doing nothing..

motorway

MichaelD · 26 August 2009

I won't multi-quote in this reply as I find this tends to get hard to read, but in reply to various points made herein;

The bell curves of possible equity returns between filtered and unfiltered trend following may overlap at the extreme ends, so I concede that some very lucky discretionary turn on/off traders may outperform some very unlucky mechanical traders, but that does not negate the fact that the median return is higher, to a very high degree of certainty.

wayneL · 26 August 2009

MichaelD said:
I won't multi-quote in this reply as I find this tends to get hard to read, but in reply to various points made herein;

The bell curves of possible equity returns between filtered and unfiltered trend following may overlap at the extreme ends, so I concede that some very lucky discretionary turn on/off traders may outperform some very unlucky mechanical traders, but that does not negate the fact that the median return is higher, to a very high degree of certainty.

It might depend on the system. I recall Jose Silva had a filter that switched entries on/off that (allegedly) improved perfotmance. http://www.metastocktools.com/ADT/ADT-ASX.htm

beamstas · 26 August 2009

I have a system with a simple index filter, that doesn't allow new entries when the market filter is "off". It also tightens stops considerably when the filter is "off" on already open positions.

The difference in MaxDD and Absolute return is huge.

daveM · 26 August 2009

theasxgorilla

Thanks for that reply.....

fapturbo · 26 August 2009

Try this code Metastock

(Mov(C,220,S)- Ref(Mov(C,220,S),- 55))/ATR(25)>3
LLV(Mov(C,50,S)*Mov(V,50,S),50)>500000
ATR(10)/Mov(C,10,S)>0.01 AND
ATR(10)/Mov(C,10,S)<0.1

ducati916 · 27 August 2009

brty said:
Duc,

This bias is in the index, not individual stocks. There are precious few stocks that remain from the indexes of the '20's or ''30's. As the losers get dumped out of indexes, and the up-commers get included, this is where the bias lives.

The classic example of this is the ETF STW based on the ASX200. When this was first listed in late April '08 opening price was ~$55.20 while the ASX200 was at ~5,520. Today however STW is at ~$41.57 while the ASX200 is ~4,416.

In other words the fund that "mimicks" the index has managed to have a performance nearly 6% below the index in 15 months.

brty

brty

Incorrect.

There exists a very powerful bias for equity [common stocks] This bias will favour different types of equity depending on the business cycle. Sentiment, fads, innovation can add to this already existing bias.

As to your indices hypothesis, you need to think that one through more carefully.

jog on
duc

brty · 27 August 2009

Duc,

Do you believe that there i no positive bias in the indexes??

Do you believe that if you had bought the basket of stocks that represented the indexes in the '20's or '30's and held until now, your performance would have been the same as the index over that time???

If I'm so wrong, could you please give examples using real numbers??

brty

Is there a point to backtesting?

theasxgorilla

Problem solved... next bubble.

wayneL

VIVA LA LIBERTAD, CARAJO!

ducati916

theasxgorilla

Problem solved... next bubble.

ducati916

bingk6

Grep

Aussiest

brty

Boggo

Attachments

tech/a

No Ordinary Duck

motorway

motorway

MichaelD

Not fooled by randomness

wayneL

VIVA LA LIBERTAD, CARAJO!

beamstas

daveM

fapturbo

ducati916

brty

Similar threads

Is there a point to backtesting?

Problem solved... next bubble.

VIVA LA LIBERTAD, CARAJO!

Problem solved... next bubble.

Attachments

No Ordinary Duck

Not fooled by randomness

VIVA LA LIBERTAD, CARAJO!

Similar threads

Privacy & Transparency

Privacy & Transparency