Australian (ASX) Stock Market Forum

Systems testing

Joined
5 January 2006
Posts
4,461
Reactions
1
Market data pre-1992 is not supposed to be useful for backtesting trading systems due to the change in market dynamics that began in July 1992 caused by the introduction of compulsory superannuation (provided constant influx of money into the markets. I got this info from Richard Dale, who works at Premium data.

So -- my question is -- how do i backtest my system over a bearish period when there was none from 1992 until now!

In this time, which is almost 15 years, we have had only 2 negative years, 2 sideways years, and our most "bearish period" was from 2000-2003 where we have had 2 years sideways (0% appreciation) and 1 year down LOL.

Any thoughts/ideas/discussion??
 
The 2000 Tech crash, Sep 11 and 2002 are all viable testing periods for a system. Run your testing from 1997 through till now, add in a small margin or error and you should be ok.

Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.
 
The 2000 Tech crash, Sep 11 and 2002 are all viable testing periods for a system. Run your testing from 1997 through till now, add in a small margin or error and you should be ok.

Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.

Thanks Nick, i appreciate your thoughts.

If you dont mind me asking -- the Hedge fund you once ran -- was that purely mechanical?
 
The 2000 Tech crash, Sep 11 and 2002 are all viable testing periods for a system. Run your testing from 1997 through till now, add in a small margin or error and you should be ok.

Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.

Nick,

Who has been your biggest influence with regard to system design and testing? Or best 3?
 
Niz,

Great topic. I've wondered about this myself and played around with some systems on market data across the periods that Nick mentions. Another issue with testing on such old data in the share market is survivorship bias.

Is it viable to test across other markets?
 
The question of survivorship.
One I have pondered for hrs.

I have not culled my data for 4 yrs.
So have atleast all the delists for the last 4 yrs in 7 bourses.
That helps.
I trade the ASX 300 (Well a margin list predominantly made up of),which to some degree minimises the problem.

While always a risk If I'm only trading 10% of my account on anyone trade then There is that minimisation.
Anything I systems trade is pretty heavily tested so I have a good idea of overall expected performance based on 1000s of portfolios.
By this I mean Ill know if one or 2 of these event appear and turn an otherwise profitable method into a loser. From testing on mine--it doesnt.

There is much to developing and designing a system.

When you REALLY REALLY understand these words---and you'll get that from experience rather than comprehension---then you'll have the Ahh of Ahh moments.

Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.

The developement and application through developement becomes a breeze.

THEN look beyond that which you have developed to how a system which is average can be made well above average!

T/T returns about 35% compounded annually---so how is it that it has returned 1300% on initial funds in 4.5 yrs?
 
nizar,
The Reefcap fund ran a mechanical system but we did have a discretionary over ride.

Snake,
Most inspiration comes from looking at the long term trading records of various managers and analysing their journey and knowing full well that its to be expected. A site such as www.iasg.com shows hundreds of old and new managers (agreeably trading futures) but the concepts ring true nonetheless. Someone like Eckhardt has his 25-year record on display. We know he's a great trader, but taking a look at his results you can see his journey. By accepting this as "the norm" for a great trader, then we should also accept it as ours as well. I think this was a defining moment for me. Too many people take a money making machine such as a trend following system and attempt to tweak it to the market conditions. Instead, they should simply stick with it. I ran a trend following system for clients into the 2002 bear market. Many clients closed their accounts because their equity curves stopped moving higher. Along comes 2003 and it becomes downright frightening how much money it has cost them. People accept that their property valuations may stagnate and even decline in certain years. Why do they think differently when trading?

ASG,
I think survivorship bias is an overstated risk when looking at stock systems, especially trend following systems. There are simply too many trends and too many opportunities that exist to say that a couple would have be the defining "make or break" trade for any particular system. Even using a system and taking Davnet trades in the historic testing is acceptable for a trend following method. Why? Because those same situations still appear today, just in other forms. We're not relient on a "single" event or "minimal sample" of those events that it will corrupt the system logic into the future. Afterall, trends must occur. They can't NOT occur. Having a system that finds those trends, stays with them if they continue and removes a position if they don't will always do well over the longer term. It also takes a small glimpse at the XAO back to 1875 to understand the concept of trends.

If however you have a "stock" specific system or a small sample universe that yoru system will only work on, then yes, survivorship bias will be a major threat. It doesn't mean that you shouldn't trade it, but you should be constantly researching other candidates. As an example, I have a system that trades ADR based stocks quite well. Problem is there are only a few of them. Such as system is prone to survivorship bias. Same with a system that only seems to work on one stock. There used to be some very good futures systems that worked well on NWS. When NWS switched identites that system imploded.

This is why I continue to harp on about understanding why the system will make money. Is it making money because some short term anomaly or is making money from a fundamental foundation within the market itself. My EOD trend following systems also work on 1-minute data because the simple reason is that trends can be found in EOD timeframes all the way down to 1-minute timeframes. Trends are a fundamental foundation of the market. They will always exist.

Another foundation is volatility. The market will travel through periods of low volatility followed by high volatility. This is a fundamental foundation that can be relied upon into the future.
 
I think one needs to be wary if backtesting a universe like the ASX200 or ASX300 that many of the stocks in there now may not have been in there some years ago, and this can dramatically affect results.

For example, ZFX is now in the ASX200 but (if I recall correctly) hasn't been for all that long. If you backtest the ASX200 and your system picks up ZFX at $2 back in 2004 and is still holding it, then it will show a very healthy profit. However, assuming ZFX wasn't in the ASX200 back then (even if it was, it's only an example), this is an unrealistic gain for the system because it's buying stocks with proven gains (otherwise it wouldn't be in the ASX200 now) before they could possibly have been bought by a system with the ASX200 as its universe.

Just something to be wary of.

Cheers,
GP
 
Nick,

Obviously any system is subject to variance in performance which results in equity curve movements away from the mean.

At which point do you deferentiate this variation from a change in the underlying market state which either temporarily or permanantly alters the effectiveness of the system?
 
I think one needs to be wary if backtesting a universe like the ASX200 or ASX300 that many of the stocks in there now may not have been in there some years ago, and this can dramatically affect results.

If I understand correctly GP, you are saying that if you allow your universe to be some present day version of the ASX300 list, and in that list are shares who just a few years ago were not in the ASX300, but are now in there due to their growth, that your backtesting could pick those shares up and ride them all the way into the ASX300. This would arguably be the time when their best gains occured, so your results would be overly favourable in backtesting...or?
 
I think one needs to be wary if backtesting a universe like the ASX200 or ASX300 that many of the stocks in there now may not have been in there some years ago, and this can dramatically affect results.

For example, ZFX is now in the ASX200 but (if I recall correctly) hasn't been for all that long. If you backtest the ASX200 and your system picks up ZFX at $2 back in 2004 and is still holding it, then it will show a very healthy profit. However, assuming ZFX wasn't in the ASX200 back then (even if it was, it's only an example), this is an unrealistic gain for the system because it's buying stocks with proven gains (otherwise it wouldn't be in the ASX200 now) before they could possibly have been bought by a system with the ASX200 as its universe.

Just something to be wary of.

Cheers,
GP

Personally I think your all missing the point re backtesting.

This is the point.

This is why I continue to harp on about understanding why the system will make money.

Think about this a bit.
A great system will work on any stock in the conditions in which it is designed to operate,provided it meets that systems criteria.
If the stock isnt in the index now but still held in a system portfolio who cares.If a stock is removed from an index today and another added today,what does it matter.The system if it has a positive expectany will make money on ANY stock which fits its criteria and conforms to the conditions in which the system is designed.

I have the original results of the original backtesting of Techtrader 5 yrs ago.
The results are within the range of the results from a systems test based upon TODAYS universe.
The results of the system traded on the net are also within that range.
So is the one I trade.
The original universe with which the method was designed and tested on is different to todays.
BUT-----why the system will make money. Remains very much the same!
 
Lets see if I can make this even clearer.

Why will a system make money?

Is it the ENTRY?
The EXIT
The STOP
The UNIVERSE
The CAPITAL BASE
The MARKET CONDITIONS
ALL OF THE ABOVE.
The DEVELOPER?

I'll go further.
How is it that a system that returns on average 35% P/A Compounding.
Returns 1300% in 4.5 yrs?

Is that ALL OF THE ABOVE.?

Well if its not what is it?

Its one of only 3 possible things.
 
How is it that a system that returns on average 35% P/A Compounding.
Returns 1300% in 4.5 yrs?

Is that ALL OF THE ABOVE.?

Well if its not what is it?

Its one of only 3 possible things.
Doh... leverage?
 
tech/a said:
If the stock isnt in the index now but still held in a system portfolio who cares.
It can make a big difference. The performance of a stock before it's large enough to be in the index could be considerably different to afterwards. If you design a system with the ASX200 as your universe, then you'll only ever pick for real trading stocks that are in it now. The character of that stock could be totally different to when it was smaller, and while your backtest could pick that stock in the earlier period and perhaps take huge gains on it, you couldn't do that now because those similar huge gains mightn't ever occur in a typical ASX200 stock.

And you're also subject to another type of survivorship bias, not from stocks that delist, but rather from stocks that have proven performance over the years to have managed to get into the index at all. By using the index as your universe but backtesting back to periods before some of the stocks would have been in there, you're choosing stocks from a time when they would have been like many other equally choosable stocks that ultimately didn't perform well enough to make it into the index, but you don't have the risk of picking any of those low performers back then because they're not in your current universe.

If your filter criteria cover all possibilities so that you really wouldn't have chosen the subsequent low performers, then your system should perform equally well on the all-ords as the ASX200.

Cheers,
GP
 
theasxgorilla said:
If I understand correctly GP, you are saying that if you allow your universe to be some present day version of the ASX300 list, and in that list are shares who just a few years ago were not in the ASX300, but are now in there due to their growth, that your backtesting could pick those shares up and ride them all the way into the ASX300. This would arguably be the time when their best gains occured, so your results would be overly favourable in backtesting?
Basically, yes.

If the system picked up a current ASX300 stock 10 years ago, particularly a long-term investing system, then you would expect good gains because you already know it's in the ASX300 now. Some other stock that looked very much the same 10 years ago but ultimately didn't perform, while it could well have been picked in reality 10 years ago, it can't be picked in your backtest now because it's not in the universe. So there's a bias towards picking known good performers.

Cheers,
GP
 
GP,
I disagree with you 100%. I am more than willing to prove my point but will it take a an extended post. Perhaps later this weekend.

Regards
Nick
 
julius,
Sorry I missed your question.

I have no definitive answer for you. Personally I use instinct and experience. I appreciate I can't offer that to you though. I have only ever turned off a few systems, one of which was quite recently. I turned it off because the equity curve was starting to fade. It was not losing money, but the rate of ascent was rolling over to such an extent that there was obviously something wrong. With that I tweaked the system with filters (as opposed to optimizing) and am now taking a cautious "watch and see" approach.

In my book Everyday Traders, one of the traders used an equity curve stop. Personally I'm not convinced of that method and have never used it myself. However, he, and many who have read the book, have adopted it successfully. My feeling is that a system that has a long term positive expectancy should be turned on when in drawdown, not turned off, assuming that the system is robust enough to carry on eventually.

When running the Reefcap fund we had a major investor wait 18-months before he finally invested. He was waiting for the drawdown to come!. I guess its no different to waiting for CBA or ANZ to have that major dip. Kerry Packer was the biggest buyer through November 1987. Buying QBE in late September 2001 was possibly the greatest contrarian trade this decade. The same applies for a robust trading system.
 
Nick,

It's an area I am unsure about and find difficult to stomach when attempting a 'systematic' approach to the development process.

At which point do we decide a coin is bias? Same problem except probabilities are only estimations in this case.

I'v read some application of regression to equity curves, but my populations of trades are always too small to infer anything meaningful.

It's definitely an area I think would be interesting on lower time frames.

Thanks for your reply.

ps. read the book
 
Nick,

Snake,
Most inspiration comes from looking at the long term trading records of various managers and analysing their journey and knowing full well that its to be expected. A site such as www.iasg.com shows hundreds of old and new managers (agreeably trading futures) but the concepts ring true nonetheless. Someone like Eckhardt has his 25-year record on display. We know he's a great trader, but taking a look at his results you can see his journey. By accepting this as "the norm" for a great trader, then we should also accept it as ours as well. I think this was a defining moment for me. Too many people take a money making machine such as a trend following system and attempt to tweak it to the market conditions. Instead, they should simply stick with it. I ran a trend following system for clients into the 2002 bear market. Many clients closed their accounts because their equity curves stopped moving higher. Along comes 2003 and it becomes downright frightening how much money it has cost them. People accept that their property valuations may stagnate and even decline in certain years. Why do they think differently when trading?

Thanks for the link and comments and perspective.

Regards
Snake
 
Top