# Systems testing



## nizar (5 July 2007)

Market data pre-1992 is not supposed to be useful for backtesting trading systems due to the change in market dynamics that began in July 1992 caused by the introduction of compulsory superannuation (provided constant influx of money into the markets. I got this info from Richard Dale, who works at Premium data.

So -- my question is -- how do i backtest my system over a bearish period when there was none from 1992 until now!

In this time, which is almost 15 years, we have had only 2 negative years, 2 sideways years, and our most "bearish period" was from 2000-2003 where we have had 2 years sideways (0% appreciation) and 1 year down LOL.

Any thoughts/ideas/discussion??


----------



## Nick Radge (5 July 2007)

The 2000 Tech crash, Sep 11 and 2002 are all viable testing periods for a system. Run your testing from 1997 through till now, add in a small margin or error and you should be ok.

Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.


----------



## nizar (5 July 2007)

Nick Radge said:


> The 2000 Tech crash, Sep 11 and 2002 are all viable testing periods for a system. Run your testing from 1997 through till now, add in a small margin or error and you should be ok.
> 
> Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.




Thanks Nick, i appreciate your thoughts.

If you dont mind me asking -- the Hedge fund you once ran -- was that purely mechanical?


----------



## It's Snake Pliskin (6 July 2007)

Nick Radge said:


> The 2000 Tech crash, Sep 11 and 2002 are all viable testing periods for a system. Run your testing from 1997 through till now, add in a small margin or error and you should be ok.
> 
> Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.




Nick,

Who has been your biggest influence with regard to system design and testing? Or best 3?


----------



## theasxgorilla (6 July 2007)

Niz,

Great topic.  I've wondered about this myself and played around with some systems on market data across the periods that Nick mentions.  Another issue with testing on such old data in the share market is _survivorship bias_.

Is it viable to test across other markets?


----------



## tech/a (6 July 2007)

The question of survivorship.
One I have pondered for hrs.

I have not culled my data for 4 yrs.
So have atleast all the delists for the last 4 yrs in 7 bourses.
That helps.
I trade the ASX 300 (Well a margin list predominantly made up of),which to some degree minimises the problem.

While always a risk If I'm only trading 10% of my account on anyone trade then There is that minimisation.
Anything I systems trade is pretty heavily tested so I have a good idea of overall expected performance based on 1000s of portfolios.
By this I mean Ill know if one or 2 of these event appear and turn an otherwise profitable method into a loser. From testing on mine--it doesnt.

There is much to developing and designing a system.

When you *REALLY REALLY *understand these words---and you'll get that from experience rather than comprehension---then you'll have the Ahh of Ahh moments.



> Testing is not an exact science. Making it so will never allow you to take a trade. Also understand why your system will make money and when it will and will not. Understand and accept the journey of the system. I think understanding the journey is the key to success.




The developement and application through developement becomes a breeze.

*THEN* look beyond that which you have developed to how a system which is average can be made well above average!

T/T returns about 35% compounded annually---so how is it that it has returned 1300% on initial funds in 4.5 yrs?


----------



## Nick Radge (6 July 2007)

nizar,
The Reefcap fund ran a mechanical system but we did have a discretionary over ride.

Snake,
Most inspiration comes from looking at the long term trading records of various managers and analysing their journey and knowing full well that its to be expected. A site such as www.iasg.com shows hundreds of old and new managers (agreeably trading futures) but the concepts ring true nonetheless. Someone like Eckhardt has his 25-year record on display. We know he's a great trader, but taking a look at his results you can see his journey. By accepting this as "the norm" for a great trader, then we should also accept it as ours as well. I think this was a defining moment for me. Too many people take a money making machine such as a trend following system and attempt to tweak it to the market conditions. Instead, they should simply stick with it. I ran a trend following system for clients into the 2002 bear market. Many clients closed their accounts because their equity curves stopped moving higher. Along comes 2003 and it becomes downright frightening how much money it has cost them. People accept that their property valuations may stagnate and even decline in certain years. Why do they think differently when trading?

ASG,
I think survivorship bias is an overstated risk when looking at stock systems, especially trend following systems. There are simply too many trends and too many opportunities that exist to say that a couple would have be the defining "make or break" trade for any particular system. Even using a system and taking Davnet trades in the historic testing is acceptable for a trend following method. Why? Because those same situations still appear today, just in other forms. We're not relient on a "single" event or "minimal sample" of those events that it will corrupt the system logic into the future. Afterall, trends must occur. They can't _*NOT*_ occur. Having a system that finds those trends, stays with them if they continue and removes a position if they don't will always do well over the longer term. It also takes a small glimpse at the XAO back to 1875 to understand the concept of trends. 

If however you have a "stock" specific system or a small sample universe that yoru system will only work on, then yes, survivorship bias will be a major threat. It doesn't mean that you shouldn't trade it, but you should be constantly researching other candidates. As an example, I have a system that trades ADR based stocks quite well. Problem is there are only a few of them. Such as system is prone to survivorship bias. Same with a system that only seems to work on one stock. There used to be some very good futures systems that worked well on NWS. When NWS switched identites that system imploded. 

This is why I continue to harp on about understanding why the system will make money. Is it making money because some short term anomaly or is making money from a fundamental foundation within the market itself. My EOD trend following systems also work on 1-minute data because the simple reason is that trends can be found in EOD timeframes all the way down to 1-minute timeframes. Trends are a fundamental foundation of the market. They will always exist.

Another foundation is volatility. The market will travel through periods of low volatility followed by high volatility. This is a fundamental foundation that can be relied upon into the future.


----------



## GreatPig (6 July 2007)

I think one needs to be wary if backtesting a universe like the ASX200 or ASX300 that many of the stocks in there now may not have been in there some years ago, and this can dramatically affect results.

For example, ZFX is now in the ASX200 but (if I recall correctly) hasn't been for all that long. If you backtest the ASX200 and your system picks up ZFX at $2 back in 2004 and is still holding it, then it will show a very healthy profit. However, assuming ZFX wasn't in the ASX200 back then (even if it was, it's only an example), this is an unrealistic gain for the system because it's buying stocks with proven gains (otherwise it wouldn't be in the ASX200 now) before they could possibly have been bought by a system with the ASX200 as its universe.

Just something to be wary of.

Cheers,
GP


----------



## julius (6 July 2007)

Nick,

Obviously any system is subject to variance in performance which results in equity curve movements away from the mean.

At which point do you deferentiate this variation from a change in the underlying market state which either temporarily or permanantly alters the effectiveness of the system?


----------



## theasxgorilla (6 July 2007)

GreatPig said:


> I think one needs to be wary if backtesting a universe like the ASX200 or ASX300 that many of the stocks in there now may not have been in there some years ago, and this can dramatically affect results.




If I understand correctly GP, you are saying that if you allow your universe to be some present day version of the ASX300 list, and in that list are shares who just a few years ago were not in the ASX300, but are now in there due to their growth, that your backtesting could pick those shares up and ride them all the way into the ASX300.  This would arguably be the time when their best gains occured, so your results would be overly favourable in backtesting...or?


----------



## tech/a (6 July 2007)

GreatPig said:


> I think one needs to be wary if backtesting a universe like the ASX200 or ASX300 that many of the stocks in there now may not have been in there some years ago, and this can dramatically affect results.
> 
> For example, ZFX is now in the ASX200 but (if I recall correctly) hasn't been for all that long. If you backtest the ASX200 and your system picks up ZFX at $2 back in 2004 and is still holding it, then it will show a very healthy profit. However, assuming ZFX wasn't in the ASX200 back then (even if it was, it's only an example), this is an unrealistic gain for the system because it's buying stocks with proven gains (otherwise it wouldn't be in the ASX200 now) before they could possibly have been bought by a system with the ASX200 as its universe.
> 
> ...




Personally I think your all missing the point re backtesting.

This is the point.



> This is why I continue to harp on about understanding why the system will make money.




Think about this a bit.
A great system will work on any stock in the conditions in which it is designed to operate,provided it meets that systems criteria.
If the stock isnt in the index now but still held in a system portfolio who cares.If a stock is removed from an index today and another added today,what does it matter.The system if it has a positive expectany will make money on ANY stock which fits its criteria and conforms to the conditions in which the system is designed.

I have the original results of the original backtesting of Techtrader 5 yrs ago.
The results are within the range of the results from a systems test based upon TODAYS universe.
The results of the system traded on the net are also within that range.
So is the one I trade.
The original universe with which the method was designed and tested on is different to todays.
BUT-----*why the system will make money.* Remains very much the same!


----------



## tech/a (6 July 2007)

Lets see if I can make this even clearer.

Why will a system make money? 

Is it the ENTRY?
The EXIT
The STOP
The UNIVERSE
The CAPITAL BASE
The MARKET CONDITIONS
ALL OF THE ABOVE.
The DEVELOPER?

I'll go further.
How is it that a system that returns on average 35% P/A Compounding.
Returns 1300% in 4.5 yrs?

Is that ALL OF THE ABOVE.?

Well if its not what is it?

*Its one of only 3 possible things.*


----------



## wayneL (6 July 2007)

tech/a said:


> How is it that a system that returns on average 35% P/A Compounding.
> Returns 1300% in 4.5 yrs?
> 
> Is that ALL OF THE ABOVE.?
> ...



Doh... leverage?


----------



## GreatPig (6 July 2007)

tech/a said:
			
		

> If the stock isnt in the index now but still held in a system portfolio who cares.



It can make a big difference. The performance of a stock before it's large enough to be in the index could be considerably different to afterwards. If you design a system with the ASX200 as your universe, then you'll only ever pick for real trading stocks that are in it now. The character of that stock could be totally different to when it was smaller, and while your backtest could pick that stock in the earlier period and perhaps take huge gains on it, you couldn't do that now because those similar huge gains mightn't ever occur in a typical ASX200 stock.

And you're also subject to another type of survivorship bias, not from stocks that delist, but rather from stocks that have proven performance over the years to have managed to get into the index at all. By using the index as your universe but backtesting back to periods before some of the stocks would have been in there, you're choosing stocks from a time when they would have been like many other equally choosable stocks that ultimately didn't perform well enough to make it into the index, but you don't have the risk of picking any of those low performers back then because they're not in your current universe.

If your filter criteria cover all possibilities so that you really wouldn't have chosen the subsequent low performers, then your system should perform equally well on the all-ords as the ASX200.

Cheers,
GP


----------



## GreatPig (6 July 2007)

theasxgorilla said:
			
		

> If I understand correctly GP, you are saying that if you allow your universe to be some present day version of the ASX300 list, and in that list are shares who just a few years ago were not in the ASX300, but are now in there due to their growth, that your backtesting could pick those shares up and ride them all the way into the ASX300.  This would arguably be the time when their best gains occured, so your results would be overly favourable in backtesting?



Basically, yes.

If the system picked up a current ASX300 stock 10 years ago, particularly a long-term investing system, then you would expect good gains because you already know it's in the ASX300 now. Some other stock that looked very much the same 10 years ago but ultimately didn't perform, while it could well have been picked in reality 10 years ago, it can't be picked in your backtest now because it's not in the universe. So there's a bias towards picking known good performers.

Cheers,
GP


----------



## Nick Radge (6 July 2007)

GP,
I disagree with you 100%. I am more than willing to prove my point but will it take a an extended post. Perhaps later this weekend.  

Regards
Nick


----------



## GreatPig (6 July 2007)

Nick,

Thanks. I'd be interested to see your reasoning.

Cheers,
GP


----------



## Nick Radge (6 July 2007)

julius,
Sorry I missed your question.

I have no definitive answer for you. Personally I use instinct and experience. I appreciate I can't offer that to you though. I have only ever turned off a few systems, one of which was quite recently. I turned it off because the equity curve was starting to fade. It was not losing money, but the rate of ascent was rolling over to such an extent that there was obviously something wrong. With that I tweaked the system with filters (as opposed to optimizing) and am now taking a cautious "watch and see" approach.

In my book Everyday Traders, one of the traders used an equity curve stop. Personally I'm not convinced of that method and have never used it myself. However, he, and many who have read the book, have adopted it successfully. My feeling is that a system that has a long term positive expectancy should be turned on when in drawdown, not turned off, assuming that the system is robust enough to carry on eventually. 

When running the Reefcap fund we had a major investor wait 18-months before he finally invested. He was waiting for the drawdown to come!.   I guess its no different to waiting for CBA or ANZ to have that major dip. Kerry Packer was the biggest buyer through November 1987. Buying QBE in late September 2001 was possibly the greatest contrarian trade this decade. The same applies for a _robust_ trading system.


----------



## julius (7 July 2007)

Nick,

It's an area I am unsure about and find difficult to stomach when attempting a 'systematic' approach to the development process.

At which point do we decide a coin is bias? Same problem except probabilities are only estimations in this case.

I'v read some application of regression to equity curves, but my populations of trades are always too small to infer anything meaningful. 

It's definitely an area I think would be interesting on lower time frames.

Thanks for your reply.

ps. read the book


----------



## It's Snake Pliskin (7 July 2007)

Nick,



> Snake,
> Most inspiration comes from looking at the long term trading records of various managers and analysing their journey and knowing full well that its to be expected. A site such as www.iasg.com shows hundreds of old and new managers (agreeably trading futures) but the concepts ring true nonetheless. Someone like Eckhardt has his 25-year record on display. We know he's a great trader, but taking a look at his results you can see his journey. By accepting this as "the norm" for a great trader, then we should also accept it as ours as well. I think this was a defining moment for me. Too many people take a money making machine such as a trend following system and attempt to tweak it to the market conditions. Instead, they should simply stick with it. I ran a trend following system for clients into the 2002 bear market. Many clients closed their accounts because their equity curves stopped moving higher. Along comes 2003 and it becomes downright frightening how much money it has cost them. People accept that their property valuations may stagnate and even decline in certain years. Why do they think differently when trading?




Thanks for the link and comments and _perspective_.

Regards
Snake


----------



## tech/a (7 July 2007)

wayneL said:


> Doh... leverage?



Compounding and re investment of profits.
Increasing trade size rather than number of trades.
Amplified again by Leverage.


----------



## Nick Radge (7 July 2007)

I'll attempt to clarify my reasoning on survivorship bias. First let's define what is is and what it can mean. Survivorship bias (or selection bias) relates to something that has been included or removed from a testing environment using hindsight. As an example, a value investor may deem that they would never have held HIH or SGW in their hypothetical portfolio's or tech stocks during the tech crash. The opposite would be including a stock such as Davnet, that, from memory, ran from 2c to over $5 or so. If these singular events create a large skew on the test results, then there is concern that these test results cannot be replicated into the future. In other words the fact that they were included is nothing more than luck.

A recent post on this forum highlighted the exact case where a poster had picked a single stock which had provided him/her with vast profits. His/her conclusion was that _*all*_ analysis was garbage and that selecting the next great stock was all that was needed. Maybe so, but basing that theory on a limited "lucky" outcome is representative of survivorship bias, even more so during this massive resources boom. 

The point being put forward in this post is that selecting a _*current*_ universe of symbols and testing them _*backward*_ would lead to survivorship bias, because had we actually started _*back*_ at that point of time the selection would have been different. The example here was based on the ASX-200 of today vs. the ASX-200 of some point in history. 

My argument is that if the system being used was robust, then the universe being used would not matter a great deal. After all, a trend is a trend is a trend. A robust trend following system is designed to pick up and ride trends until they reverse. Which symbol is actually being ridden is secondary. I stress this is not a discussion on the market phases in which certain phases create great results for trend following whilst other phases show poor results. My argument is, so what if OneTel or Davnet were included in a trend following system test in the 90's? Today contains trends that are just as spectacular but using different stocks. The key element is that the system _*can*_ pick up the trend when and where they come along. However, as stated earlier, so long as a single stock does not make up the vast majority of the P&L then there is no risk of survivorship bias. 

Let's use a real time example but to show the survivorship bias I will work backward. Below are the real time results of one of my trend following systems. 







Before we move on I will ask if you feel these results are acceptable? There are no severe outliers that are skewing the results. If so, read on. 

What is/are the survivorship bias?

Well, there are two occurring. The universe of stocks selected for this system must have a minimum Broker Consensus rating of BUY. As we know, Broker Consensus is a liquid measure, in other words Brokers change their minds so testing with one universe at one period of time will be quite different to trading that same universe at another period of time. One example here is ORG which was entered on Dec 4th. The issue is that Brokers now do not have a BUY ranking on the stock. In essence it should not be included in these results because if we were to run the test again now we would not select ORG to be involved.. 

However, there are plenty of others. Note the following 3 charts and that none of them appear in the current results even though the testing suggest they should. 

*IIN*






*RIO*





*MBL*





These 3 charts all show clear BUY signals that are all nicely profitable. At the time of entry for these 3 signals Broker Consensus did not meet the required criteria so these 3 missed being in the universe in actual trading. If were to now do the testing we'd now be selecting these stocks.

In summary we have a moving universe of stocks, both for the testing phase and for the real time trading phase BUT I did pose the question right at the start about being agreeable with the real time results and whether or not they were acceptable. If the answer was yes, then we've managed to achieve this even though there is clear survivorship bias occurring. In turn this means that the system is able to continue to operate and generate profits regardless of what actually contained within that universe.

If I now take this one step even further back I can show you the actual testing before these trades were taken.






We can see a reasonable out performance in real time verse testing. But the issue is really this, "how did you test the Broker Consensus criteria?". You can't. I didn't. I ran the test raw THEN added that overlay. It's a common sense assumption. We work with the worst possible raw data then make common sense assumptions in real time. The end result is more than likely going to be better than the test and adds another element of robustness.


----------



## wayneL (7 July 2007)

tech/a said:


> Compounding and re investment of profits.
> Increasing trade size rather than number of trades.
> Amplified again by Leverage.




Yes, but only point three explains the 1300%


----------



## tech/a (7 July 2007)

wayneL said:


> Yes, but only point three explains the 1300%




On its own no.
While it has a large impact some trades were double the initial parcel size traded.
Doubling AND Leverage are very powerful in the end result.

Note the time held and return achieved V length of trend ridden.
ZFX V ASX


----------



## wayneL (7 July 2007)

tech/a said:


> On its own no.
> While it has a large impact some trades were double the initial parcel size traded.
> Doubling AND Leverage are very powerful in the end result.
> 
> ...




Well no thats true. But while compounding and increasing trade size is jolly essential, they are a given. (IMHO) 

But leverage is the turbocharger here.


----------



## theasxgorilla (7 July 2007)

Nick Radge said:


> We can see a reasonable out performance in real time verse testing. But the issue is really this, "how did you test the Broker Consensus criteria?". You can't. I didn't. I ran the test raw THEN added that overlay. It's a common sense assumption. We work with the worst possible raw data then make common sense assumptions in real time. The end result is more than likely going to be better than the test and adds another element of robustness.




The same seems to be true for the discretionary entry on TechTrader which states the the share must be clearly breaking out or reversing trend (or words to that effect)...practically impossible to apply in testing.

Additionally there is the $10 price limit and the money flow filter.  Both of which might ultimately be affected by inflation, unless of course $10 has been some psychological level for decades...so unless you adjust such filters going back in time you will pick the RIOs and the BHPs back when they ticked those boxes and make yourself a killing.  But maybe that is okay?  Maybe we would have traded those if the system was running back then.  I'm not convinced and I'm not sure it's wise to over-rate hindsight.  

I do agree that such a system is robust enough to pick such trends going forward though.  There is perhaps then less value in the actual statistical results of backtesting shares and more value in going back and looking at the charts, as you have done with IIN, RIO and MAP, to see if your system is catching what you intend it to.

I would never have thought about the broker consensus rating as a filter or a liquidity measure.  Does it mean that if enough brokers rate it a buy then either them or their clients are making a market for it??  Clearly not everyone gets common sense assumptions


----------



## GreatPig (7 July 2007)

Nick,

Thanks for your explanation.



			
				Nick Radge said:
			
		

> His/her conclusion was that all analysis was garbage and that selecting the next great stock was all that was needed.



Not sure if you're referring to my recent post here, but I certainly didn't say or conclude that.

My conclusion was essentially that one needs to be wary of backtesting over conditions that rarely occur in the universe being tested.



> My argument is that *if the system being used was robust*, then the universe being used would not matter a great deal.



That is perhaps the significant point. With a _robust_ system it shouldn't matter, but on the road to developing a robust system you need to be sure that your backtesting is providing reasonably reaslistic results, otherwise you might end up thinking you have a profitable and robust system when in fact you haven't.

As an example, during testing of a system under development I was using the ASX300 as the universe and getting some pretty good results, even with Monte Carlo testing. However, on closer inspection of the results I noticed that the main winning trades were ones purchased at very low share prices, which would have mostly been before those stocks were in the ASX300. As an attempt to closer simulate the current ASX300 I then set a minimum share price of $2. Running the same backtests again showed a drop in return to almost nothing. That's the sort of thing I'm suggesting one needs to be wary of.

Cheers,
GP


----------



## wayneL (7 July 2007)

GreatPig said:


> Not sure if you're referring to my recent post here, but I certainly didn't say or conclude that.




I'm pretty sure Nick was referring to STC's thread. i.e. Just get out your crystal ball and buy the next PDN (or whatever share it was)


----------



## theasxgorilla (7 July 2007)

GreatPig said:


> As an attempt to closer simulate the current ASX300 I then set a minimum share price of $2. Running the same backtests again showed a drop in return to almost nothing. That's the sort of thing I'm suggesting one needs to be wary of.




Definately!

Ive found that during this bull market some of those shares that caused my portfolio (actual, not test) to outperform were the likes of MCR and OXR when they were trading below $1.  A lot of my other winners/losers have been around 1R either way and hence tended to a cancel one another out or just track the index.  Although this probably more so reflects my ability to trade as opposed to something concrete.

Having said all that, Im surprised that removing trades on shares below $2 caused the system to return "almost nothing".


----------



## weird (7 July 2007)

Applying a $2 filter on my current ASX300 list, I would be removing 75 candidates from testing =(

TT applies a less than $10 filter, I have even read Nick mentioning an idea once of possibly using a less than $5 filter to catch the early ones ...

GP, just then testing 2 mechanical systems I trade, with a greater than $2 filter , the performance did drop incredibly ... both systems dropped to around 13% PA for a 6 year period.

However I don't use a greater than $2 filter though in system design, and trading. I have in the past used a larger than $0.40 filter in backtesting, but that number was choosen for no real good reason.

I do remove large outliners from backtesting, however the occasional 300% + return from some stocks is not that unusual from some long term trend trading systems.


----------



## Sir Burr (8 July 2007)

GreatPig said:


> As an attempt to closer simulate the current ASX300 I then set a minimum share price of $2.




Try whole database with a variable turnover (highest avg turnover = generally "top stocks"). Makes more sence than using ASX300 :headshake

Edit: and purchase data containing delisted stocks.


----------



## GreatPig (8 July 2007)

theasxgorilla said:
			
		

> Im surprised that removing trades on shares below $2 caused the system to return "almost nothing"



When I say "system", I mean an idea for a system that I'd recently coded and had started doing some testing on. It was by no means a well-tested and robust system - although of course I had hopes that it might get to be .

And there was nothing special about the $2 figure. It was just a number I quickly picked out of the air to see what difference it made to the results after noticing many of the large-gain trades having much lower purchase prices - lower than most stocks currently in the ASX300 (and yes, I know there are _some_ stocks in the ASX300 with lower prices than that).

Cheers,
GP


----------



## Nick Radge (8 July 2007)

GP,
That comment was certainly not aimed at you. I think Wayne has pointed out the correct individual. 

Some other things to consider when testing a system and its viability in the real world.

Firstly there is no real point building a simplistic system that, when operational, requires a discretionary over ride to distinguish signals. I have numerous people come to me because they've done certain courses, which all shall remain nameless,  that teaches trend following in various formats. The problem is that when they run the system rules they get 10 signals each and every day. From that juncture one must then pick and choose which to take and as we all know, we'll always pick the wrong ones. 

The above issue comes from "simplistic" systems. A simplistic system will have several inherent errors when it comes to real world trading. The one I just mentioned above is the most common, especially in this kind of environment. The other is more serious when it comes it comes to the psychology of the system. A simplistic system will tend to generate many false signals during a bearish market phase which in turn will lead to a lot of frustration but also a dwindling account balance. I see this as a real risk for those embarking on this journey in the current environment. Any old simple trend following system will do okay in this environment. I know of a very well known broker touting such a system to the public that uses a 22/35 week crossover method. Yes it will catch killer long term trends but it will get mauled in any longer term bearish environment and will also be prone to false signals in a sideways period. Another simplistic system would be some kind of price breakout, like a channel breakout of a n-day high. 

HERE is an interesting paper on trend following in stocks. I think this is a simple way to overcome the n-day breakout suggested above.

Another "common sense" assumption is finding those golden trends, albeit with added volatility and risk. Essentially a business will grow and with it so will the share price. One therefore needs to find the "emerging" businesses, that is those that may end up being the next TOL or CSL. Almost every one of these will be a relatively new listing and therefore be trading below $10.00. I recommend to my subscribers looking to get a bit more "bang for their buck" to only take signals on those stocks trading below $10 without any lower limitations. If you remove the signals above under $10.00 you get a reasonable increase in the risk adjusted return. If however you only take signals below $5 you get a _*substantial*_ increase in risk adjusted return.

This process will also reduce the universe and, with simplistic systems, reduce the potential trade frequency and therefore the discretionary input levels.

What is an easy way to find these? Focus on stocks that _*do not*_ pay dividends, which is, funnily enough, another portfolio I run.


----------



## theasxgorilla (8 July 2007)

Sir Burr said:


> Edit: and purchase data containing delisted stocks.




This idea has merit.  Can anyone suggest who provides such data for the ASX?

I just read the following quote from the paper that you linked to Nick:

"*Survivorship bias*:
The database used for this project included historical data for all stocks that were delisted at some point
between 1983 and 2004. Slightly more than half of the database is comprised of delisted stocks."

Whilst I gather from your comments that survivorship bias has a tendency to be overrated as a factor and a good, robust system will catch trends and backtest okay regardless, for my peace of mind it would be useful to have such data.  Particularly if our own ASX historical data could claim to have around half of its quantity in delisted shares! (as their study did).

ASX.G


----------



## Nick Radge (8 July 2007)

I think it is available.

But here is my contention; stocks that delist don't just evaporate over night. They_ usually_ trend continually down for an extended period of time. Granted, a super long term weekly/monthly system may not capture that trend, but most daily and weekly systems would and therefore even though its delisted it would still have provided the opportunity prior. ONE is a classic example. It was a great trending and highly profitable stock _at the time_. Yes it no longer exists and could be considered as a candidate for survivorship bias. However, it should not have been detrimental to a portfolio's performance at the time.

Lets use an assumption in todays terms. Evans and Tate would be an example of a stock that is slowly going to oblivion; no different from HIH, ONE or SGW. Its clearly in no position to trigger a buy for such a system outlined in that report. 

I think perhaps the difference in thinking is that whilst I may trade a trend in ABC Company, I do not look at that being a candidate ever again. There will always be another to take its specific place. Whereas survivorship bias relates specifically ABC company being available again.


----------



## Sir Burr (8 July 2007)

theasxgorilla said:


> Can anyone suggest who provides such data for the ASX?



Justdata - Bodhi History.


----------



## theasxgorilla (8 July 2007)

Nick Radge said:


> They_ usually_ trend continually down for an extended period of time.




Then there is the flipside...the likes of say GCN.  I think that was the code...I dont have the chart as my provider pulls feeds from delisted companies ...small gold miner, formed a consolidation pattern like OXR or MCR, then broke out...went .35 to .50 in some weeks and was then taken over (for all intents and purposes, delisted).  In all likelihood a TechTrader like system would have traded it (assuming it met the BT list, moneyflow and breakout criteria, which I cant be sure of), but now in backtesting *with my data at least*, that trade would not be there to be taken.

It seems to a degree to be swings and roundabouts as they say.  My curiousity just wonders to what degree.  It wouldn't stop me trading a good system, but if you wanted to produce a white paper on the ASX like the guys at the link you showed it would be handy to have ALL the data.


----------



## theasxgorilla (8 July 2007)

GGN! Gallery Gold...from the chart it looks like TechTrader would certainly have traded it.


----------



## GreatPig (8 July 2007)

theasxgorilla said:
			
		

> and was then taken over (for all intents and purposes, delisted)



If it was taken over though, then presumably it wouldn't have been detrimental to your portfolio at the time since I assume you would have got some equivalent value in the parent stock.

GP


----------



## theasxgorilla (8 July 2007)

GreatPig said:


> If it was taken over though, then presumably it wouldn't have been detrimental to your portfolio at the time since I assume you would have got some equivalent value in the parent stock.




 So the next logical question is, would NOT testing with the full history of data including delisted shares understate backtesting results?  Since as Nick pointed out the system would not trade the downtrending dogs, but in times of high MA activity like now and perhaps the late 90s our testing may not include of GGN as they broke out prior to being taken over.


----------



## Nick Radge (8 July 2007)

No disrespect to tech/a, but I think TT is a simplistic system as discussed above. Secondly, the M&A activity would therefore be seen as a "common sense" assumption, but also not one to be relied upon for "making or breaking" the system. I was around in 1987 - in fact trading a "simplistic" trend following system. The same influences were around then, as they were in 1998 through 2000. Greed, far lack of a better work, is simply showing itself in another format.

On that last line, how can we ever forget...

_"Teldar Paper, Mr. Cromwell, Teldar Paper has 33 different vice presidents each earning over 200 thousand dollars a year. Now, I have spent the last two months analyzing what all these guys do, and I still can't figure it out. One thing I do know is that our paper company lost 110 million dollars last year, and I'll bet that half of that was spent in all the paperwork going back and forth between all these vice presidents. The new law of evolution in corporate America seems to be survival of the unfittest. Well, in my book you either do it right or you get eliminated. In the last seven deals that I've been involved with, there were 2.5 million stockholders who have made a pretax profit of 12 billion dollars. Thank you. I am not a destroyer of companies. I am a liberator of them! *The point is, ladies and gentleman, that greed, for lack of a better word, is good. Greed is right, greed works. Greed clarifies, cuts through, and captures the essence of the evolutionary spirit. Greed, in all of its forms; greed for life, for money, for love, knowledge has marked the upward surge of mankind. And greed, you mark my words, will not only save Teldar Paper, but that other malfunctioning corporation called the USA.* Thank you very much."  _ Gordon Gekko, Wall Street.


----------



## weird (8 July 2007)

Hi GP, just to correct my original post.

The testing results I posted where based on migrated (incomplete ... which I will work on ... and on the growing to do list) code using Amibroker, as opposed to Metastock/Tradesim which the systems were originally developed on and how they are actually traded. 

The returns dropped to 19%PA using a $2 filter, not %13 PA.

As Stevo has also mentioned in the past, testing on different constituent lists may also help in testing systems.  

And as Nick mentioned, knowing why the system works.


----------



## theasxgorilla (9 July 2007)

Nick Radge said:


> HERE is an interesting paper on trend following in stocks. I think this is a simple way to overcome the n-day breakout suggested above.




I'm not sure I follow.

Do you mean that the system they use overcomes the false-breakouts of an n-day breakout system during a prolonged bear market because it only buys shares trading to new all time highs in a clear uptrend?

Wonderful paper BTW.  I notice the worst drawdown occured through '02 into early '03.  Now look where we are...probably a message in that somewhere.  Interesting that they adjust all their data for dividends too.  How on earth would we do that for the ASX??  Does failure to account for it invalidate testing on dividend paying shares??  I also notice that did back adjust their "minimum average daily dollar volume" filter at a decay-rate consistent with inflation.  Its a very serious paper...not something you just knock up over a handful of weekends.


----------



## theasxgorilla (9 July 2007)

I also noticed that they kept the minimum share price filter level at $15 for the life of the testing.  An interesting phenomenon that irrespective of the purchasing power of $1 that level remains constant.  I remember thinking when I read _Reminiscenses of a Stock Operator_ that the share prices he quotes back then, particularly of boom stocks, sounded consistent with dotcom prices ie. up $300 a share, even though the relative purchasing power of $300 back then much have been immense compared with today.  Sounds reasonable that the trick is to estimate those levels for the market you are trading and don't worry about indexing them to inflation.  As they say themselves, $15 wasn't scientifically selected.


----------



## Nick Radge (9 July 2007)

> Interesting that they adjust all their data for dividends too.




Yet another "common sense" assumption.

Bear markets contain sharp upward rallies that are short lived. A system that say buys the breakout of a 50-day high will be sucked into these types of bear market rallies. There is no doubt that a 50-day breakout has performed exceptionally well over the last 4-years, but being a simplistic system it will struggle through a bear market. A system that buys new all-time highs however will not be sucked into bear market rallies as its rare that many stocks will be making ATH during a bear market.

Remember that the paper was based on US stocks. A $15 minimum in the US is probably nearer a $0.20 minimum here.

The basis of the paper, that is reading between the lines, is that trend following skews the expectancy far in one's favour. For those that have read my book Adapative Analysis and recall the expecatncy curve, these systems operate in upper right hand quandrant.


----------



## Nick Radge (13 July 2007)

Here is an example that makes a difference between a "simplistic" trend following system and a more complex one. Below is the actual system signals for PMM which I recommended to subscribers. In the end we banked a 102% profit. I should stress that they don't comply as well as this one has all of the time, but from an _*equity curve volatility perspective*_, its a tweak that I think makes a major difference. A simplistic system can give back substantial open profit at all times. This one measures volatility and as it expands it tightens the stop to a point of actually exiting as prices are rising. MMX was another recent example as well.


----------



## nizar (15 July 2007)

Wow.

I havent come back here since i started this thread and got only 2 replies, one from Nick and the other from Snake.

Id like to thank everybody for their responses, especially you Nick, alot of this stuff is GOLD.

Regarding data including delisted stocks and ASX indices being upto date with every quarterly change, i would expect professional data providers such as Just Data or Premium Data to provide this (They claim to on their websites). Serious traders need serious data. 

Nick, you make an interesting point about the differences in return between a shorter term system eg. 50-day breakout compared to one which enters on alltime highs during a bearmarket. But during a bullmarket, often with a system with alltime high entry the stop is more loose and lagging the price hence it would be a higher drawdown system?

Which one works better over the longer term?

I pulled up the link for Nick's article but its 20 pages so i will have a good read of it 2mrw morning. Also got the tradesim manual to read.

Below is something for me to print out and paste on the wall:


> Some other things to consider when testing a system and its viability in the real world.
> 
> Firstly there is no real point building a simplistic system that, when operational, requires a discretionary over ride to distinguish signals. I have numerous people come to me because they've done certain courses, which all shall remain nameless, that teaches trend following in various formats. The problem is that when they run the system rules they get 10 signals each and every day. From that juncture one must then pick and choose which to take and as we all know, we'll always pick the wrong ones.
> 
> ...


----------



## nizar (15 July 2007)

tech/a said:


> The question of survivorship.
> One I have pondered for hrs.
> 
> I have not culled my data for 4 yrs.
> ...





Tech what level of drawdown was acceptable (after system testing) to achieve the 22-28% return?

I recall initial drawdown was on the system was about 12%.

What was the maximum MaxDD for techtrader during system trading?

Tech i recall you saying you tested the system between 1993-2001. And you started forward testing the system in realtime in 2002? So in your first year you mustve had quite a shock and sitting on pretty significant drawdowns yeh?

Just a bit more to add (generally not in response to tech)

Survivorship bias and indices makeup are issues raised in this thread but like i said thats to do with your data provider. They should take care of this.

I think a (more) major problem with systems design and backtesting is curve-fitting. To avoid this we can forward test the system. Ie. Design/Test a system between 1992-1999 and then run it over the years 1999-2006, for example.


----------



## nizar (15 July 2007)

Nick Radge said:


> HERE is an interesting paper on trend following in stocks. I think this is a simple way to overcome the n-day breakout suggested above.




Very good read this one.
Top paper.

Style of the paper almost like a journal article. Speaking of journals, do they have a journal for mechanical trading like they have in science and medicine? That'll be nice lol.

I found the results of the study was very interesting, for a longterm trend following system to hit 49% winners and R/R of 2.5. I would expect, especially for a system when average trade time is almost a year, for win % to be 30-40% and R/R to be much higher (4-5++). I wander what the stats would be for exactly the same system to be used for the ASX, could be an interesting comparison.

Just one thing i didnt understand on pg12 it says some of the principles the system strictly adhered to with one of them being:


> Winning trades are only reduced to alleviate risk concentrations.




What does this mean?

Thanks.


----------



## nizar (15 July 2007)

*Winning trades are only reduced to alleviate risk concentrations. *

Does this mean that if a winner is such a champion that it grows to over 20% of the portfolio and your portfolio limit is 20% max per stock, then you reduce the position??

And also the author of the paper details why they did not consider a short system and one of the reasons was due to forced buy-ins (which cannot be backtested). 

Is a forced buy-in the short equivalent to a margin call?


----------



## theasxgorilla (15 July 2007)

nizar said:


> *Winning trades are only reduced to alleviate risk concentrations. *
> 
> Does this mean that if a winner is such a champion that it grows to over 20% of the portfolio and your portfolio limit is 20% max per stock, then you reduce the position??
> 
> ...




Re: concentrations, that is what I understood it to mean.

The answer to what is a "forced buy-in", I don't know.


----------



## GreatPig (15 July 2007)

I believe a forced buy-in is when your broker buys shares to cover your short position for you, possibly due to a shortage of shares so that he no longer has enough to lend you.

GP


----------



## tech/a (21 July 2007)

nizar said:


> Tech what level of drawdown was acceptable (after system testing) to achieve the 22-28% return?
> 
> I recall initial drawdown was on the system was about 12%.
> 
> ...





Nizar.
Sorry

Just back from Island sitting/exploring.


----------



## theasxgorilla (21 July 2007)

nizar said:


> I think a (more) major problem with systems design and backtesting is curve-fitting. To avoid this we can forward test the system. Ie. Design/Test a system between 1992-1999 and then run it over the years 1999-2006, for example.




I'm only reciting stuff from a book here...no experience (testing or trading) to draw from...but in the book Way of the Turtle, Curtis Faith was of the opinion that whilst curve-fitting is bad, optimisation was not.  From memory he defined curve-fitting (or _overfitting_) as being lots of indicators and their  variables tuned to produce exceptional results in historical testing, usually covering a shorter than adequate period of time.

Optimisation on the other hand he saw as tuning things like MA or breakout lengths.  His theory was that by chosing the most optimal value, if/when markets change characteristics in the future, your chances of having settings in your system that are close to the new optimal values are greater than if you'd purposely (or ignorantly) chosen a non-optimal value.

Eg.  Historical testing reveals that the optimal breakout length for entries in your market is 100.  20 days either side of that are entries that result in 5% less CAR, 80 and 120.  In the future the markets change and the new optimal value is 120.  You're CAR is only 5% less than optimal.  If you'd chosen 80 on the other hand, for no apparent reason other than it seemed 'robust', you system might earn 10% less.  The explanation is oversimplified, but it made sense to me.


----------



## nizar (21 July 2007)

theasxgorilla said:


> I'm only reciting stuff from a book here...no experience (testing or trading) to draw from...but in the book Way of the Turtle, Curtis Faith was of the opinion that whilst curve-fitting is bad, optimisation was not.  From memory he defined curve-fitting (or _overfitting_) as being lots of indicators and their  variables tuned to produce exceptional results in historical testing, usually covering a shorter than adequate period of time.
> 
> Optimisation on the other hand he saw as tuning things like MA or breakout lengths.  His theory was that by chosing the most optimal value, if/when markets change characteristics in the future, your chances of having settings in your system that are close to the new optimal values are greater than if you'd purposely (or ignorantly) chosen a non-optimal value.
> 
> Eg.  Historical testing reveals that the optimal breakout length for entries in your market is 100.  20 days either side of that are entries that result in 5% less CAR, 80 and 120.  In the future the markets change and the new optimal value is 120.  You're CAR is only 5% less than optimal.  If you'd chosen 80 on the other hand, for no apparent reason other than it seemed 'robust', you system might earn 10% less.  The explanation is oversimplified, but it made sense to me.





Hi ASX.G,

Thanks for your post.
Yes i am halfway through reading way of the turtle, upto about page100 and i think i read that section last night, so still very fresh in my mind.

Top read that book.


----------



## nizar (21 July 2007)

tech/a said:


> Nizar.
> Sorry
> 
> Just back from Island sitting/exploring.




Hi Tech.
Hope it was fun.

Can you please explain curve-fitting to me?

Thanks.


----------



## Sir Burr (21 July 2007)

nizar said:


> Can you please explain curve-fitting to me?



Nizar,

If you have a look through this page and to the 3D picture:

http://www.amibroker.com/guide/h_optimization.html

It's a picture of profit against a couple of variables in a system. Setting variables to that very highest peak would be curve fitting, picking the best result possible from historical data.

Any change "in the market" and choosing those variables may cause a bit of a drop-off in profits!

Look for a high flatish area to set your variables, apparently that produces a robust system. :dance:

SB


----------

