Systems testing

Nick Radge · 8 July 2007

No disrespect to tech/a, but I think TT is a simplistic system as discussed above. Secondly, the M&A activity would therefore be seen as a "common sense" assumption, but also not one to be relied upon for "making or breaking" the system. I was around in 1987 - in fact trading a "simplistic" trend following system. The same influences were around then, as they were in 1998 through 2000. Greed, far lack of a better work, is simply showing itself in another format.

On that last line, how can we ever forget...

"Teldar Paper, Mr. Cromwell, Teldar Paper has 33 different vice presidents each earning over 200 thousand dollars a year. Now, I have spent the last two months analyzing what all these guys do, and I still can't figure it out. One thing I do know is that our paper company lost 110 million dollars last year, and I'll bet that half of that was spent in all the paperwork going back and forth between all these vice presidents. The new law of evolution in corporate America seems to be survival of the unfittest. Well, in my book you either do it right or you get eliminated. In the last seven deals that I've been involved with, there were 2.5 million stockholders who have made a pretax profit of 12 billion dollars. Thank you. I am not a destroyer of companies. I am a liberator of them! The point is, ladies and gentleman, that greed, for lack of a better word, is good. Greed is right, greed works. Greed clarifies, cuts through, and captures the essence of the evolutionary spirit. Greed, in all of its forms; greed for life, for money, for love, knowledge has marked the upward surge of mankind. And greed, you mark my words, will not only save Teldar Paper, but that other malfunctioning corporation called the USA. Thank you very much." Gordon Gekko, Wall Street.

weird · 8 July 2007

Hi GP, just to correct my original post.

The testing results I posted where based on migrated (incomplete ... which I will work on ... and on the growing to do list) code using Amibroker, as opposed to Metastock/Tradesim which the systems were originally developed on and how they are actually traded.

The returns dropped to 19%PA using a $2 filter, not %13 PA.

As Stevo has also mentioned in the past, testing on different constituent lists may also help in testing systems.

And as Nick mentioned, knowing why the system works.

theasxgorilla · 9 July 2007

Nick Radge said:
HERE is an interesting paper on trend following in stocks. I think this is a simple way to overcome the n-day breakout suggested above.

I'm not sure I follow.

Do you mean that the system they use overcomes the false-breakouts of an n-day breakout system during a prolonged bear market because it only buys shares trading to new all time highs in a clear uptrend?

Wonderful paper BTW. I notice the worst drawdown occured through '02 into early '03. Now look where we are...probably a message in that somewhere. Interesting that they adjust all their data for dividends too. How on earth would we do that for the ASX?? Does failure to account for it invalidate testing on dividend paying shares?? I also notice that did back adjust their "minimum average daily dollar volume" filter at a decay-rate consistent with inflation. Its a very serious paper...not something you just knock up over a handful of weekends.

theasxgorilla · 9 July 2007

I also noticed that they kept the minimum share price filter level at $15 for the life of the testing. An interesting phenomenon that irrespective of the purchasing power of $1 that level remains constant. I remember thinking when I read Reminiscenses of a Stock Operator that the share prices he quotes back then, particularly of boom stocks, sounded consistent with dotcom prices ie. up $300 a share, even though the relative purchasing power of $300 back then much have been immense compared with today. Sounds reasonable that the trick is to estimate those levels for the market you are trading and don't worry about indexing them to inflation. As they say themselves, $15 wasn't scientifically selected.

Nick Radge · 9 July 2007

Interesting that they adjust all their data for dividends too.

Yet another "common sense" assumption.

Bear markets contain sharp upward rallies that are short lived. A system that say buys the breakout of a 50-day high will be sucked into these types of bear market rallies. There is no doubt that a 50-day breakout has performed exceptionally well over the last 4-years, but being a simplistic system it will struggle through a bear market. A system that buys new all-time highs however will not be sucked into bear market rallies as its rare that many stocks will be making ATH during a bear market.

Remember that the paper was based on US stocks. A $15 minimum in the US is probably nearer a $0.20 minimum here.

The basis of the paper, that is reading between the lines, is that trend following skews the expectancy far in one's favour. For those that have read my book Adapative Analysis and recall the expecatncy curve, these systems operate in upper right hand quandrant.

Nick Radge · 13 July 2007

Here is an example that makes a difference between a "simplistic" trend following system and a more complex one. Below is the actual system signals for PMM which I recommended to subscribers. In the end we banked a 102% profit. I should stress that they don't comply as well as this one has all of the time, but from an equity curve volatility perspective, its a tweak that I think makes a major difference. A simplistic system can give back substantial open profit at all times. This one measures volatility and as it expands it tightens the stop to a point of actually exiting as prices are rising. MMX was another recent example as well.

nizar · 15 July 2007

Wow.

I havent come back here since i started this thread and got only 2 replies, one from Nick and the other from Snake.

Id like to thank everybody for their responses, especially you Nick, alot of this stuff is GOLD.

Regarding data including delisted stocks and ASX indices being upto date with every quarterly change, i would expect professional data providers such as Just Data or Premium Data to provide this (They claim to on their websites). Serious traders need serious data.

Nick, you make an interesting point about the differences in return between a shorter term system eg. 50-day breakout compared to one which enters on alltime highs during a bearmarket. But during a bullmarket, often with a system with alltime high entry the stop is more loose and lagging the price hence it would be a higher drawdown system?

Which one works better over the longer term?

I pulled up the link for Nick's article but its 20 pages so i will have a good read of it 2mrw morning. Also got the tradesim manual to read.

Below is something for me to print out and paste on the wall:

Some other things to consider when testing a system and its viability in the real world.

Firstly there is no real point building a simplistic system that, when operational, requires a discretionary over ride to distinguish signals. I have numerous people come to me because they've done certain courses, which all shall remain nameless, that teaches trend following in various formats. The problem is that when they run the system rules they get 10 signals each and every day. From that juncture one must then pick and choose which to take and as we all know, we'll always pick the wrong ones.

The above issue comes from "simplistic" systems. A simplistic system will have several inherent errors when it comes to real world trading. The one I just mentioned above is the most common, especially in this kind of environment. The other is more serious when it comes it comes to the psychology of the system. A simplistic system will tend to generate many false signals during a bearish market phase which in turn will lead to a lot of frustration but also a dwindling account balance. I see this as a real risk for those embarking on this journey in the current environment. Any old simple trend following system will do okay in this environment. I know of a very well known broker touting such a system to the public that uses a 22/35 week crossover method. Yes it will catch killer long term trends but it will get mauled in any longer term bearish environment and will also be prone to false signals in a sideways period. Another simplistic system would be some kind of price breakout, like a channel breakout of a n-day high.

HERE is an interesting paper on trend following in stocks. I think this is a simple way to overcome the n-day breakout suggested above.

Another "common sense" assumption is finding those golden trends, albeit with added volatility and risk. Essentially a business will grow and with it so will the share price. One therefore needs to find the "emerging" businesses, that is those that may end up being the next TOL or CSL. Almost every one of these will be a relatively new listing and therefore be trading below $10.00. I recommend to my subscribers looking to get a bit more "bang for their buck" to only take signals on those stocks trading below $10 without any lower limitations. If you remove the signals above under $10.00 you get a reasonable increase in the risk adjusted return. If however you only take signals below $5 you get a substantial increase in risk adjusted return.

This process will also reduce the universe and, with simplistic systems, reduce the potential trade frequency and therefore the discretionary input levels.

What is an easy way to find these? Focus on stocks that do not pay dividends, which is, funnily enough, another portfolio I run.

nizar · 15 July 2007

tech/a said:
The question of survivorship.
One I have pondered for hrs.

I have not culled my data for 4 yrs.
So have atleast all the delists for the last 4 yrs in 7 bourses.
That helps.
I trade the ASX 300 (Well a margin list predominantly made up of),which to some degree minimises the problem.

While always a risk If I'm only trading 10% of my account on anyone trade then There is that minimisation.
Anything I systems trade is pretty heavily tested so I have a good idea of overall expected performance based on 1000s of portfolios.
By this I mean Ill know if one or 2 of these event appear and turn an otherwise profitable method into a loser. From testing on mine--it doesnt.

There is much to developing and designing a system.

When you REALLY REALLY understand these words---and you'll get that from experience rather than comprehension---then you'll have the Ahh of Ahh moments.

The developement and application through developement becomes a breeze.

THEN look beyond that which you have developed to how a system which is average can be made well above average!

T/T returns about 35% compounded annually---so how is it that it has returned 1300% on initial funds in 4.5 yrs?

Tech what level of drawdown was acceptable (after system testing) to achieve the 22-28% return?

I recall initial drawdown was on the system was about 12%.

What was the maximum MaxDD for techtrader during system trading?

Tech i recall you saying you tested the system between 1993-2001. And you started forward testing the system in realtime in 2002? So in your first year you mustve had quite a shock and sitting on pretty significant drawdowns yeh?

Just a bit more to add (generally not in response to tech)

Survivorship bias and indices makeup are issues raised in this thread but like i said thats to do with your data provider. They should take care of this.

I think a (more) major problem with systems design and backtesting is curve-fitting. To avoid this we can forward test the system. Ie. Design/Test a system between 1992-1999 and then run it over the years 1999-2006, for example.

nizar · 15 July 2007

Nick Radge said:
HERE is an interesting paper on trend following in stocks. I think this is a simple way to overcome the n-day breakout suggested above.

Very good read this one.
Top paper.

Style of the paper almost like a journal article. Speaking of journals, do they have a journal for mechanical trading like they have in science and medicine? That'll be nice lol.

I found the results of the study was very interesting, for a longterm trend following system to hit 49% winners and R/R of 2.5. I would expect, especially for a system when average trade time is almost a year, for win % to be 30-40% and R/R to be much higher (4-5++). I wander what the stats would be for exactly the same system to be used for the ASX, could be an interesting comparison.

Just one thing i didnt understand on pg12 it says some of the principles the system strictly adhered to with one of them being:

Winning trades are only reduced to alleviate risk concentrations.

What does this mean?

Thanks.

nizar · 15 July 2007

Winning trades are only reduced to alleviate risk concentrations.

Does this mean that if a winner is such a champion that it grows to over 20% of the portfolio and your portfolio limit is 20% max per stock, then you reduce the position??

And also the author of the paper details why they did not consider a short system and one of the reasons was due to forced buy-ins (which cannot be backtested).

Is a forced buy-in the short equivalent to a margin call?

theasxgorilla · 15 July 2007

nizar said:
Winning trades are only reduced to alleviate risk concentrations.

Does this mean that if a winner is such a champion that it grows to over 20% of the portfolio and your portfolio limit is 20% max per stock, then you reduce the position??

And also the author of the paper details why they did not consider a short system and one of the reasons was due to forced buy-ins (which cannot be backtested).

Is a forced buy-in the short equivalent to a margin call?

Re: concentrations, that is what I understood it to mean.

The answer to what is a "forced buy-in", I don't know.

GreatPig · 15 July 2007

I believe a forced buy-in is when your broker buys shares to cover your short position for you, possibly due to a shortage of shares so that he no longer has enough to lend you.

GP

tech/a · 21 July 2007

nizar said:
Tech what level of drawdown was acceptable (after system testing) to achieve the 22-28% return?

I recall initial drawdown was on the system was about 12%.

What was the maximum MaxDD for techtrader during system trading?

In Peak to valley from memory around 22%
Initial drawdown around 8%

Tech i recall you saying you tested the system between 1993-2001. And you started forward testing the system in realtime in 2002? So in your first year you mustve had quite a shock and sitting on pretty significant drawdowns yeh?

No There was always open equity to ofset the initial and inevitable string of first loses as the portfolio takes shape I can post some curve figure piks if you want.

Just a bit more to add (generally not in response to tech)

Survivorship bias and indices makeup are issues raised in this thread but like i said thats to do with your data provider. They should take care of this.

I think a (more) major problem with systems design and backtesting is curve-fitting. To avoid this we can forward test the system. Ie. Design/Test a system between 1992-1999 and then run it over the years 1999-2006, for example.

Youve not explained curve fitting correctly

Nizar.
Sorry

Just back from Island sitting/exploring.

theasxgorilla · 21 July 2007

nizar said:
I think a (more) major problem with systems design and backtesting is curve-fitting. To avoid this we can forward test the system. Ie. Design/Test a system between 1992-1999 and then run it over the years 1999-2006, for example.

I'm only reciting stuff from a book here...no experience (testing or trading) to draw from...but in the book Way of the Turtle, Curtis Faith was of the opinion that whilst curve-fitting is bad, optimisation was not. From memory he defined curve-fitting (or overfitting) as being lots of indicators and their variables tuned to produce exceptional results in historical testing, usually covering a shorter than adequate period of time.

Optimisation on the other hand he saw as tuning things like MA or breakout lengths. His theory was that by chosing the most optimal value, if/when markets change characteristics in the future, your chances of having settings in your system that are close to the new optimal values are greater than if you'd purposely (or ignorantly) chosen a non-optimal value.

Eg. Historical testing reveals that the optimal breakout length for entries in your market is 100. 20 days either side of that are entries that result in 5% less CAR, 80 and 120. In the future the markets change and the new optimal value is 120. You're CAR is only 5% less than optimal. If you'd chosen 80 on the other hand, for no apparent reason other than it seemed 'robust', you system might earn 10% less. The explanation is oversimplified, but it made sense to me.

nizar · 21 July 2007

theasxgorilla said:
I'm only reciting stuff from a book here...no experience (testing or trading) to draw from...but in the book Way of the Turtle, Curtis Faith was of the opinion that whilst curve-fitting is bad, optimisation was not. From memory he defined curve-fitting (or overfitting) as being lots of indicators and their variables tuned to produce exceptional results in historical testing, usually covering a shorter than adequate period of time.

Optimisation on the other hand he saw as tuning things like MA or breakout lengths. His theory was that by chosing the most optimal value, if/when markets change characteristics in the future, your chances of having settings in your system that are close to the new optimal values are greater than if you'd purposely (or ignorantly) chosen a non-optimal value.

Eg. Historical testing reveals that the optimal breakout length for entries in your market is 100. 20 days either side of that are entries that result in 5% less CAR, 80 and 120. In the future the markets change and the new optimal value is 120. You're CAR is only 5% less than optimal. If you'd chosen 80 on the other hand, for no apparent reason other than it seemed 'robust', you system might earn 10% less. The explanation is oversimplified, but it made sense to me.

Hi ASX.G,

Thanks for your post.
Yes i am halfway through reading way of the turtle, upto about page100 and i think i read that section last night, so still very fresh in my mind.

Top read that book.

nizar · 21 July 2007

tech/a said:
Nizar.
Sorry

Just back from Island sitting/exploring.

Hi Tech.
Hope it was fun.

Can you please explain curve-fitting to me?

Thanks.

Sir Burr · 21 July 2007

nizar said:
Can you please explain curve-fitting to me?

Nizar,

If you have a look through this page and to the 3D picture:

http://www.amibroker.com/guide/h_optimization.html

It's a picture of profit against a couple of variables in a system. Setting variables to that very highest peak would be curve fitting, picking the best result possible from historical data.

Any change "in the market" and choosing those variables may cause a bit of a drop-off in profits!

Look for a high flatish area to set your variables, apparently that produces a robust system. :dance:

SB

Systems testing

Nick Radge

weird

theasxgorilla

Problem solved... next bubble.

theasxgorilla

Problem solved... next bubble.

Nick Radge

Nick Radge

nizar

nizar

nizar

nizar

theasxgorilla

Problem solved... next bubble.

GreatPig

Pigs In Space

tech/a

No Ordinary Duck

theasxgorilla

Problem solved... next bubble.

nizar

nizar

Sir Burr

Similar threads

Systems testing

Problem solved... next bubble.

Problem solved... next bubble.

Problem solved... next bubble.

Pigs In Space

No Ordinary Duck

Problem solved... next bubble.

Similar threads

Privacy & Transparency

Privacy & Transparency