System Robustness

[theasxgorilla]

For those who have Amibroker and are interested some code for computing the e-ratio live at:

http://theasxgorilla.blogspot.com/2007/09/e-ratio-40w-moving-average-system-code.html

If anyone reviews this and finds errors please let me know as I've developed a winning system based on it and I'm in the final stages of re-mortgaging and going 98% LVR on my house to raise funds for trading...joking, but seriously, if you find errors please post a comment and I'll revise the code.

 

[nizar]

If anyone reviews this and finds errors please let me know as I've developed a winning system based on it and I'm in the final stages of re-mortgaging and going 98% LVR on my house to raise funds for trading...joking,

LOL you almost gave me a heart attack!!

 

[nizar]

Greetings --

Just a few comments to points made in the last few days.

1. I think the folks who suggest that we traders need to work on the psychology of trading so that we can accept the behavior of our systems have it backwards. By designing our own objective function, we encapsulate our preferences and our comfort zone into the trading systems right from the start.

If we prefer short holding periods, we should ask for short holding periods -- that is, we should put a term in our objective function that rewards short holding periods -- then trading systems that score well will already be biased toward short holding periods and we will not have to adapt ourselves to systems that have long holding periods. Substitute any characteristic that is important to you for holding period -- RAR, equity curve slope, equity curve smoothness, Sharpe ratio, commissions paid, and so forth.

2. I have no complaint about trend-following systems. If they work, use them. My caution is that the 1982 to 2007 period has been an exceptionally strong bull market in equities. Any trading system that takes long-only positions should have some way of identifying that the market conditions are not right for it, and it should exit open trades and inhibit new trades.

Some skeptical friends of mine think that we will need to wait until after the next ice age to see a bull market like this again.

Whether that turns out to be true or not, prices of equities have risen far above their mean. When prices revert to the mean, they always pass through the mean and are extended to the other extreme by about the same amount. If that happens to equity prices, the excursion could take broad market averages down by 80% or more from here. If there is a market drop, and if trend-following systems work to profit from it, those systems will not be long-only.

3. Stops hurt systems. That does not mean do not use stops. It is just an observation that there are many ways to exit from a trade, and the worst of those ways is a maximum-loss stop.

Exits can be triggered:
1. By the action of an indicator or recognition of a pattern, similar to what caused the entry.
a. The parameters can be the same as those that caused the entry, but in the other direction.
b. The parameters can be different for exit than for entry.
c. Some other indicator can be used.
2. By the price reaching a profit target.
3. By the time in the trade reaching a maximum holding period.
4. By the price falling back to the level of a trailing stop.
5. By the price falling back to the level of a maximum-loss stop.

In my experience and research, exits caused by 1, 2, or 3 are preferable. Exits caused by 5 are worst.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

Hi Howard.

Firstly, thanks for contributing another top post, i like especially the discussion on exits.
The points 2 and 3 on exits is something that im going to start working on shortly.

Just one comment on what you said about trend following systems (Iv highlighted this section in your post).

Dont you think that if you have a good entry, like that used in the study by Blackstar, when a stock makes an all-time high, that this would still perform well in sideways or bearish periods? (Not outstanding, but can beat the market perhaps?)

A stock making an all time high in a bearmarket or sideways market is a rare find, but one that does so is likely to be a champion. Say if in a bullmarket you have 50 stocks a year making all time highs, and in a bearmarket/sideways market you have only 3-4, if your entry is good enough to identify these winners, and you set your system to enable pyramiding, then theres a good chance that you will find your portfolio full of large positions in very few (champion) stocks.

The above is just my opinion, alot of it still hasnt been tested.

Comments and further discussion very much appreciated.

 

[julius]

Should have been more clear - by 'Black Box' I was refering to purely mechanical systems.

 

[tech/a]

"Black Box" means undisclosed mechanics of a system.
A Purely mechanical system need not be a Black Box.

 

[nizar]

Ok i did a walk-forward analysis on my system and it performed well, as was expected, because of the bullish market conditions.

The initial period for testing and optimisation was 01-01-1998 until 31-12-2003 (6 years).

The 3.5 year period from 01-01-2004 until 30-06-2007 was left untouched (out-of-sampel) until this morning when the walk-forward analysis was conducted.

Now the system is essentially a long-term trend following system with average holding time for winners to be about 200 days (6-7 months). Note that several of the big winners usually last for a year or longer.

To be able to compare the returns from my system to that of the market (XAO) on an annual basis, I told TradeSim to close out ALL TRADES at the end of the year (using the OR SetTriggerAtDate function as part of the ExitTrigger).

Now while this gave me an idea of yearly performance, what it also does is underestimate the results, because many of the winners are cut prematurely. This is shown because when i test it over 2-year blocks, the outperformance of the market is much more substantial.

These are the tests that i did to try to increase system robustness:

*Delisted stocks are always included. In fact, excluding these results in slightly decreased performance!

*Run monte carlo simulations (20,000) on a year by year basis and in 2-year blocks in both the in-sample and out-of-sample. This reduces start and end date biases.
- In 2001, average return was 16.16%, with 99.93% of portfolio tests being profitable.
- In 2002, average return was 12.17% with 99.97% of portfolio tests being profitable
- In any 2-year block, the returns were always greater than the sum of the single years, and every portfolio being profitable.

*Run the system over a period I like to call, THE WORST, from 01-07-2001 until 01-03-2003. Pretty much peak to trough of the closest thing we've had to a bearmarket the last 15 years. XAO was -18% over this period. My system on average gained 8%, 82.4% of the portfolios in the green.

*Remove the best 3 and worst 3 trades from the trade database. Difference in results compared to when these are included is negligible.

*Model in worst case slippage. Profitability drops by 40-50%. Is this acceptable?

Any other further ideas as to how to increase system robustness would be appreciated (before i get too excited!).

Thanks,
Nizar.

 

[stevo]

Nizar
One comment that I would make, aside from a pretty thorough testing approach, is that closing out the trades prematurely on a long term system is introducing another exit criteria into the strategy.

I replicated the test period you used for some ideas that I had recently, closing the trades as you did, as well as letting them run to conclusion. So buy orders were taken during the test period (2001-2002), but in one test the trades were closed at the end, whilst with the other test let them run until the system exit criteria were met.

I found that results, in this test and over this time period, were improved slightly when trades were automatically closed at the end of the test period. The results from this approach would be different from actual trading and would be dependent on the end date chosen.

I attached results from 2 tests. "ATR stop 2001 2002" closes trades at the end of the test period, whilst the other file doesn't. I guess that the differences are not all that big but, my thoughts are that just closing the trades at the end of the test period, especially for long term systems, will give a slightly distorted picture. With shorter term systems this may not be an issue.

Also if you step through 2 year periods, moving the start date forward 3 months at a time then the results will show the variability due to different startup times. Startup can be tough with a long term system because sometimes you won't see any decent closed trades profits for quite some time.

regards
stevo

 

[nizar]

Nizar
One comment that I would make, aside from a pretty thorough testing approach, is that closing out the trades prematurely on a long term system is introducing another exit criteria into the strategy.

I replicated the test period you used for some ideas that I had recently, closing the trades as you did, as well as letting them run to conclusion. So buy orders were taken during the test period (2001-2002), but in one test the trades were closed at the end, whilst with the other test let them run until the system exit criteria were met.

I found that results, in this test and over this time period, were improved slightly when trades were automatically closed at the end of the test period. The results from this approach would be different from actual trading and would be dependent on the end date chosen.

I attached results from 2 tests. "ATR stop 2001 2002" closes trades at the end of the test period, whilst the other file doesn't. I guess that the differences are not all that big but, my thoughts are that just closing the trades at the end of the test period, especially for long term systems, will give a slightly distorted picture. With shorter term systems this may not be an issue.

Also if you step through 2 year periods, moving the start date forward 3 months at a time then the results will show the variability due to different startup times. Startup can be tough with a long term system because sometimes you won't see any decent closed trades profits for quite some time.

regards
stevo

Thank you for your thoughts, Stevo.