This is a mobile optimized page that loads fast, if you want to load the real page, click this text.

System Robustness

Jump to new
tech/a said:
Your results are to be expected.
They simply mimick the market.
Nothing special there.
The equity curve would look like
a chart of a composite of all those stocks
in your universe.
Click to expand...

It doesn't always mimic the market, thats the point of the study. It's the point of Monte Carlo too isn't it? A single successful run through your data is a precarious basis for trading a system with real money. And its not only the averages or the outliers but the distribution of the results.

GP, I forgot to mention, I used a universe of todays XAO500, and a liquidity filter. I don't want to swamp this thread with a mass of code...I PMed you instead.
 
AG,

I'll like to have a look at the code myself. Is this a long only system ? I'll be surprised if the system managed those sorts of numbers if it traded both long and short.
 
bingk6 said:
AG,

I'll like to have a look at the code myself. Is this a long only system ? I'll be surprised if the system managed those sorts of numbers if it traded both long and short.
Click to expand...

I would too. Long only. I'll PM you the code too.
 
theasxgorilla said:
What do you mean by this GP?
Click to expand...
Meaning with the quick example I tried, by changing the trade prices from buy/sell at close to buy high, sell low, the typical return went from around +10%pa to seriously negative.

Which to me means it's very sensitive to the actual price of the trade on the day, probably because it picks up a lot of very short trades with thin margins.

Anyway, I'll give your code a go soon. Thanks for that.

Cheers,
GP
 

A great way to test for system robustness.
 
theasxgorilla said:
after 2500 runs using 10 years of data from 1/1/97 until 31/12/06, showed a 95% chance of a CAGR > 7.7% and a 95% chance of experiencing a Max DD no worse than 33.3%. No single run had a negative CAGR or a Max DD greater than 55%.
Click to expand...
I don't have an XAO500 list, but have tried some preliminary trials of 100-300 runs over the same time period with both an all-stocks list and an ASX300 list using your exact code (less runs on the all-stocks list since it takes longer). Firstly I tried buy and sell on close, then buy high, sell low. You can see that it makes quite a lot of difference. There's also a significant difference between using all stocks and limiting it to the current ASX300.

ASX300 buy/sell on close:

Max Profit: $646,118
Min Profit: $21,970
Avg Profit: $215,475 = 12.2% pa
Median Profit: $203,032
StdDev: $107,748
Median-StdDev: $95,283
Median-2*StdDev: -$12,465

All stocks buy/sell on close:

Max Profit: $369,977
Min Profit: -$19,291
Avg Profit: $118,653 = 7.6% pa
Median Profit: $107,479
StdDev: $76,872
Median-StdDev: $30.607
Median-2*StdDev: -$46,265

ASX300 buy high, sell low:

Max Profit: $136,079
Min Profit: -$68,053
Avg Profit: $10,742 = 1.03% pa
Median Profit: $4,664
StdDev: $35,556
Median-StdDev: -$30,892
Median-2*StdDev: -$66,448

All stocks buy high, sell low:

Max Profit: -$21,770
Min Profit: -$87,851
Avg Profit: -$63,038 = -9.5% pa
Median Profit: -$64,831
StdDev: $14,413
Median-StdDev: -$79,244
Median-2*StdDev: -$93,657

To me this highlights two points:

1. This particular system is rather sensitive to the trade price on the day. Perhaps if the system was modified to make it a longer-term system (eg. increase the minimum/average hold time) then it might become less sensitive.

2. Using the current version of an ASX300 or similar universe and then backtesting over earlier periods when those stocks may not have been in that universe provides a form of survivorship bias with a demonstrable advantage. If there were no advantage in that, then you'd expect the same tests of all stocks to give similar results.

Note that I'm using free data which has only been manually adjusted when I see obvious cases, so there may be some discrepancy between these results and those with clean data. However, I don't think that discrepancy would come close to accounting for the differences mentioned in the two points above.

Cheers,
GP
 
Tried another random trading system of my own with the following properties:

- Buy with random probability of 5% at each bar.
- Sell with random probability of 0.15% at each bar.
- Minimum hold time of 30 bars.
- No stops at all.
- Position score random.
- Initial capital $1m.
- Position size 2% of equity (ie. initially $20K).
- Number of shares < 10% of 100 day volume EMA.
- Number of shares < 20% of trade day volume.
- Minimum trade $5K.
- Maximum trade $100K.
- Minimum average daily turnover on trade day $50K.
- No minimum or maximum price.
- Trade price random between high and low on trade day.
- One bar trade delays.
- All current stocks universe (or most of them). Also tried ASX300.
- Commission of $30 per trade.
- No limit on number of open positions (set to 1000).
- 10 year period from 1/1/1997 to 31/12/2006.

The buy and sell probabilities and minimum hold time were set to try and emulate a medium to long term system. After a purchase, a position is held for a minimum of 30 bars and then has 0.15% probability of being sold on any subsequent bar. The volume tests are to ensure that the trades would have been reasonably possible on the day, based on available volume. The initial capital and position size is to allow a fairly large number of positions (typically 50 to start with) but still give a good chance that they'll be above the minimum trade value of $5K. That minimum has been chosen to try and avoid the commission becoming too significant a percentage of the trade (it's essentially what I use in real trading). The random buy and sell prices have been rounded to real trade values (nearest cent, half cent, or tenth of cent depending on price).

As a return comparison, the XAO averaged 8.82% pa over the same 10 year period, from 2424.6 to 5644.3.

The results over 600 runs across all stocks are:

Max Return: 20.86%
Min Return: 4.11%
Average Return: 11.18%
Median Return: 10.68%
StdDev: 5.78%
Median-StdDev: 7.20%

Max MaxDD: -87.00% (system MaxDD)
Min MaxDD: -19.86%
Avg MaxDD: -33.38%
Median MaxDD: -32.30%
StdDev: 6.98%
Median-StdDev MaxDD: -39.28%

Max Avg bars held: 622.84
Min Avg bars held: 451.84
Avg Avg bars held: 532.20 (bit over 2 years)
Median Avg bars held: 530.91
StdDev: 28.01

Max % winners: 60.74%
Min % winners: 44.97%
Avg % winners: 53.31%
Median % winners: 53.33%
StdDev: 2.55%

Max Win Avg % Profit: 192.22%
Min Win Avg % Profit: 61.44%
Avg Win Avg % Profit: 99.02%
Median Win Avg % Profit: 95.40%
StdDev: 18.80%

Max Loss Avg % Loss: -44.28%
Min Loss Avg % Loss: -30.31%
Avg Loss Avg % Loss: -36.88%
Median Loss Avg % Loss: -36.81%
StdDev: 2.25%

The results for 600 passes using the (nearly) current ASX300:

Max Return: 23.75%
Min Return: 12.02%
Average Return: 17.52%
Median Return: 17.35%
StdDev: 6.38%
Median-StdDev: 15.14%

Max MaxDD: -29.02% (system MaxDD)
Min MaxDD: -17.05%
Avg MaxDD: -20.99%
Median MaxDD: -20.52%
StdDev: 2.14
Median-StdDev MaxDD: -22.66%

Max Avg bars held: 606.17
Min Avg bars held: 439.35
Avg Avg bars held: 527.25 (bit over 2 years)
Median Avg bars held: 526.30
StdDev: 27.17

Max % winners: 76.17%
Min % winners: 57.72%
Avg % winners: 68.58%
Median % winners: 68.51%
StdDev: 2.5%

Max Win Avg % Profit: 221.73%
Min Win Avg % Profit: 64.28%
Avg Win Avg % Profit: 106.47%
Median Win Avg % Profit: 104.54%
StdDev: 18.91%

Max Loss Avg % Loss: -33.41%
Min Loss Avg % Loss: -19.36%
Avg Loss Avg % Loss: -25.95%
Median Loss Avg % Loss: -25.87%
StdDev: 2.25%

As can be seen, there is a significant improvement in results when using the current ASX300 as the universe. The two backtest runs were otherwise identical. Also interesting to note that the random system out-performed the XAO over the same period. Perhaps that's because of the survivorship bias of not having delisted stocks in the universe any more, or just because I need to do more runs (the system is a little slow, I think due to to all the available signals and volume testing, so even 600 runs took a while).

Cheers,
GP
 
nizar said:
Amazing how not even 1 run from the 600 resulted in failure.
Click to expand...

This was exactly the entire point of the test. On the basis of a random system, how lucky or unlucky could a 'trader' with no edge be over the last 10 years. The answer seems to be, not that unlucky, and in some extreme cases very lucky. First point, consider this when you see even 10 years of outstanding results from a fund manager. The interesting thing is that if you add in a few 'common sense' components like 10x10% position sizing, 10% stoploss, and a rate-of-change momentum filter (arguably the most simple form of trend indentification there is) you can enormously increase avg. CAGR, decrease avg. MaxDD and squeeze the distributions up such that the probability of results close to the averages goes up massively. You can see the shift in distributions from adding each of these components to my test results on my blog.

http://theasxgorilla.blogspot.com/2007/08/2500.html

It isn't that I'm trying to say there isn't any risk of survivorship bias in testing this way...because there clearly could be. Although what should also be considered is that by testing delisted stocks you will introduce both positive and negative survivorshiop bias...not all stocks delist from the XAO due to failure...what are the chances that if a stock reaches the XAO it delists due to failure compared with say M&A activity??? Impossible to know.

In any case I explained the discrepancies between the XAO and the testing by three things:

1. Survivorship bias (somewhat)
2. A 4 in 5 chance of not picking an ASX100 stock. Where do the fast moving, high bang-for-buck shares live?
3. The index is market cap weighted...unless you also position size according to market cap there is a very high chance that you will place equal amounts on money into the fast moving issues as the slow moving ones, and it's the latter which ultimately determine the index.

Admittedly GPs is quite different to mine...I tried to make my system buy and sell more frequently as I wanted to mimic a more frantic trader who thinks they have an edge and are out smarting the market, but clearly don't. I also didn't want long hold times, specifically because I knew people looking at the study would say, well, your just immitating buy and hold and we all know the market has gone up for 10 years.

I retain my original comment. If you are testing on similar data for the last 10 years and you can't outperform (ie. tighter and better distribution of CAGR and Max DD from your Monte Carlo testing) a system that has no or minimal 'edge' then your system is quite likely more dependent upon market conditions than you realise.
 
GreatPig said:
1. This particular system is rather sensitive to the trade price on the day. Perhaps if the system was modified to make it a longer-term system (eg. increase the minimum/average hold time) then it might become less sensitive.
Click to expand...

This is really unusual and interesting.

My testing was done on open prices, 1 day delay after buy and sell signal. If you test with buying and selling closes, or buying highs and selling lows, what does this infer? Is buying the open an 'edge'??? If we add a reasonable assumption for slippage and get similar results buying on open, can we conclude that buying on 'open' might be an edge?
 
I would expect a totally random long-term system over all stocks to roughly have the return of the market index. In my test, I deliberately left out any stops and went for purely random sell, except for the minimum hold time which is quite short compared to the average hold time anyway. For that reason, the maximum trade draw down was nearly always 99%+, as there would always be some shares that got picked up at $5-$10 and later sold for a few cents. Typically there were a number of those in each run, but the return held despite that.

I also chose a medium to long term outlook in this test to try and give decent size moves per trade, to minimise the impact of brokerage (I could have also set brokerage to zero I guess). I'll try again later using a shorter term outlook, but that generates a lot more signals and slows the backtesting down considerably more.

Cheers,
GP
 
theasxgorilla said:
If you test with buying and selling closes, or buying highs and selling lows, what does this infer?
Click to expand...
I think it just shows how sensitive the system is to trade price. I would naturally expect shorter term systems to be more sensitive, as they would typically have smaller gains and thus a small difference in trade price could make a significant difference to the gain.

If in your tests you find that trading using opening prices gives an advantage over say closing prices, then provided you can get in during the opening election each time and get the opening price, then I guess it should be a reasonable test. However, if you regularly miss the opening price and get some other price, then it could have a significant impact on the return. If the system can be made relatively insensitive to trade price, then it really wouldn't matter when you trade during the day. Hard to do though I think with a very short-term system.

Cheers,
GP
 

Brokerage was another reason I wanted shorter hold periods and more trades. Everyone says that the first barrier to success in the markets is overcoming the built in costs. This is going to be more prevalent with the punter who trades more frequently. One of the flaws of my testing was actually that I used a fixed brokerage price. Can you use minimum and fixed in combinion with Amibroker...so as to simulate the pricing structures of the online brokers? Something for me to look into.

I made 2500 runs each time and it took the better part of a week to process all system variations. It was even worse when I made a mistake and had to re-run a test.
 
You can make brokerage quite flexible if you use the commission table option.

From some quick preliminary tests of a shorter term random system, with average hold times of about 30 bars, I get about the same average return (approx 10% pa) but the variance goes through the roof. There are both big positive and big negative results.

Will do more testing and try adding stops, etc.

Cheers,
GP
 
ASX.G
My broker charges $22 per trade upto $50k parcels.
Then $44 per trade for parcels upto $100k.

So i just put $44 each way as my transaction costs when backtesting. Overestimate it a bit.
I dont think i will be trading $100k+ parcels until some time down the track.

Also, with regard to your random testing.

I do agree with you that at least one of the reasons is that you have an 80% chance of picking a stock outside the index (ie. not a large cap) and thus have a greater chance of picking a stock with a higher bang for buck rating.

But with that said -- what do you think about GPs results in which case the test over the ASX300 outperforms the one over the whole market?

I think the minimum hold time does lift the result here.

And ASX.G,
I read your blog daily

I wish I could get mine fixed though, the damn pics wont enlarge when you click them for some reason, makes it really annoying as you cant read the charts properly.
 
The reason I think the return is better for the current ASX300 list when backtested over that many years is that you're only testing over the stocks that have now proven to be successful.

It's a bit like saying I know that PDN, ZFX, etc. have done really well over the last few years, so I'm going to restrict my universe to just them and leave out all the ones that I know haven't performed. Of course the results are going to be outstanding. Using the current ASX300 or whatever is a less-extreme example of that, the way I see it.

Cheers,
GP
 

Yeh.
Its a shame that data providers dont have historical index constituents, a real shame.
It restricts backtesting to only over the entire market.
Yeh you can put price and volume turnover filters but thats not the same as making an index your universe.
 
Radge keeps saying it.

The solution is simple.
Know WHY your system returns profit.

Know why it works as a trend following system
OR
Know why it works as a short system
OR
Know why it works as a forward and reverse system
OR
Know why it works as a combination of systems.

Ever thought of investing some $$s in Radges "Building a System" course?
 
You must log in or register to reply here.

Similar threads

Emergency Exit for Weekly System
2
Replies
27
Views
5K
MovingAverage
System optimisation: IS training vs. OOS testing, and Paper Trading
Replies
10
Views
2K
Garpal Gumnut
R
21 Lessons in Investing
Replies
6
Views
3K
divs4ever
S
Chasing participation
Replies
5
Views
4K
skyQuake
Review: Trading Systems (a new approach...) - Jaekle and Tomasini
Replies
0
Views
5K
markhocky
Menu
Log in
Register
Cookies are required to use this site. You must accept them to continue using the site. Learn more...