# Testing a moving average crossover system



## howardbandy (7 June 2009)

Greetings all --

The thread that prompted me to make this posting began (on the AmiBroker Yahoo forum) with the question of whether a system based on a simple-moving-average crossover was profitable and would remain profitable.

I used this code to test that:
// Test MA Crossover.afl
//

BuyPrice = SellPrice = C;
ShortPrice = CoverPrice = C;
SetTradeDelays(0,0,0,0);

OptimizerSetEngine("cmae");

FastMALength = Optimize("FMAL",1,1,200,1);
SlowMALength = Optimize("SMAL",10,1,200,1);

FastMA = MA(C,FastMALength);
SlowMA = MA(C,SlowMALength);

Buy = Cross(FastMA,SlowMA);
Sell = Cross(SlowMA,FastMA);
Short = Sell;
Cover = Buy;

e = Equity();

ArrowShape = Buy*shapeUpArrow + Sell*shapeDownArrow;

Plot(C,"C",colorBlack,styleCandle);
PlotShapes(ArrowShape,IIf(Buy,colorGreen,colorRed));
Plot(e,"equity",colorGreen,styleLine|styleOwnScale);

//////////////////////

Samantha suggests Buy when the monthly price is above the 10 month SMA, Go to cash when it is below.

When tested on the 500 stocks currently in the S&P 500, using end-of-day data from Quotes Plus that goes back to about 1993 (for companies with history that long), the median exposure is about 50% and the median RAR is about +9%.

As asked, the question assumes that being long or cash is the correct allocation. For many reasons, a trader may want to consider only long positions. But recent history (1984 to the present) has been a period of very strong rising markets. The future might be different. Using SPY as a surrogate for the broad market, the result of that same test are an exposure of 61% and an RAR of 12.6%.

We can ask several questions:
1. Using full knowledge of the history, what are the best parameters for a trader who wants to use only long positions?

First, we need to decide how to measure best. We'll make a few runs with different objective functions and note the results.

Net % Profit.
The best values for the parameters are 2 and 12. Net profit is 304% (every initial $1.00 becomes $4.04). Maximum drawdown is 15.3%.

CAR/MDD.
Best values are 116 and 15. Backwards. This implies the trade should be flat when the faster moving average is above the slower, be long when it is below. Net profit is 51%. Exposure is 6%, RAR is 39%.

Maximum System Drawdown.
The best result is to never trade.

Sharpe Ratio -- used by many money management firms.
Best values are 97 and 13. Backwards again. 64% return with 13% exposure, for an RAR of 22.7%.

RRR (Same as K-Ratio).
Best values are 1 and 12. 292% return, exposed 63%, RAR of 13.8%.

The best results depend very much on the definition of best.

2. If the trader was able and willing to take short positions, what is best for them?

Net % profit.
Best values are 7 and 18. 64% return, exposed 26%, RAR of 11.7%.

CAR/MDD.
Best values are 1 and 24. 63% return, exposed 25%, RAR of 12.1%.

Maximum System Drawdown.
The best result is to never trade.

Sharpe Ratio.
Best values are 5 and 22. 63% return, exposed 24%, RAR of 12.5%.

RRR.
Best values are 7 and 18. 64% return, exposed 26%, RAR of 11.7%.

3. If the trader wants a system that is always exposed, long or short, and a single set of values that signal reversals.

Net % Profit.
Best values are 1 and 22. Net profit is 534%, exposure 88% (flat at the beginning until the averages have enough data), RAR of 13.6%. Maximum drawdown is 16.3%.

CAR/MDD.
Best values are 1 and 23. Net profit is 534%, exposure 88%, RAR 13.6%.

Maximum System Drawdown.
The best result is to never trade.

Sharpe Ratio -- used by many money management firms.
Best values are 78 asnd 47. Backwards again. 76% return with 34% exposure, for an RAR of 10.3%.

RRR (Same as K-Ratio).
Best values are 7 and 18. 522% return, exposed 89%, RAR of 13.2%.

However --- those are all in-sample, backward looking results. And none of the results have more than a handful of trades. Beware of making estimates of future performance based on in-sample results (even when there are a lot of data points, but that is another topic).

4. What are the results when a period of several years is analyzed, the best parameter values select based on those years, and the future results computed? Since the moving average lengths are so long -- typically around 24 months -- the in-sample period must be longer.

Five years is a long enough in-sample period to get some results. But, in part due to the edge effects of the walk forward process, one year is not a long enough out-of-sample period.

An in-sample length of six years, out-of-sample three years, starts to give results, but the out-of-sample results are not profitable.

Six years and four years gives only two steps -- one profitable, the other not.

My conclusion is that we cannot determine whether a moving average crossover system based on monthly bars is likely to be profitable in the future or not. Sixteen years of monthly data -- 192 data points is insufficient to allow meaningful validation.

What next?

1. Add more data. We might think that adding data from earlier periods will help. Testing to find out what moving average lengths worked best in the 1930s through the 1980s might be interesting, but it is still in-sample and has no value in estimating future performance. But we could run walk forward tests beginning at an earlier date and observe more steps. Whether those results look promising or not, we are still stuck with using the data we have from 1993 through the current. So we might find out what the best lengths of the in-sample and out-of-sample periods are. But we already know that the system is not sufficiently profitable or stable over the past eight years or more to actually trade it or to use it as a filter.

2. Use short bars. Moving to weekly bars increases the amount of data to about 830 data points. Moving to daily bars increases the amopunt of data to about 4000 data points.

Using weekly data, walking forward, two years in-sample, one year out-of-sample, long only, RRR as the objective function. The results are pretty promising. The system is profitable in nine OOS periods, unprofitable in three, and does not trade in two. But the values of the parameters are the interesting part. In 11 of the 15 OOS periods, the values are backwards. That is, be long when the faster moving average is below the slower moving average, and go flat when it crosses from below to above. The values of the parameters vary quite a lot, with four of the steps having both lengths greater than 60 weeks. In the steps where the moving average periods are both below 30, 10 of 11 have the values backwards -- often the faster period is one bar.

The conclusion I draw is that the traditional thoughts on using moving average crossovers as filters for trading systems have it backwards. When run as a walk forward test, which is the only way to estimate what future performance is likely to be, the S&P 500 is mean reverting, not trend following.

Thanks for listening,
Howard


----------



## beamstas (7 June 2009)

Hi Howard

I noticed you touched on moving average crossover systems in your book, and that they are backwards. Personally i have never come across a moving average crossover system that is profitable. This is a shame as most newcomers to system development and trading always seem to start with looking at MACD.. This may be why the systems are never profitable, because the amount of people using them is too much for the market and it can start to change. 

I believe there is a good reason moving average crossover systems are backwards, and maybe a slight addition to your system could make it profitable. If you have a 116 and a 15 moving average, which you optimized for CAR (which i personally use as a yard stick), if the market is trending well enough, like it was from say 2003-2007, they 15day moving average will allow you to buy in pullbacks in part of a larger trend, as long as the price reverses and then rockets ahead of the 15day moving average, as seen in the image attached, a profit can be made.

*daily chart






*weekly chart





*monthly chart





There are some major simaliarites in all three of those charts, each one of them the price dips, and then rockets out of that dip before the moving average can catch up. Allowing us to swing trade - rather than following a trend.

I really don't believe that moving average systems will ever be profitable, simply because of the number of people who revert to it at the start of trading. The only sort of system that uses a moving average crossover will be a backwards one, because that is the type of system that over the years, will be abused the least.

I truly believe that if they are to be profitable, the focus needs to be shifted from trend following, buying when a fast ma crosses a slow ma, to being backwards and swing trading on fast price movements. The system needs to find minor pullbacks that will give the price action enough room to move up at a fast rate, so that once the fast MA catches up the price is already well ahead of both averages. 

Cheers
Brad


----------



## bingk6 (7 June 2009)

Hi Howard,

It is interesting the conclusions that you came to regarding the S&P500 being mean reverting rather than trend following. I developed my systems using ASX data many years ago and always found that trend following provided excellent results. Indeed, there are quite a few traders here in this forum and others who have done very well out of trend following systems (some even using weekly data). I had always thought that all markets were the same until I read your book (QTS) where you mentioned that the S&P 500 tended to mean revert and that the standard MA crossover methodology was in fact the incorrect way to apply it.

I got access to some US data and tested my systems on them and sure enough, they totally bombed out. They all went from >30% CAR from the ASX to <-30% CAR when used on the S&P500 stocks. Never in my wildest dreams did I anticipate that the difference between the ASX stocks and the S&P 500 stocks to be so great. The S&P500 stocks simply do not want to move !!!

I then spent quite a bit of time developing a mean reversion system, specifically to trade the S&P500 stocks. I now currently have a MR system that I am happy with and I thought that the acid test would be how it would perform using ASX data. Sure enough, the results came back almost identical to my previous experiment, the MR system totally bombed out using ASX data, with a CAR of -20%.

I used to think that one should be able to develop a sufficiently robust system to trade across all markets, whether they be futures, commdities, stocks etc etc, after all a price and volume chart is a price and volume chart right ?? Well from what I have seen, the answer is no. Each market has its own distinct characteristics and there are right and wrong ways to trade each market.

So in conclusion, I agree with your findings re S&P500 stocks. I would however be very interested to see what conclusions other traders would come to when they apply their trend following systems on S&P500 data.


----------



## daveM (23 August 2009)

For some reason I am getting  Error 30 Syntax with


OptimizerSetEngine("cmae");

I have looked at this for quite a while and do not see the error.

Can someone enlighten me please....

Thanks


----------



## daveM (23 August 2009)

I have a book by Hill, Pruitt and Hill,  called 'The Ultimate Trading Guide'.

The research done by the authors indicates that the Moving Average crossover system may be the second most profitable over a long period, with a Donchian Channel system being the most profitable.

There may well be great merit in developing this application and refining it.


----------



## howardbandy (23 August 2009)

daveM said:


> For some reason I am getting  Error 30 Syntax with
> 
> 
> OptimizerSetEngine("cmae");
> ...




Hi Dave --

It may be that you are using an old version of AmiBroker.  Bring it up to at least 5.20 and that statement will work fine.

Thanks,
Howard


----------



## howardbandy (23 August 2009)

daveM said:


> I have a book by Hill, Pruitt and Hill,  called 'The Ultimate Trading Guide'.
> 
> The research done by the authors indicates that the Moving Average crossover system may be the second most profitable over a long period, with a Donchian Channel system being the most profitable.
> 
> There may well be great merit in developing this application and refining it.




Hi Dave --

I also have that book.  And I have exchanged emails about it with John Hill.

1.  Most of the analysis of systems in that book are on in-sample data.  None is clearly identified as out-of-sample data.  In-sample results are not valuable in predicting future performance.  The creation of multi-systems is all done with in-sample data and is very misleading.

2.  Both moving average crossover and Donchian breakout systems worked well from early days of trading commodities into the 1980s, then they began to break down.  If you have a chance, run the Donchian breakout system on commodity data from the first contract to the present, and you will see the equity curve turn from rising to flat as the system was revealed, became easier to program, and became cheaper to trade.

Both moving average crossover and breakout are trend following systems.  They depend on relatively large winning trades, but there are many more losing trades than winning trades.  

The keys to successfully trading breakout systems are:
Be able to limit losses of losing trades.
Be able to let your winning trades run.
Trade a portfolio of many commodities.  
Have a lot of money in your trading account -- $1,000,000 plus.
Systematically follow precise and aggressive position sizing methods.

Trend following depends on the market being traded trending, which happens more in less mature, less sophisticated markets than in more mature markets (US common stocks and stock indexes do not trend well at all).  It can work well in well defined sectors, such as sector ETFs.

As always, do your own testing following good modeling and simulation techniques, including walk forward testing and statistical analysis of the out-of-sample results.

Thanks for listening,
Howard


----------



## tech/a (24 August 2009)

Howard.
Longterm long only trading methods are pretty popular with those with larger funds looking to self manage their investments.They dont take a lot of maintenance.

Have you had any experience with or ever looked at System switches which turn a system on or off in periods of less than idea trading conditions.In the example above prolonged Bearish periods?


----------



## brty (24 August 2009)

As no-one else has mentioned it, it must just be me, but when I read the following early on in Howards post......



> When tested on the 500 stocks currently in the S&P 500, using end-of-day data from Quotes Plus that goes back to about 1993 (for companies with history that long)




it meant a totally useless test to me. 

The survivorship bias built in makes this type of study meaningless to me.

Surely you would want to start with the stocks that constituted the S&P500 in 1993 and walk forward to have any meaning.

brty


----------



## beamstas (24 August 2009)

brty said:


> As no-one else has mentioned it, it must just be me, but when I read the following early on in Howards post......
> 
> 
> 
> ...




Actually no.

If the moving average crossover system can't profit over possibly the most stable stocks from 1993-2009 (the survivors if you will), then there is no way that it can be a good system. 

*You don't have to remove survivorship bias to prove that a system will not work, you only have to remove it to prove that it will work.*

Brad


----------



## Chorlton (24 August 2009)

bingk6 said:


> I used to think that one should be able to develop a sufficiently robust system to trade across all markets, whether they be futures, commdities, stocks etc etc, after all a price and volume chart is a price and volume chart right ?? Well from what I have seen, the answer is no. *Each market has its own distinct characteristics and there are right and wrong ways to trade each market.*




This has been my conclusion to, after looking at different Mrkts......

Chorlton


----------



## brty (24 August 2009)

Brad,



> You don't have to remove survivorship bias to prove that a system will not work, you only have to remove it to prove that it will work.




You may have misinterpreted my post, my point being that it is a meaningless test to do in the first place.

What is the point of doing any test if the results cannot be duplicated in the real world, all it does is waste time and brainpower. Far better/only place to start at, is the real world situation, as it existed at the time, and walk forward.

brty


----------



## motorway (24 August 2009)

bingk6 said:


> Hi Howard,
> 
> 
> I used to think that one should be able to develop a sufficiently robust system to trade across all markets, whether they be futures, commdities, stocks etc etc, *after all a price and volume chart is a price and volume chart right ?? Well from what I have seen, the answer is no.* Each market has its own distinct characteristics and there are right and wrong ways to trade each market.
> ...




How much is this difference an artifact
of the lens that  reality is viewed with

maybe 99.999% of the things people test and back test
are useless

But that does not mean there are things , significant things that  are not important.
It just probably means that the universe of the unimportant that people wallow in is so huge..

What do markets do ?

they consolidate 
they trend
they consolidate
they trend
consolidate = congest = reversion



> *after all a price and volume chart is a price and volume chart right ??  *



*

No PRICE IS PRICE and VOLUME IS VOLUME

BUT a chart is just a lens
which often just distorts
and makes differences seem to be where there are none
Great fodder for  a back testing wilderness

motorway*


----------



## tech/a (24 August 2009)

Chorlton said:


> This has been my conclusion to, after looking at different Mrkts......
> 
> Chorlton




Have heard this before but not found it myself. I suspect it could be another truism.



brty said:


> Brad,
> 
> You may have misinterpreted my post, my point being that it is a meaningless test to do in the first place.
> 
> brty




Not if all the tickers are included.Trading or defunct.I don't clear any of my data for this reason.Its only 4 yrs old but getting better everyday.
If its not then I see your point.



motorway said:


> How much is this difference an artifact
> of the lens that  reality is viewed with
> 
> maybe 99.999% of the things people test and back test
> ...




All true Motorway 
but the point of developing a system is a far cry from trading in a discretionary manner.

Systems are simply an application of a set of conditions repeatedly over a given data set and a given time frame,with the goal of achieving a positive expectancy when traded R/Time.


----------



## Chorlton (24 August 2009)

tech/a said:


> Have heard this before but not found it myself. I suspect it could be another truism.




Fair Point.

My comment was based on my own initial testing, between the Australian and UK Mrkts.

I looked at similar (longer) timeframes, entry and exit conditions and MM / Position Sizing models and found that a particular idea would work well in one Mrkt but poorly in the other.


----------



## daveM (24 August 2009)

Howard

I thank you for your reply....

I have your book........  lots of good material therein..... especially for a person attempting to master Amibroker...

Being new to system writing, I want to use as few variables as possible and then gradually add some items as I gain some experience with the tools....

There is a lot to absorb...

I will like to post some results once I get to the stage where I feel I can add to the discussion.

Thanks again


dave


----------



## daveM (25 August 2009)

A concern that I have with any system is that initial results may show some excellent profit or devastating loss........ based on a few trades that are a small portion of the whole.....

Would it be prudent to limit the size of the profits and losses from the outset so that the trade sizes would be uniform to begin with..... ie....evaluate to see if there is a good distribution of profitable trades throughout the test period....... and then look for increased profits one the basis of the system is established.

I am not of a math background so I am not sure if it makes sense........... lol

Someone else may have some insights.

quote from 'Mechanical Trading Systems'by Weissman

In his book, Design, Testing and Optimization of Trading Systems,
Robert Pardo discusses a phenomenon that he calls outlier curve fitting.
Outlier curve fitting occurs when a single trade makes up a disproportionate
percentage of a trading system’s profits (Pardo specifically warns
against performance histories in which a single trade accounts for over 30
percent of a system’s profits.9) Certain types of trading systems are more
susceptible to this problem than others. In general, because mean reversion
trading systems exit with profits when the market reverts to its mean, outlier
curve fitting is unlikely. By contrast, trend-following systems have a
much greater tendency to contain single, disproportionately large profitable
trades within their performance histories.


----------



## howardbandy (27 August 2009)

brty said:


> As no-one else has mentioned it, it must just be me, but when I read the following early on in Howards post......
> 
> 
> 
> ...




Brty --

You are correct.  There is a significant survivorship bias from using the current constituents.  If the members of the index were chosen at the beginning of each year and tested forward for a year, the results would be much poorer.

Thanks,
Howard


----------



## howardbandy (27 August 2009)

tech/a said:


> Howard.
> Longterm long only trading methods are pretty popular with those with larger funds looking to self manage their investments.They dont take a lot of maintenance.
> 
> Have you had any experience with or ever looked at System switches which turn a system on or off in periods of less than idea trading conditions.In the example above prolonged Bearish periods?




Hi Tech/a --

One of the reasons that long-only trend following systems have a good record over recent history is that the market has been in a once in a multi-century bull market for most of recent history.  In my opinion, that is very unlikely to reoccur in any of our lifetimes.

We should be looking for / developing systems that can recognize conditions under which they do poorly and either adjust or go flat.  And systems that trade short as well as long.

Thanks,
Howard


----------



## howardbandy (27 August 2009)

brty said:


> As no-one else has mentioned it, it must just be me, but when I read the following early on in Howards post......
> 
> 
> 
> ...




Hi Brty --

One more comment --

The test I performed did walk forward.  The selection of the issues to test was those that presently make up the S&P 500 -- it could have been any group.  The walk forward testing removes the survivorship bias of the issue selection.  If it did not, your comments would apply to any list of issues being tested.  Norgate Premium Data distributes data on issues that have been delisted as well as those that are trading today.  You can, if you wish, include a selection of delisted issues in your test universe.  It will act as a stress test, and may be worth doing.

Thanks,
Howard


----------



## howardbandy (27 August 2009)

brty said:


> Brad,
> 
> 
> 
> ...




Hi Brty --

You will probably not find a stronger advocate of walk forward testing than I am.  

If a test shows that something did work in the past, that is no guarantee that it will work in the future.  Additional testing and validation is required.  

On the other hand, if a test shows that something did not work in the past, there is very little chance that it will work in the future.  

The point of my posting was to show that something did not work well in the past and, therefore, is not likely to work well in the future.

Thanks,
Howard


----------



## howardbandy (27 August 2009)

daveM said:


> A concern that I have with any system is that initial results may show some excellent profit or devastating loss........ based on a few trades that are a small portion of the whole.....
> 
> Would it be prudent to limit the size of the profits and losses from the outset so that the trade sizes would be uniform to begin with..... ie....evaluate to see if there is a good distribution of profitable trades throughout the test period....... and then look for increased profits one the basis of the system is established.
> 
> ...




Hi Dave --

If you plan to test the statistical significance of your testing results --

1.  Only the out-of-sample results are useful for testing.  In-sample results significantly over estimate the results -- regardless of whether you are looking at compound annual profit, win/loss ratio, expectancy, ...
2.  A common validation test is to test to see whether the test results are significantly different than random or zero or buy and hold.  A t-test is a common test.  In order for the t-test to show significance at the 0.05 level or higher, it must score at or above about 2.0.  The t-test is proportional to the mean, inversely proportional to the standard deviation, and proportional to the square root of the number of test points.

In almost all -- maybe all -- trading systems, the standard deviation of the metric being tested is greater than the mean.  (If someone can show me, including the code, a system where the standard deviation of expectancy of closed trades on a major equity index or liquid futures contract is less than the mean in out-of-sample testing, I will buy them dinner and drinks when I am in Melbourne in October.)  If you compute the t-test and do not show statistical significance, there are three ways to improve the score:
Increase the mean.
Decrease the standard deviation.
Increase the number of data points.

All three of these are much easier to do with systems that trade frequently and hold only a short time.  It is very difficult to validate a system that uses monthly bars and holds for several months.  It is relatively easy to validate a system that uses daily or intra-day bars and holds for a few days.

One of the characteristics of systems that hold for long periods is the reliance on a few large winning trades.  True, you do not care about the fact that a winning trade is an outlier, but it does distort the data and increase the standard deviation.  

Thanks,
Howard


----------



## brty (27 August 2009)

Howard,

My apologies, your initial post sounded like you had taken the S&P500 stocks from today and gone back to those that existed in '93 and tested them, rather than the walk forward of the stocks from the '93 S&P500.

This is the bit that was ambiguous to me....



> When tested on the 500 stocks currently in the S&P 500, using end-of-day data from Quotes Plus that goes back to about 1993 (for companies with history that long)




brty


----------



## howardbandy (27 August 2009)

brty said:


> Howard,
> 
> My apologies, your initial post sounded like you had taken the S&P500 stocks from today and gone back to those that existed in '93 and tested them, rather than the walk forward of the stocks from the '93 S&P500.
> 
> ...




Hi Brty --

I did use the stocks that are the current constituents of the S&P 500.  And I did test them from the 90s forward.  But I did use a walk forward testing procedure on those stocks over that period of time, and only the out-of-sample results were reported.  Unless I had intentionally included some stocks that were delisted sometime during that period, any list I tested would suffer survivorship bias.  The list is unimportant, and the survivorship bias is unimportant in this instance -- the walk forward testing was the point of my test and posting.

Thanks for listening,
Howard


----------



## brty (27 August 2009)

I'm sorry Howard, but I have to go back to my original stance that the test is meaningless, and there is no point in doing it.

By taking todays stocks in the index, and testing them from the past to today, will always have the bias. Only taking the stocks that existed in the index in '93 , all 500 of them and walking forward to today has any meaning.

If you were to buy todays stocks that make up the S&P500, in 16 years time many of them will not be in the S&P500. There will have been takeovers, and de-listings, bankruptcies, or just flicked out of the top index, (but still exist).
To do any test on the existing stocks only, cannot have any meaningful trading relevancy.  



> Unless I had intentionally included some stocks that were delisted sometime during that period, any list I tested would suffer survivorship bias.




You have to include ALL the stocks that were in the index in '93, including those that were subsequently delisted. You may only have 300 out of the 500 left.



> The list is unimportant, and the survivorship bias is unimportant in this instance -- the walk forward testing was the point of my test and posting.




I completely disagree. The list has to have meaning if the results of the test are to have any meaning. GIGO. 

brty


----------



## beamstas (27 August 2009)

brty;

You missed the point of my post, something which Howard commented on.

By only using currently listed stocks, the results will be better than normal, because you are only testing the stocks that survived. Even with "better than normal" results, the moving average crossover system doesn't make a profit. Therefore there is no point in going any further to remove survivorship bias, it's pointless. Like i said, you don't need to remove survivorship bias to prove a system doesn't work, you only need to remove it to prove it does work! 


Brad


----------



## tech/a (27 August 2009)

> By taking todays stocks in the index, and testing them from the past to today, will always have the bias. Only taking the stocks that existed in the index in '93 , all 500 of them and walking forward to today has any meaning.




By this logic then the only walk forward testing that would be benificial to you would be realtime and you will only know the results after the period of testing.
Which will then be immediately useless as the next period could well have a few more delisted stocks!


I have learnt to drive a car.
Today I didnt have an accident tommorow and the next day maybe different.
What I learnt from driving the car wont be less benificial going forward even if I keep having accidents!


----------



## brty (27 August 2009)

Brad,

I fully understand your point about this test being "better than normal" yet still failing. My point was that even if it showed good results, it did not prove anything, because those results had no meaning going forward.

Tech,

I think you misunderstand, by taking ALL the stocks from the S&P500 in '93 and using these results, would give an expectation of future results, these could be potentially useful. By only using the CURRENT stocks in the S&P500, including those that were not part of the index in '93 (but became part of it because of their growth in market cap), means a huge bias in the results. 

Excluding all the de-listed companies, which have probably gone down in price, is not close to the real world situation.

Hopefully I am not the only person who can see something wrong with tests that involve only CURRENT stocks and work with them from 16 years ago, and expect some real world performance from the results.

brty


----------



## tech/a (27 August 2009)

Oh no I understand your point very clearly.
What your missing is WHY the system makes money.
It will be either more wins than losses or bigger wins than losses.
Wether you have delistings in your portfolio or not will only determine your profit and if you have quite a few trades in a portfolio this will be suprisingly minor.Unless of course you have designed a system which finds stocks prior to delisting!
If you then run 5000 Montecarlo sims and the deviation is low ,you forward test R/T and out of sample even out of Data Set and your still profitable,I personally think Id be ready to go.


----------



## howardbandy (28 August 2009)

brty said:


> Brad,
> 
> I fully understand your point about this test being "better than normal" yet still failing. My point was that even if it showed good results, it did not prove anything, because those results had no meaning going forward.
> 
> ...




Hi Brty --

I may be missing your point, but it seems you are saying:

If I backtest, even using walk forward, on any stock that is still trading today, my results are meaningless.  Instead, I must backtest on stocks that have been delisted and are no longer trading.

My point is:  The S&P 500 just provides a list of stocks.  Whatever results my testing shows, I can only apply that system to stocks that are trading today -- I wish I could trade yesterday's newspaper, but I cannot.

Or have I missed your point completely?

Thanks,
Howard


----------



## TradeSim (29 August 2009)

beamstas said:


> brty;
> 
> You missed the point of my post, something which Howard commented on.
> 
> ...




We added survivorship bias free back testing to TradeSim and it can make a marked difference to the results. What may appear to test well with survivorship bias can perform rather poorly without survivorship bias  I was a bit surprised with the difference in results myself 

A comparison of results can be viewed in the following document:-

http://www.compuvision.com.au/Downl...hip_Bias_Free_Back_Testing_using_TradeSim.pdf

regards
david


----------



## tech/a (29 August 2009)

David.

Simply awesome.
How on earth do you get the time!


----------



## howardbandy (29 August 2009)

TradeSim said:


> We added survivorship bias free back testing to TradeSim and it can make a marked difference to the results. What may appear to test well with survivorship bias can perform rather poorly without survivorship bias  I was a bit surprised with the difference in results myself
> 
> A comparison of results can be viewed in the following document:-
> 
> ...




Hi David --

Very well done! This looks like a nice tool.

Thanks,
Howard


----------



## beamstas (29 August 2009)

TradeSim said:


> We added survivorship bias free back testing to TradeSim and it can make a marked difference to the results. What may appear to test well with survivorship bias can perform rather poorly without survivorship bias  I was a bit surprised with the difference in results myself
> 
> A comparison of results can be viewed in the following document:-
> 
> ...




Thanks David.

I am aware of the difference between systems which have excluded survivorship bias, and those that havn't, as i ilke to test my systems without survivorship bias. It can be done, rather crudely, in other programs. I'll need to pick up a copy of TradeSim to have a look at this tool  Does it require the user to upload their own watchlists of the indexes at certain dates? This is what i've been doing in Amibroker, though finding historial watchlists in a right pain in the a$$. 

Next thing on my wishlist (read hopelist) is a simple way to transfer results from amibroker into tradesim for monte carlo testing. It seems that it will be quite difficult to do.

Brad


----------



## TradeSim (29 August 2009)

beamstas said:


> Thanks David.
> 
> I am aware of the difference between systems which have excluded survivorship bias, and those that havn't, as i ilke to test my systems without survivorship bias. It can be done, rather crudely, in other programs. I'll need to pick up a copy of TradeSim to have a look at this tool  Does it require the user to upload their own watchlists of the indexes at certain dates? This is what i've been doing in Amibroker, though finding historial watchlists in a right pain in the a$$.
> 
> ...




Yes the biggest problem is building the filter tables which is essentially a list of all securities and their inclusive date ranges in the index over the whole period of interest. We provide a tool to assist in building this table which takes data from the S&P website and reconstructs the filter table. However the data from the S&P site can contain errors and omissions so some additional work usually needs to be done. Also you have to be careful of de-listed and expired securities as well as name changes. Your data provider should provide de-listed securities otherwise it is not possible to remove the bias.

I believe TradeSim can work with Amibroker. Some information here https://www.aussiestockforums.com/forums/showthread.php?t=8849

regards
david


----------

