Testing a moving average crossover system

[howardbandy]

Brad,



You may have misinterpreted my post, my point being that it is a meaningless test to do in the first place.

What is the point of doing any test if the results cannot be duplicated in the real world, all it does is waste time and brainpower. Far better/only place to start at, is the real world situation, as it existed at the time, and walk forward.

brty

Hi Brty --

You will probably not find a stronger advocate of walk forward testing than I am.

If a test shows that something did work in the past, that is no guarantee that it will work in the future. Additional testing and validation is required.

On the other hand, if a test shows that something did not work in the past, there is very little chance that it will work in the future.

The point of my posting was to show that something did not work well in the past and, therefore, is not likely to work well in the future.

Thanks,
Howard

 

[howardbandy]

A concern that I have with any system is that initial results may show some excellent profit or devastating loss........ based on a few trades that are a small portion of the whole.....

Would it be prudent to limit the size of the profits and losses from the outset so that the trade sizes would be uniform to begin with..... ie....evaluate to see if there is a good distribution of profitable trades throughout the test period....... and then look for increased profits one the basis of the system is established.

I am not of a math background so I am not sure if it makes sense........... lol

Someone else may have some insights.

quote from 'Mechanical Trading Systems'by Weissman

In his book, Design, Testing and Optimization of Trading Systems,
Robert Pardo discusses a phenomenon that he calls outlier curve fitting.
Outlier curve fitting occurs when a single trade makes up a disproportionate
percentage of a trading system’s profits (Pardo specifically warns
against performance histories in which a single trade accounts for over 30
percent of a system’s profits.9) Certain types of trading systems are more
susceptible to this problem than others. In general, because mean reversion
trading systems exit with profits when the market reverts to its mean, outlier
curve fitting is unlikely. By contrast, trend-following systems have a
much greater tendency to contain single, disproportionately large profitable
trades within their performance histories.

Hi Dave --

If you plan to test the statistical significance of your testing results --

1. Only the out-of-sample results are useful for testing. In-sample results significantly over estimate the results -- regardless of whether you are looking at compound annual profit, win/loss ratio, expectancy, ...
2. A common validation test is to test to see whether the test results are significantly different than random or zero or buy and hold. A t-test is a common test. In order for the t-test to show significance at the 0.05 level or higher, it must score at or above about 2.0. The t-test is proportional to the mean, inversely proportional to the standard deviation, and proportional to the square root of the number of test points.

In almost all -- maybe all -- trading systems, the standard deviation of the metric being tested is greater than the mean. (If someone can show me, including the code, a system where the standard deviation of expectancy of closed trades on a major equity index or liquid futures contract is less than the mean in out-of-sample testing, I will buy them dinner and drinks when I am in Melbourne in October.) If you compute the t-test and do not show statistical significance, there are three ways to improve the score:
Increase the mean.
Decrease the standard deviation.
Increase the number of data points.

All three of these are much easier to do with systems that trade frequently and hold only a short time. It is very difficult to validate a system that uses monthly bars and holds for several months. It is relatively easy to validate a system that uses daily or intra-day bars and holds for a few days.

One of the characteristics of systems that hold for long periods is the reliance on a few large winning trades. True, you do not care about the fact that a winning trade is an outlier, but it does distort the data and increase the standard deviation.

Thanks,
Howard

 

[brty]

Howard,

My apologies, your initial post sounded like you had taken the S&P500 stocks from today and gone back to those that existed in '93 and tested them, rather than the walk forward of the stocks from the '93 S&P500.

This is the bit that was ambiguous to me....

When tested on the 500 stocks currently in the S&P 500, using end-of-day data from Quotes Plus that goes back to about 1993 (for companies with history that long)

brty

 

[howardbandy]

Howard,

My apologies, your initial post sounded like you had taken the S&P500 stocks from today and gone back to those that existed in '93 and tested them, rather than the walk forward of the stocks from the '93 S&P500.

This is the bit that was ambiguous to me....



brty

Hi Brty --

I did use the stocks that are the current constituents of the S&P 500. And I did test them from the 90s forward. But I did use a walk forward testing procedure on those stocks over that period of time, and only the out-of-sample results were reported. Unless I had intentionally included some stocks that were delisted sometime during that period, any list I tested would suffer survivorship bias. The list is unimportant, and the survivorship bias is unimportant in this instance -- the walk forward testing was the point of my test and posting.

Thanks for listening,
Howard

 

[brty]

I'm sorry Howard, but I have to go back to my original stance that the test is meaningless, and there is no point in doing it.

By taking todays stocks in the index, and testing them from the past to today, will always have the bias. Only taking the stocks that existed in the index in '93 , all 500 of them and walking forward to today has any meaning.

If you were to buy todays stocks that make up the S&P500, in 16 years time many of them will not be in the S&P500. There will have been takeovers, and de-listings, bankruptcies, or just flicked out of the top index, (but still exist).
To do any test on the existing stocks only, cannot have any meaningful trading relevancy.

Unless I had intentionally included some stocks that were delisted sometime during that period, any list I tested would suffer survivorship bias.

You have to include ALL the stocks that were in the index in '93, including those that were subsequently delisted. You may only have 300 out of the 500 left.

The list is unimportant, and the survivorship bias is unimportant in this instance -- the walk forward testing was the point of my test and posting.

I completely disagree. The list has to have meaning if the results of the test are to have any meaning. GIGO.

brty

 

[beamstas]

brty;

You missed the point of my post, something which Howard commented on.

By only using currently listed stocks, the results will be better than normal, because you are only testing the stocks that survived. Even with "better than normal" results, the moving average crossover system doesn't make a profit. Therefore there is no point in going any further to remove survivorship bias, it's pointless. Like i said, you don't need to remove survivorship bias to prove a system doesn't work, you only need to remove it to prove it does work!


Brad

 

[tech/a]

By taking todays stocks in the index, and testing them from the past to today, will always have the bias. Only taking the stocks that existed in the index in '93 , all 500 of them and walking forward to today has any meaning.

By this logic then the only walk forward testing that would be benificial to you would be realtime and you will only know the results after the period of testing.
Which will then be immediately useless as the next period could well have a few more delisted stocks!


I have learnt to drive a car.
Today I didnt have an accident tommorow and the next day maybe different.
What I learnt from driving the car wont be less benificial going forward even if I keep having accidents!

 

[brty]

Brad,

I fully understand your point about this test being "better than normal" yet still failing. My point was that even if it showed good results, it did not prove anything, because those results had no meaning going forward.

Tech,

I think you misunderstand, by taking ALL the stocks from the S&P500 in '93 and using these results, would give an expectation of future results, these could be potentially useful. By only using the CURRENT stocks in the S&P500, including those that were not part of the index in '93 (but became part of it because of their growth in market cap), means a huge bias in the results.

Excluding all the de-listed companies, which have probably gone down in price, is not close to the real world situation.

Hopefully I am not the only person who can see something wrong with tests that involve only CURRENT stocks and work with them from 16 years ago, and expect some real world performance from the results.

brty

 

[tech/a]

Oh no I understand your point very clearly.
What your missing is WHY the system makes money.
It will be either more wins than losses or bigger wins than losses.
Wether you have delistings in your portfolio or not will only determine your profit and if you have quite a few trades in a portfolio this will be suprisingly minor.Unless of course you have designed a system which finds stocks prior to delisting!
If you then run 5000 Montecarlo sims and the deviation is low ,you forward test R/T and out of sample even out of Data Set and your still profitable,I personally think Id be ready to go.

 

[howardbandy]

Brad,

I fully understand your point about this test being "better than normal" yet still failing. My point was that even if it showed good results, it did not prove anything, because those results had no meaning going forward.

Tech,

I think you misunderstand, by taking ALL the stocks from the S&P500 in '93 and using these results, would give an expectation of future results, these could be potentially useful. By only using the CURRENT stocks in the S&P500, including those that were not part of the index in '93 (but became part of it because of their growth in market cap), means a huge bias in the results.

Excluding all the de-listed companies, which have probably gone down in price, is not close to the real world situation.

Hopefully I am not the only person who can see something wrong with tests that involve only CURRENT stocks and work with them from 16 years ago, and expect some real world performance from the results.

brty

Hi Brty --

I may be missing your point, but it seems you are saying:

If I backtest, even using walk forward, on any stock that is still trading today, my results are meaningless. Instead, I must backtest on stocks that have been delisted and are no longer trading.

My point is: The S&P 500 just provides a list of stocks. Whatever results my testing shows, I can only apply that system to stocks that are trading today -- I wish I could trade yesterday's newspaper, but I cannot.

Or have I missed your point completely?

Thanks,
Howard

 

[TradeSim]

brty;

You missed the point of my post, something which Howard commented on.

By only using currently listed stocks, the results will be better than normal, because you are only testing the stocks that survived. Even with "better than normal" results, the moving average crossover system doesn't make a profit. Therefore there is no point in going any further to remove survivorship bias, it's pointless. Like i said, you don't need to remove survivorship bias to prove a system doesn't work, you only need to remove it to prove it does work!


Brad

We added survivorship bias free back testing to TradeSim and it can make a marked difference to the results. What may appear to test well with survivorship bias can perform rather poorly without survivorship bias I was a bit surprised with the difference in results myself

A comparison of results can be viewed in the following document:-

http://www.compuvision.com.au/Downl...hip_Bias_Free_Back_Testing_using_TradeSim.pdf

regards
david

 

[tech/a]

David.

Simply awesome.
How on earth do you get the time!

 

[howardbandy]

We added survivorship bias free back testing to TradeSim and it can make a marked difference to the results. What may appear to test well with survivorship bias can perform rather poorly without survivorship bias I was a bit surprised with the difference in results myself

A comparison of results can be viewed in the following document:-

http://www.compuvision.com.au/Downl...hip_Bias_Free_Back_Testing_using_TradeSim.pdf

regards
david

Hi David --

Very well done! This looks like a nice tool.

Thanks,
Howard

 

[beamstas]

We added survivorship bias free back testing to TradeSim and it can make a marked difference to the results. What may appear to test well with survivorship bias can perform rather poorly without survivorship bias I was a bit surprised with the difference in results myself

A comparison of results can be viewed in the following document:-

http://www.compuvision.com.au/Downl...hip_Bias_Free_Back_Testing_using_TradeSim.pdf

regards
david

Thanks David.

I am aware of the difference between systems which have excluded survivorship bias, and those that havn't, as i ilke to test my systems without survivorship bias. It can be done, rather crudely, in other programs. I'll need to pick up a copy of TradeSim to have a look at this tool Does it require the user to upload their own watchlists of the indexes at certain dates? This is what i've been doing in Amibroker, though finding historial watchlists in a right pain in the a$$.

Next thing on my wishlist (read hopelist) is a simple way to transfer results from amibroker into tradesim for monte carlo testing. It seems that it will be quite difficult to do.

Brad

 

[TradeSim]

Thanks David.

I am aware of the difference between systems which have excluded survivorship bias, and those that havn't, as i ilke to test my systems without survivorship bias. It can be done, rather crudely, in other programs. I'll need to pick up a copy of TradeSim to have a look at this tool Does it require the user to upload their own watchlists of the indexes at certain dates? This is what i've been doing in Amibroker, though finding historial watchlists in a right pain in the a$$.

Next thing on my wishlist (read hopelist) is a simple way to transfer results from amibroker into tradesim for monte carlo testing. It seems that it will be quite difficult to do.

Brad

Yes the biggest problem is building the filter tables which is essentially a list of all securities and their inclusive date ranges in the index over the whole period of interest. We provide a tool to assist in building this table which takes data from the S&P website and reconstructs the filter table. However the data from the S&P site can contain errors and omissions so some additional work usually needs to be done. Also you have to be careful of de-listed and expired securities as well as name changes. Your data provider should provide de-listed securities otherwise it is not possible to remove the bias.

I believe TradeSim can work with Amibroker. Some information here https://www.aussiestockforums.com/forums/showthread.php?t=8849

regards
david