Top post there, the bold and underlined sections is just what i thought was key.
Please Howard, visit us more often to share your wisdom
I think I may have to get my hands on your book.
System Robustness
- Thread starter nizar
- Start date
Top post there, the bold and underlined sections is just what i thought was key.
Please Howard, visit us more often to share your wisdom
I think I may have to get my hands on your book.
- Joined
- 18 January 2007
- Posts
- 178
- Reactions
- 0
So can I use the ASX Historical data as an in-sample data for system development and the US Historical data for performance measurement ? Or the vice-versa ??
In my view.
Yes.
Another way is to do all your fine tuning and testing from, say, the period 1992-1999, and then once you are happy with results, you forward test it through 1999-2007.
I personally intend to do both.
ie. I will test my system across different timeframes of the same market AND also across different markets (US stocks and other otherseas bourses).
Or alternatively, segregate the ASX stocks, eg in sample may be ASX100, and out of sample may be ASX200 exclude ASX100 or ASX300 exclude ASX200 etc etc.
Howard, top post. I am curious as to the period one should allocate for the insample testing. Some have commented that you should have bearish, flatish and bullish conditions all within the insample period, which could potentially make it a fairly long period. I am interested in your views on the minimum period one should alocate for insample testing.
Me too.
I dont think there is a minimum period.
It probably depends on how longterm you wish to make your system.
But im keen to hear Howard's thoughts on this matter.
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Greetings all --
Thanks for the kind words and civilized response -- my views are upsetting to some people.
----
On the question of how to select the in-sample data and how to select the out-of-sample data.
If you are working with a single issue, say a major stock index or a specific commodity, then all of the in-sample data will be coming from one ticker and the question becomes "what periods of time should be used?" The short answer is to use whatever periods of time give good results. My feeling is that every market we model is dynamic -- non-stationary. The characteristics of the market change over time. The most obvious changes are easily measured -- the slope of a moving average, the average true range, the number of overnight gaps, as so forth. Any single version of the trading system we build to model that market is static for the entire period we use it. As long as the underlying market does not "move" too much, the model continues to accurately represent it, and the buy and sell signals continue to be profitable. Eventually the characteristics of the market change and the model is out-of-synch with the market. Maybe the market will return to its earlier state, but usually it will not, and a new model is needed, so we must reoptimize. That is, we must perform the next walk-forward iteration.
The length of time that the model and the market stay in synch determines the maximum length of the out-of-sample period. While there is no theoretical need that all out-of-sample periods be the same length, it is common for them to be. The algorithms to perform the walk-forward testing automatically are easiest to implement when reoptimization is done after a set number of bars. The most often reoptimization can take place is after every bar, and that is acceptable, although computationally intensive. The least often reoptimization can take place is never, and that is acceptable as long as the model and the market stay sufficiently in synch that the trades are profitable.
Which brings up another issue I'll address here briefly, but put off for a later posting. "Will a trading system that once worked, but is now broken, ever return to profitability?" My answer is that all trading system eventually break and that broken systems very seldom return to profitability. Trading systems are unique as models go (in the sense of statistical models of physical processes), in that the act of modeling changes the process being modeled -- every profitable trade made removes some of the inefficiency that the model has identified. Every profitable trade actually made (paper trades do not count) makes it less likely that the next trade will be profitable. For example, the Donchian-style breakout systems that worked so well in the 1970s and 80s no longer work, and will probably never work again.
Back to selecting the length of the in-sample period. There must be a large enough number of data points such that the model can detect some general feature of the market. The relationship between the number of data points and the number of parameters in the model is very similar to the relationship between the number of data points and the degree of a polynomial being fit to them. As the degree of the polynomial increases, the goodness of the fit increases, but the accuracy of the prediction of the next data point does not necessarily increase. The model becomes curve-fit to the data. In some models that is desirable -- a model of the rotation of two stars around a common center of gravity. But in trading systems, there is a peak in the fitness to the in-sample data where the general features of the market have been identified and learned, but the noise has not yet been learned. To deal with this, some model development platforms use three data periods -- the first is the in-sample period used to pick the best parameter values, the second is another in-sample period used to guide the modeling process and determine when the fit has found its peak and stop, and the third is the out-of-sample period used to validate the model.
What is the practical solution? For an end-of-day trading system, try out some different lengths for the in-sample period -- say two years, one year, six months, three months. The best length will depend on the complexity of the model and the stability of that portion of the underlying market that your model is trying to match. What length for the out-of-sample period? Any length shorter than the amount of time it usually takes for the market to shift away from the model. If a model is going to be accurate and profitable, it should immediately be accurate and profitable. So try out-of-sample periods of one month, one week, one day.
I can hear someone saying "But wait, trying all these combinations and then picking the lengths is using results from out-of-sample testing to determine the model." And they are correct. But only the length of the in-sample period is being chosen by optimization, so only four data points are being tested -- two years, one year, six months, three months. But at that, there is still a contamination of the purity of the out-of-sampleness. So be careful.
----
Which leads directly to one of the other questions posted -- can a trading system be developed using one ticker and validated using other tickers? That is, can the in-sample data be one stock and the out-of-sample data be another stock? Yes, but there is a caution here as well.
First, my opinion is that there is no requirement that a trading system must work on every data series in order for it to be considered robust -- that will just never happen. For whatever fundamental reasons prices vary, I have no reason to expect that the share prices of a gold exploration company, an automobile manufacturer, a bank, a food producer, and a soft drink company all act the same. But I should see similarities within sectors -- if a model works well for one insurance company, it should work reasonably well for most other insurance companies, and maybe even for banks.
I can use one company as in-sample data to build a model, then other similar companies as out-of-sample data to validate it.
But avoid what I call "optimizing the ticker space." It is not valid to build a trading model using the data series from one ticker, then test it over, say, 500 different tickers and trade only those 10 of the 500 that were profitable. That is just curve-fitting. The 10 are probably profitable because they got lucky. If 400 were profitable, there is hope.
There is a technique that helps determine which tickers to trade. Develop a model using one ticker over one in-sample period of time, but reserve two out-of-sample periods of time. Test the model on the universe of say 500 tickers for the first out-of-sample period and count the proportion of the 500 that are profitable. Make a separate list of those that are profitable, and test them over the second out-of-sample period. If the proportion that are profitable in the second out-of-sample period is about the same as the proportion that were profitable in the first in-sample period, then the model is not robust. If the proportion that is profitable in the second out-of-sample period is high, then the model is probably robust.
----
Let me clarify one point. Throughout this posting I have been using the term "is profitable." That is shorthand for saying "score a high value using the objective function." All measurements of the merit of a trading system are made by computing the score for the objective function -- not just the profitability.
----
This has become longer, and a little more theoretical, than I intended. I hope it is useful.
Thanks for listening,
Howard
www.quantitativetradingsystems.com
Thanks for the kind words and civilized response -- my views are upsetting to some people.
----
On the question of how to select the in-sample data and how to select the out-of-sample data.
If you are working with a single issue, say a major stock index or a specific commodity, then all of the in-sample data will be coming from one ticker and the question becomes "what periods of time should be used?" The short answer is to use whatever periods of time give good results. My feeling is that every market we model is dynamic -- non-stationary. The characteristics of the market change over time. The most obvious changes are easily measured -- the slope of a moving average, the average true range, the number of overnight gaps, as so forth. Any single version of the trading system we build to model that market is static for the entire period we use it. As long as the underlying market does not "move" too much, the model continues to accurately represent it, and the buy and sell signals continue to be profitable. Eventually the characteristics of the market change and the model is out-of-synch with the market. Maybe the market will return to its earlier state, but usually it will not, and a new model is needed, so we must reoptimize. That is, we must perform the next walk-forward iteration.
The length of time that the model and the market stay in synch determines the maximum length of the out-of-sample period. While there is no theoretical need that all out-of-sample periods be the same length, it is common for them to be. The algorithms to perform the walk-forward testing automatically are easiest to implement when reoptimization is done after a set number of bars. The most often reoptimization can take place is after every bar, and that is acceptable, although computationally intensive. The least often reoptimization can take place is never, and that is acceptable as long as the model and the market stay sufficiently in synch that the trades are profitable.
Which brings up another issue I'll address here briefly, but put off for a later posting. "Will a trading system that once worked, but is now broken, ever return to profitability?" My answer is that all trading system eventually break and that broken systems very seldom return to profitability. Trading systems are unique as models go (in the sense of statistical models of physical processes), in that the act of modeling changes the process being modeled -- every profitable trade made removes some of the inefficiency that the model has identified. Every profitable trade actually made (paper trades do not count) makes it less likely that the next trade will be profitable. For example, the Donchian-style breakout systems that worked so well in the 1970s and 80s no longer work, and will probably never work again.
Back to selecting the length of the in-sample period. There must be a large enough number of data points such that the model can detect some general feature of the market. The relationship between the number of data points and the number of parameters in the model is very similar to the relationship between the number of data points and the degree of a polynomial being fit to them. As the degree of the polynomial increases, the goodness of the fit increases, but the accuracy of the prediction of the next data point does not necessarily increase. The model becomes curve-fit to the data. In some models that is desirable -- a model of the rotation of two stars around a common center of gravity. But in trading systems, there is a peak in the fitness to the in-sample data where the general features of the market have been identified and learned, but the noise has not yet been learned. To deal with this, some model development platforms use three data periods -- the first is the in-sample period used to pick the best parameter values, the second is another in-sample period used to guide the modeling process and determine when the fit has found its peak and stop, and the third is the out-of-sample period used to validate the model.
What is the practical solution? For an end-of-day trading system, try out some different lengths for the in-sample period -- say two years, one year, six months, three months. The best length will depend on the complexity of the model and the stability of that portion of the underlying market that your model is trying to match. What length for the out-of-sample period? Any length shorter than the amount of time it usually takes for the market to shift away from the model. If a model is going to be accurate and profitable, it should immediately be accurate and profitable. So try out-of-sample periods of one month, one week, one day.
I can hear someone saying "But wait, trying all these combinations and then picking the lengths is using results from out-of-sample testing to determine the model." And they are correct. But only the length of the in-sample period is being chosen by optimization, so only four data points are being tested -- two years, one year, six months, three months. But at that, there is still a contamination of the purity of the out-of-sampleness. So be careful.
----
Which leads directly to one of the other questions posted -- can a trading system be developed using one ticker and validated using other tickers? That is, can the in-sample data be one stock and the out-of-sample data be another stock? Yes, but there is a caution here as well.
First, my opinion is that there is no requirement that a trading system must work on every data series in order for it to be considered robust -- that will just never happen. For whatever fundamental reasons prices vary, I have no reason to expect that the share prices of a gold exploration company, an automobile manufacturer, a bank, a food producer, and a soft drink company all act the same. But I should see similarities within sectors -- if a model works well for one insurance company, it should work reasonably well for most other insurance companies, and maybe even for banks.
I can use one company as in-sample data to build a model, then other similar companies as out-of-sample data to validate it.
But avoid what I call "optimizing the ticker space." It is not valid to build a trading model using the data series from one ticker, then test it over, say, 500 different tickers and trade only those 10 of the 500 that were profitable. That is just curve-fitting. The 10 are probably profitable because they got lucky. If 400 were profitable, there is hope.
There is a technique that helps determine which tickers to trade. Develop a model using one ticker over one in-sample period of time, but reserve two out-of-sample periods of time. Test the model on the universe of say 500 tickers for the first out-of-sample period and count the proportion of the 500 that are profitable. Make a separate list of those that are profitable, and test them over the second out-of-sample period. If the proportion that are profitable in the second out-of-sample period is about the same as the proportion that were profitable in the first in-sample period, then the model is not robust. If the proportion that is profitable in the second out-of-sample period is high, then the model is probably robust.
----
Let me clarify one point. Throughout this posting I have been using the term "is profitable." That is shorthand for saying "score a high value using the objective function." All measurements of the merit of a trading system are made by computing the score for the objective function -- not just the profitability.
----
This has become longer, and a little more theoretical, than I intended. I hope it is useful.
Thanks for listening,
Howard
www.quantitativetradingsystems.com
- Joined
- 18 January 2007
- Posts
- 178
- Reactions
- 0
Why would your view upset someone when they make sense is beyond me.....
tech/a
No Ordinary Duck
- Joined
- 14 October 2004
- Posts
- 20,417
- Reactions
- 6,356
Its common.
Known as the tall Poppy syndrome.
Ive been at the end of it myself
And accused (sometimes rightly) of dishing it out.
I have some questions for Howard but wish to have the time to ask them in full.Perhaps tommorow.
- Joined
- 18 January 2007
- Posts
- 178
- Reactions
- 0
ok here is another attempt at getting my system off the ground. The testing universe is ASX200. It was developed on the Nasdaq 100. ( as howard bandy mentioned) . I have pasted the backtest reports from Tradesim and Amibroker. Time Period was 1 year.
Please feel free to comment, good and bad all welcome
Please feel free to comment, good and bad all welcome
Attachments
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Greetings --
The first question to ask is always "Are these results from in-sample or out-of-sample periods?"
If the answer is in-sample, then ask to see the out-of-sample results. The in-sample results give absolutely no indication of the likely profitability of trading the system.
If the answer is out-of-sample, then ask to see the equity curve from the concatenated out-of-sample runs. If the equity curve looks like the kind that you would be comfortable to trade, then learn more about the system.
Thanks,
Howard
www.quantitativetradingsystems.com
The first question to ask is always "Are these results from in-sample or out-of-sample periods?"
If the answer is in-sample, then ask to see the out-of-sample results. The in-sample results give absolutely no indication of the likely profitability of trading the system.
If the answer is out-of-sample, then ask to see the equity curve from the concatenated out-of-sample runs. If the equity curve looks like the kind that you would be comfortable to trade, then learn more about the system.
Thanks,
Howard
www.quantitativetradingsystems.com
- Joined
- 18 January 2007
- Posts
- 178
- Reactions
- 0
Howard the results are from out of sample data.
nizar My other system too suffers from really bad drawdown.
What can be done to bring it down. Help from anyone would be much appreciated.
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Greetings --
Sometimes -- drawdowns can be reduced by shortening holding periods. Or, maybe the entries are too early or too late.
Try this:
Look at the length of your average winning trade and average losing trade. Say they are each 10 days. You will be using half that length in the next step. If they have different lengths, try this with both lengths.
Leave all of your buy logic in place. Comment out your sell logic.
Use this for your experimental sell logic:
Sell = BarsSince(Buy) >=5;
Run your backtests.
If your drawdown is now lower, that is an indication that your holding period is too long. Make your exits sooner.
If your drawdown has not changed much or is higher, then your entries need work.
Thanks,
Howard
www.quantitativetradingsystems.com
Sometimes -- drawdowns can be reduced by shortening holding periods. Or, maybe the entries are too early or too late.
Try this:
Look at the length of your average winning trade and average losing trade. Say they are each 10 days. You will be using half that length in the next step. If they have different lengths, try this with both lengths.
Leave all of your buy logic in place. Comment out your sell logic.
Use this for your experimental sell logic:
Sell = BarsSince(Buy) >=5;
Run your backtests.
If your drawdown is now lower, that is an indication that your holding period is too long. Make your exits sooner.
If your drawdown has not changed much or is higher, then your entries need work.
Thanks,
Howard
www.quantitativetradingsystems.com
tech/a
No Ordinary Duck
- Joined
- 14 October 2004
- Posts
- 20,417
- Reactions
- 6,356
Howard.
Struggling with this test.
By cutting the exit back to 50% of the average trading period surely this alters the landscape of the system dramatically.
The reason for the exit is no longer valid and now becomes an arbitory look back period. Arbitory as it will vary dramatically lengths of trades.
Those trades which were the trades that caught the trend (Any trend) would be instantly depleated.
All trends regardless of length would be on average cut back dramatically.
All your doing is altering the exit criteria and then applying the results to a system which has a different criteria for exit.Results surely for both are independant of each other.To me its like comparing one system to another and expecting them both to tell me something about each other---??
Getting the head around this as well.
Wouldnt number of exits caused by initial stops be important for entries.
Infact the importance of an entry can quickly disappear the longer a stock is held.The original reason a stock is purchased becomes in significant and the reason you sell it becomes the most important focus.
Are we talking initial system drawdown or Peak to Valley drawdown.?
Arent initial drawdowns more dependant on entry and Peak to Valley Exit?
IE for Initial Drawdown I have found through testing that the optimum value is 7-12% of initial purchase price, regardless of how you set your exit.% or technical point.
Shorter exits tend to increase the number of stops above 20% which has an impact on staying on an emerging trend beyound "noise".
Longer exits cut down stop outs (20% will cut out around 95% in most cases) however tends to trap most stocks in to long periods of no mans land.The point between buy,stop and profit.
There becomes an opportunity cost as trades are stuck doing nothing.
Its been my experience that its at the ENTRY end of the trade where INITIAL drawdowns have an impact and virtually no impact on Peak to Valley D/D
and at the EXIT end where Peak to Valley drawdowns are most affected and initial drawdown virtually un affected.
Interested in your views.
Hmmm yeh i thought the same thing as well.
Does tradesim tell you initial drawdown on its own?
tech/a
No Ordinary Duck
- Joined
- 14 October 2004
- Posts
- 20,417
- Reactions
- 6,356
Although I have done this myself Ive never looked at it as sussinctly as you have put it here.
Excellent
Could this simply be data from another bourse or commodity?
Objective function
Highest value of the objective function do you simply mean those conditions which you see as your main objectives being met? Can you elaborate on the "In Black quote?"
I'm feeling like you mean optimisation of Parameters.
Isnt an in sample set of results and in particular a lengthy montecarlo test result supplying you with what I call a "Blueprint" which if your trading falls within the blueprints returned results will return you a profit within the parameters of profit returned in testing? Therefore confidence.
Think I'll become a client as well.
I dont like using Cards over the Nett---been done once---us there another way?Cheque--phone a card number through?
- Joined
- 18 January 2007
- Posts
- 178
- Reactions
- 0
optimisation of Parameters
Is it better to optimise paramaters of indicators of a system seperately or do the whole optimisation in one go. The reason I ask is doing it seperately is a bit faster and I can do it over various times while doing the whole system takes considerable time.
Is it better to optimise paramaters of indicators of a system seperately or do the whole optimisation in one go. The reason I ask is doing it seperately is a bit faster and I can do it over various times while doing the whole system takes considerable time.
tech/a
No Ordinary Duck
- Joined
- 14 October 2004
- Posts
- 20,417
- Reactions
- 6,356
Ron.
I maybe wrong but I dont think Howard is talking about the optimisation of Variables.
Optimisation of parameters I am assuming are the components of Drawdown,String of Losses,Average time held Profit to loss etc,whatever those parameters are that are important to you in your systems developement.
But to your question.
Which comes with another question.
How do you avoid Curve fitting?
What makes you believe optimisation of variables will increase profitability "Walking forward"?
I maybe wrong but I dont think Howard is talking about the optimisation of Variables.
Optimisation of parameters I am assuming are the components of Drawdown,String of Losses,Average time held Profit to loss etc,whatever those parameters are that are important to you in your systems developement.
But to your question.
Which comes with another question.
How do you avoid Curve fitting?
What makes you believe optimisation of variables will increase profitability "Walking forward"?
- Joined
- 18 January 2007
- Posts
- 178
- Reactions
- 0
Tech, you just love to make my brain work hard on a sunday
to answer you, to avoid curve fitting one can test it with one market and test it out on another market but not optomise it to that market.
To answer your last question with a question
isin't optimisation of variables a method to just skew the numbers in your favor. But Curve fitting = when you skew numbers to prove your point, to prove that you have nailed the system ??
Similar threads
