System Robustness

[bingk6]

ok here is another attempt at getting my system off the ground. The testing universe is ASX200. It was developed on the Nasdaq 100. ( as howard bandy mentioned) . I have pasted the backtest reports from Tradesim and Amibroker. Time Period was 1 year.

Please feel free to comment, good and bad all welcome

Hi ROn1n,

If the results were from out of sample testing, you've got yourself a very decent system. The CARs are very good and the drawdowns more than acceptable. Also the number of trade opportunities available to you over the one year period is more than useful. You mentioned the test period of 1 year, which specific period is that ?

I am also assuming that both the Tradesim reports and Amibroker results were from testing the same system over the same period. The thing that surprises me somewhat is that teh Amibroker report - which would only be for a single run has produced results that exceeded the best results from 20,000 simulated runs using Tradesim. Possible off course, but highly unlikely, I would have thought.

 

[theasxgorilla]

Time Period was 1 year.

I wouldn't think this is long enough. Try and smash it up a bit...run it from 1/4/01 until 31/3/03 and see how it holds up.

 

[howardbandy]

Hi Tech/a --

You ask about "find the optimum parameters -- optimum is always determined by the highest value of the objective function "

I think that developing trading systems must always begin with the person or organization deciding what they want the outcome to be.

For example, organizations that have enough capital to trade many markets may be able to trade a portfolio of systems, each of which has a low percentage of winning trades but with a high win to loss ratio for those trades taken. Individuals may be uncomfortable with that and instead prefer systems that have smooth equity curves with a high percentage of winning trades, even though the win to loss ratio is about the same.

The statistics for those two examples might be:
1: win 30%, w/l ratio 5:1
2: win 70%, w/l ratio 1:1

If a person is trying to trade a system that makes him or her nervous about calling in the order and sometimes overriding the signal, that person is suffering a cognitive dissonance. In my opinion, those books that talk about the psychology of trading are often talking about ways for the trader to convince him or her self to make the trades even though that person does not believe in the system.

I think that is all wrong. The way to begin is to decide what kind of system you, personally -- very personally -- want. Define the characteristics of that system. How many trades a year (minimum or maximum), what is the minimum percentage of trades that should be winners, what is the minimum win to loss ratio, what is the maximum percentage system drawdown, and so forth. Combine everything that is important into one objective function. That objective function has a single value. Every trading system over every ticker over every time period can be evaluated using this objective function with the result that every alternative has a single number associated with it -- the objective function score.

Be realistic about the objective function. Asking for a minimum of 80% winning trades with a minimum of 5 to 1 win to loss ratio will result in system that take one trade every five years -- when the signal does come, you will not trust it.

Having chosen the objective function first, begin evaluating possible trading systems. When choosing among alternatives, the alternative you prefer will have the highest objective function score. If you find any case where you prefer a trading system that has a lower score to one that has a higher score, then the objective function must be modified so that the system you prefer has the higher score. You must be willing to accept that the best, for you, system is the one at the top of the ranking.

Now begin optimizing. After an optimization run that evaluates thousands of alternatives, the best set of values, for you, will be the set at the top of the list.

Remember where we are headed -- automated walk-forward. In every walk-forward step: the trading system is optimized over the in-sample data; the alternatives are ranked; the values of the optimized variable that rank highest are used to trade over the out-of-sample data; the results from all the out-of-sample periods are evaluated in one take -- trade it or discard it -- to decide whether to trade the system with real money. The goal is to be able to click one button "Walk-forward," and see the concatenated out-of-sample equity curve appear on your screen. If you like the looks of the equity curve, trade the system with real money. That process only works if the objective function incorporates everything that is important to you and if alternatives with higher objective function scores are always preferred to alternatives with lower objective function scores.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

 

[howardbandy]

Hi Nizar --

Me too.
I dont think there is a minimum period.
It probably depends on how longterm you wish to make your system.

But im keen to hear Howard's thoughts on this matter.

The question is how long to make the in-sample period.

The markets we are modeling are very dynamic, are non-stationary, and are changed by every profitable trade that is made. What works keeps changing, and what used to work will never work again.

The choice of how long to make the in-sample period is often tricky. Begin by ignoring advice that by using a very long period the system will be able to recognize more possible conditions. Very long in-sample periods result in systems that are unable to recognize anything. Yes, you want your system to be able to recognize that a bear market has started and stop taking long trades. But including data back to 1980 so that it can "see" October 1987 will not help.

Think about applying a standard moving average to a set of data -- add up the values and divide by the count. The resulting number represents the average over the entire period and has a lag of one-half the number of data points. If a trading system is based on moving averages, the longer the lag is, the later the signal will be. If a moving average is being fit to a long data series, it will not fit any of it very closely -- if it is being fit to a short data series, the fit will be better.

So, my advice is to make the in-sample period as short as possible, consistent with producing good out-of-sample results. Too long an in-sample period and the system will not perform well over any part of it. Too short an in-sample period and the system will be curve-fit to the in-sample data and will not perform well out-of-sample.

I am assuming that you are developing the model using a walk-forward process, so that you have more than one in-sample period. If there is just one in-sample period, then there is just one out-of-sample period. You will need to find additional out-of-sample data to validate the system. You may be more disciplined than I am, but I cannot resist changing my models after I have seen the out-of-sample results. It is OK to do that, provided you realize that you just transformed the out-of-sample data into in-sample data. And in-sample results have no value in terms of predicting the out-of-sample performance.

Experiment until you find out what works.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

 

[howardbandy]

Hi Ronin --

ok here is another attempt at getting my system off the ground. The testing universe is ASX200. It was developed on the Nasdaq 100. ( as howard bandy mentioned) . I have pasted the backtest reports from Tradesim and Amibroker. Time Period was 1 year.

Please feel free to comment, good and bad all welcome

It looks like this is daily data. Your winning trades average 44 days and your losing trades average 13 days. Based on my analysis of US NASDAQ stocks, you would expect drawdowns to average about 15% in 44 days, and the distribution of those drawdowns is quite wide. The drawdowns reported are in the 10% to 13% range, so it appears you are doing well.

Thanks,
Howard

 

[nizar]

Post #143 and #144 probably the best Iv read on this board and others.
I think im gonna have to get that book ASAP.
Thanks Howard.

 

[theasxgorilla]

The choice of how long to make the in-sample period is often tricky. Begin by ignoring advice that by using a very long period the system will be able to recognize more possible conditions. Very long in-sample periods result in systems that are unable to recognize anything. Yes, you want your system to be able to recognize that a bear market has started and stop taking long trades. But including data back to 1980 so that it can "see" October 1987 will not help.

Hmmm, and what does one do when the system you've developed has been optimised on data that recognises ideal market conditions actually encounters less ideal market conditions?

Is it not possible that someone turning their system on today might be picking the beginning of the next great decline? Wouldn't it be helpful to know how badly the system you've developed might survive such a decline? I bet a lot of people stopped trend following equities sometime in the latter half of 2002, probably much earlier on the NASDAQ.

ASX.G

 

[R0n1n]

Bingk6 - its for the last 365 days, including the correction we got recently.

yes both the reports are for the same system and same period. I haven't got much experience system backtesting and hence I post it here to get a good feedback.

ASXG - I will run it for the time period u mention as well. I did one backtest for the last 3 years and posting the AB report.

Howard - how would one go about fixing a massive drawdown problem in a system ? The previous report was for daily data, I am still working on a weekly system.

So does a system with win 70%, w/l ratio 5:1, that does gives a couple of trades a month exist ? or is that the holy grail system urban myth lol..

Here is the report for the last 3 year back test. In-sample was Nasdaq and out sample was ASX200. One question is that Max % system drawdown over the full three years or is it per year, if its 59.45% / year I'll have to go to the drawing board again..

 

[howardbandy]

Hi Tech/a --

Think I'll become a client as well.
I dont like using Cards over the Nett---been done once---us there another way?Cheque--phone a card number through?

We use PayPal to collect funds. We never see either a PayPal account number or a credit card number. When PayPal is satisfied that you have paid, they let us know and we ship the book.

I make purchases over the net literally every day using either a credit card or a PayPal account. Great selection, no sales tax, fast shipping, no traffic jams, etc.

I, personally, have never had a bad experience using PayPal. To my knowledge, none of the customers for our books has had a data security problem using PayPal.

If you have a PayPal account with money already in it, the transaction takes about 30 seconds.

But, you do not need a PayPal account at all. You can just use your credit card for the one purchase.

Both MasterCard and VISA have a system where they create a "sub-account" with a limited credit limit and a quick expiration. If I am buying something on-line from a vendor I do not know, I always use a sub-account. Say the XYZ company has a computer utility program for $45 that I want to buy. I go to their web site. If it does not have a "secure" indicator (small padlock, etc), I do not buy. If it is secure, I fill out the address information, then run the sub-account routine and generate a credit card number with a maximum credit limit of $46 and an expiration of next month. The purchase goes through as usual.

If XYZ is a scam, they cannot charge anything else on that account because the credit limit has been used up. And if they do not send my computer utility, the credit card company will help me get a refund. Credit card companies are always on the side of the consumer. Any charge of less than US$1000 is immediately reversed whenever the customer asked it to be.

But --- if you are more comfortable mailing a check (US funds on a US bank, please), the information you need to do that is on the book's web site also.

Thanks,
Howard
www.quantitativetradingsystems.com

 

[howardbandy]

Hi Ronin --

optimisation of Parameters

Is it better to optimise paramaters of indicators of a system seperately or do the whole optimisation in one go. The reason I ask is doing it seperately is a bit faster and I can do it over various times while doing the whole system takes considerable time.

The procedure of optimizing one variable at a time is called "evolutionary operation" and is often used by industrial engineers. It works when the relationship between the variables being optimized is "well behaved." Those of you who have Quantitative Trading Systems, can read about it on page 43.

Financial data is notoriously not well behaved.

If it works (that is, if it finds the global peaks of objective function values you are looking for), use it -- it is much faster than optimizing everything at once.

Thanks,
Howard
www.quantitativetradingsystems.com

 

[howardbandy]

Greetings --

Ron.

I maybe wrong but I dont think Howard is talking about the optimisation of Variables.
Optimisation of parameters I am assuming are the components of Drawdown,String of Losses,Average time held Profit to loss etc,whatever those parameters are that are important to you in your systems developement.

But to your question.
Which comes with another question.
How do you avoid Curve fitting?
What makes you believe optimisation of variables will increase profitability "Walking forward"?

There is exactly one non-optimized, non-walk-forward method of trading. That is to throw darts at the newspaper stock page.

Every time any of us decides to use one method rather than a second, we are ranking two alternatives and choosing among them. Why stop at two alternatives? Use an optimizer and look at thousands.

If we read about a system in a magazine, we have absolutely no way to know whether that system is robust or not. It could be highly curve-fit. If it appears in an advertisement, you can be certain that it is not the poorest performing example they could find. Simply because we are not doing the curve-fitting ourselves does not insure that a system is not curve-fit. On the other hand, if we do the development work ourselves, we can tell when a system is curve-fit and does not work out-of-sample.

Every time we make a real trade with real money we are performing a walk-forward test on out-of-sample test. Why wait until real money is on the table? Use good system development, testing, and validation techniques and gain some knowledge about how the system will work before calling the broker.

Thanks,
Howard
www.quantitativetradingsystems.com

 

[howardbandy]

Hi Gorilla --

Hmmm, and what does one do when the system you've developed has been optimised on data that recognises ideal market conditions actually encounters less ideal market conditions?

Is it not possible that someone turning their system on today might be picking the beginning of the next great decline? Wouldn't it be helpful to know how badly the system you've developed might survive such a decline? I bet a lot of people stopped trend following equities sometime in the latter half of 2002, probably much earlier on the NASDAQ.

ASX.G

Financial markets are often described as being in one of three states -- trending up, trading range, or trending down. The definition of a trend often depends on the time frame.

Assume I want to trade all three conditions. The market has been in an uptrend. I know this because my trending system has been profitable. It just exited a long position as prices turned down. Is this the start of a downtrend, in which case I want to be short, or the start of a trading range, in which case I want to buy into weakness, of a minor pullback in a continuing uptrend, in which case my exit was too early?

Identification of trend is The question. I wish I knew.

(Trend following used to work fairly well for commodities, but no long does, and probably never will again. It has never worked very well for individual equities, at least for my objective function. It does work for sector mutual funds and indices -- like FSELX and XLB.)


But, practically, every system must have a way of knowing it is wrong and exiting. Once in a position, there are several ways to exit -- a signal, a time limit, a trailing stop, a loss limit stop, etc. Build at least one exit method into every trading system.

Note, certain to generate some interest -- stops hurt systems! Try to design your systems so that very few exits are caused by a stop, particularly a maximum loss stop.

Systems very rarely perform better out-of-sample than in-sample. So I should not expect to achieve the same results I saw while testing.

I do not need to know specifics of how my long-only system would have done in the market crash of 1929 or 1987. All I need to know is that it would have exited.

Thanks,
Howard
www.quantitativetradingsystems.com

 

[theasxgorilla]

I do not need to know specifics of how my long-only system would have done in the market crash of 1929 or 1987. All I need to know is that it would have exited.

Have exited, yes, and stayed out. How do you decide how it stays out? Is it at the discretion of the system designer, or has it been built into the system? If the former, fine, as long as it's recognised that intuition is an ingredient in the trading of said system. If the latter, then I absolutely think its vital to expose your system to worst case conditions and see how it holds up.

ASX.G

 

[howardbandy]

hI Ronin --

Bingk6 - its for the last 365 days, including the correction we got recently.

yes both the reports are for the same system and same period. I haven't got much experience system backtesting and hence I post it here to get a good feedback.

ASXG - I will run it for the time period u mention as well. I did one backtest for the last 3 years and posting the AB report.

Howard - how would one go about fixing a massive drawdown problem in a system ? The previous report was for daily data, I am still working on a weekly system.

So does a system with win 70%, w/l ratio 5:1, that does gives a couple of trades a month exist ? or is that the holy grail system urban myth lol..

Here is the report for the last 3 year back test. In-sample was Nasdaq and out sample was ASX200. One question is that Max % system drawdown over the full three years or is it per year, if its 59.45% / year I'll have to go to the drawing board again..

This is a three year out-of-sample result? If so, this looks pretty promising. I find looking at the equity curve very helpful. Can you post an image of it?

Thanks,
Howard

 

[howardbandy]

Greetings --

Here is the 3 year equity curve of a system that trades 2 times a month, has 70% winning trades, wins average 5%, losses average 1%. Every $1 becomes $10 in three years. Don't we all wish?

Thanks,
Howard

 

[howardbandy]

Hi Gorilla --

Have exited, yes, and stayed out. How do you decide how it stays out? Is it at the discretion of the system designer, or has it been built into the system? If the former, fine, as long as it's recognised that intuition is an ingredient in the trading of said system. If the latter, then I absolutely think its vital to expose your system to worst case conditions and see how it holds up.

ASX.G

Only mechanical systems can be tested and validated. Discretionary trading is always a possibility, but that is outside my area of expertise.

If the system is a mean-reversion system, it will want to buy weakness. In a serious bear market, there are strong rallys. Your system may identify them and buy them, or may avoid all long positions until the bear market is over -- it is all up to you and your program code. Use anything that you can design, test, and validate. Validation is the key.

Feel free to test over 1987, but testing over data earlier than that used for development has little or no meaning. In developing models for trading systems, the out-of-sample period must be more recent that the in-sample period. Every time any trading system makes a profitable trade, the market it trades becomes more efficient and more difficult to trade profitably. Testing over data that is earlier is misleadingly encouraging.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

 

[theasxgorilla]

Every time any trading system makes a profitable trade, the market it trades becomes more efficient and more difficult to trade profitably.

For very short-term systems that co-exist in markets where edges are sought out and arb'd away via massive computing power, yes, I can appreciate that this is observable and real.

But are you saying that the next time greed over powers fear on some stock somewhere that the further off into the future that I take such a trade the less likely it is that my current trend following system parameters can actually profit from any trend that manifests?

Is the entire ASX or NASDAQ or NSYE a market, or is each individual stock its own market by your definition?

ASX.G

 

[howardbandy]

Hi Gorilla --

For very short-term systems that co-exist in markets where edges are sought out and arb'd away via massive computing power, yes, I can appreciate that this is observable and real.

But are you saying that the next time greed over powers fear on some stock somewhere that the further off into the future that I take such a trade the less likely it is that my current trend following system parameters can actually profit from any trend that manifests?

Is the entire ASX or NASDAQ or NSYE a market, or is each individual stock its own market by your definition?

ASX.G

Yup. Every profitable trade reduces the likelihood that the next trade will be profitable.

Howard

 

[Nick Radge]

It has never worked very well for individual equities, at least for my objective function.

I can't disagree more.

 

[tech/a]

(Trend following used to work fairly well for commodities, but no long does, and probably never will again. It has never worked very well for individual equities, at least for my objective function.)

Worked fine for most of us over the last 5 yrs.
no longer does---from when?
Objective function---Id be interested in yours.