System Robustness

[howardbandy]

Hi Gorilla --

Really good point Howard. It appears to mean that stats generated from testing a system on data flawed in the ways you describe must be dubious. But no-one seems to have ASX data that isn't adjusted for splits or with the dividends adjusted into the share price. I've seen the Blackstar study and agree it would be far nicer to have this kind of data to test with (not to mention a CPI adjusted liquidity filter...this is obviously far easier to do). Without this kind of data, what do you do? This could be yet another reason why the big boys don't play in this part of the market..the Aust market isn't big enough to warrant fixing 22 years of data for testing systems like what the Blackstar guys do in the US.

The problem of finding high quality historical data is not unique to the Australian markets.

Even for US stocks, historical data that lists the actual prices and volumes as traded, unadjusted for splits, distributions, and dividends, is very expensive.

To remove survivor bias, the data should include delisted issues as well.

Once the data is located, the problem is further complicated Because the data is unadjusted. As most of you who have tried to clean your data know, it is often impossible to tell when a 10 or 20 or 50% price change is a price change or a split. The trading system software must be able to cope with the discontinuities, which few (none?) can. For comparison, think about the way trading system packages that are designed to work with futures contracts handle the roll-over from one front month to the next. They have an algorithm based on either a calendar or on volume that tells them when to switch, and they take the differential between the two contracts into account as they report the trading results. Neither of those methods can be applied to stock distributions.

In an ideal world: we would have perfectly clean data; there would be a field for Actual price along with Open, High, Low, Close; there would be a field for a code that identifies what happened at every discontinuity; and our development software would be able to handle all of this.

But that still does not enable us to handle situations where the fundamentals of the company change without the ticker changing and without a distribution. For example, an automobile manufacturer sells its financing division. Among US automakers, the financing division accounts for nearly all of the profit of the company. The profitability and consequent price action change, but there is no way to determine what happened without reading the press releases, and there is no way to adjust previous prices to remove the contribution of the financing division.

I do not think we can completely, or even adequately, test for the effect of historical actual prices, splits, dividends, takeovers, spinoffs, or survivor bias.

And I think it is Very dangerous to draw conclusions from long-only systems in the 1982 to 2007 time period.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

 

[nizar]

And I think it is Very dangerous to draw conclusions from long-only systems in the 1982 to 2007 time period.

Why so?

 

[Shane Baker]

Long term secular bull market for stocks.

 

[Sir Burr]

Note, certain to generate some interest -- stops hurt systems! Try to design your systems so that very few exits are caused by a stop, particularly a maximum loss stop.

Thanks,
Howard
www.quantitativetradingsystems.com

Howard,

This is very interesting. In testing any addition to a timed stop a system "degrades".

Adding the following stops (1 to 4) individually with the timed stop degrades the system with 1 being least 4 being most degrading (profit,drawdowns) as you mentioned above.

Timed stop +

1. Trailing stop % - exit next bar
2. Trailing stop % - intra day
3. Max stop loss % - exit next bar
4. Max stop loss % - exit intra day

Do/can people here trade without the above stops as a security blanket?

SB

 

[tech/a]

And I think it is Very dangerous to draw conclusions from long-only systems in the 1982 to 2007 time period.

I find this an amazing statement.
Howard is not the first to make it.Seen it often.

Would then if that period been a full on Bear market systems successfully developed around short trading be just as dangerous.Long only systems are and should be designed to catch periods exactly like these.If one had been developed before 1982 or even 1990 you would have taken part in periods for which it was designed.Whats wrong with that?

Note, certain to generate some interest -- stops hurt systems! Try to design your systems so that very few exits are caused by a stop, particularly a maximum loss stop.

Not the first time Ive seen this statement either and I have proven that to be the case in my own testing of various ideas. BUT Given

(1) The discussion previous along the lines of developing a method thats suits the individual AND
(2) The arguement that Trend systems are not to be trusted---the idea itself suggests to be profitable youd eventually need long trends in the direction your trading
AND
(3) The psychological element of applying a system.

I would have thought that most would find a combination of capital preservation and positive expectancy,coupled with turbo charged money management principals would be preferable.

 

[howardbandy]

Greetings --

Just a few comments to points made in the last few days.

1. I think the folks who suggest that we traders need to work on the psychology of trading so that we can accept the behavior of our systems have it backwards. By designing our own objective function, we encapsulate our preferences and our comfort zone into the trading systems right from the start.

If we prefer short holding periods, we should ask for short holding periods -- that is, we should put a term in our objective function that rewards short holding periods -- then trading systems that score well will already be biased toward short holding periods and we will not have to adapt ourselves to systems that have long holding periods. Substitute any characteristic that is important to you for holding period -- RAR, equity curve slope, equity curve smoothness, Sharpe ratio, commissions paid, and so forth.

2. I have no complaint about trend-following systems. If they work, use them. My caution is that the 1982 to 2007 period has been an exceptionally strong bull market in equities. Any trading system that takes long-only positions should have some way of identifying that the market conditions are not right for it, and it should exit open trades and inhibit new trades.

Some skeptical friends of mine think that we will need to wait until after the next ice age to see a bull market like this again.

Whether that turns out to be true or not, prices of equities have risen far above their mean. When prices revert to the mean, they always pass through the mean and are extended to the other extreme by about the same amount. If that happens to equity prices, the excursion could take broad market averages down by 80% or more from here. If there is a market drop, and if trend-following systems work to profit from it, those systems will not be long-only.

3. Stops hurt systems. That does not mean do not use stops. It is just an observation that there are many ways to exit from a trade, and the worst of those ways is a maximum-loss stop.

Exits can be triggered:
1. By the action of an indicator or recognition of a pattern, similar to what caused the entry.
a. The parameters can be the same as those that caused the entry, but in the other direction.
b. The parameters can be different for exit than for entry.
c. Some other indicator can be used.
2. By the price reaching a profit target.
3. By the time in the trade reaching a maximum holding period.
4. By the price falling back to the level of a trailing stop.
5. By the price falling back to the level of a maximum-loss stop.

In my experience and research, exits caused by 1, 2, or 3 are preferable. Exits caused by 5 are worst.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

 

[howardbandy]

Greetings --

One more point. Tech/a mentions expectancy and wants his trading systems to have a positive expectancy.

Expectancy is the gain or loss expected from the average trade (or bet, if you are reading about expectancy as related to gambling).

Tech/a is Absolutely correct!

In order for a trading system to have a chance of working profitably, it Must have a positive expectancy. There is no technique for position sizing or money management that will convert a system that has a negative expectancy into a profitable system. But, it is possible to apply inappropriate position sizing or money management to a system that has a positive expectancy and turn it into a losing system.

There are exactly two variables in the equation that will tell you what the final equity of a system will be: the number of trades and the expectancy expressed as a percentage.

final equity = initial equity * ((1+expectancy) ^ number of trades)

If expectancy is negative, the final equity will be less than the initial equity.

Thanks,
Howard

 

[It's Snake Pliskin]

Greetings --

Just a few comments to points made in the last few days.

1. I think the folks who suggest that we traders need to work on the psychology of trading so that we can accept the behavior of our systems have it backwards. By designing our own objective function, we encapsulate our preferences and our comfort zone into the trading systems right from the start.

If we prefer short holding periods, we should ask for short holding periods -- that is, we should put a term in our objective function that rewards short holding periods -- then trading systems that score well will already be biased toward short holding periods and we will not have to adapt ourselves to systems that have long holding periods. Substitute any characteristic that is important to you for holding period -- RAR, equity curve slope, equity curve smoothness, Sharpe ratio, commissions paid, and so forth.

2. I have no complaint about trend-following systems. If they work, use them. My caution is that the 1982 to 2007 period has been an exceptionally strong bull market in equities. Any trading system that takes long-only positions should have some way of identifying that the market conditions are not right for it, and it should exit open trades and inhibit new trades.

Some skeptical friends of mine think that we will need to wait until after the next ice age to see a bull market like this again.

Whether that turns out to be true or not, prices of equities have risen far above their mean. When prices revert to the mean, they always pass through the mean and are extended to the other extreme by about the same amount. If that happens to equity prices, the excursion could take broad market averages down by 80% or more from here. If there is a market drop, and if trend-following systems work to profit from it, those systems will not be long-only.

3. Stops hurt systems. That does not mean do not use stops. It is just an observation that there are many ways to exit from a trade, and the worst of those ways is a maximum-loss stop.

Exits can be triggered:
1. By the action of an indicator or recognition of a pattern, similar to what caused the entry.
a. The parameters can be the same as those that caused the entry, but in the other direction.
b. The parameters can be different for exit than for entry.
c. Some other indicator can be used.
2. By the price reaching a profit target.
3. By the time in the trade reaching a maximum holding period.
4. By the price falling back to the level of a trailing stop.
5. By the price falling back to the level of a maximum-loss stop.

In my experience and research, exits caused by 1, 2, or 3 are preferable. Exits caused by 5 are worst.
Thanks for listening,
Howard
www.quantitativetradingsystems.com

Howard thanks for the informative post. I agree with exiting on some criteria like you posted.

 

[Temjin]

Howard, how would you compare the correlation between the Edge Ratio of an entry indicator verse the overall expectancy of a system?

I understand that the expectancy of a system is largely a function of both entry and exit, while money management / position sizing are simply used to keep yourself in the game to exploit that expectancy and/or increase reward through increasing risk at the same time.

3. Stops hurt systems. That does not mean do not use stops. It is just an observation that there are many ways to exit from a trade, and the worst of those ways is a maximum-loss stop.

Exits can be triggered:
1. By the action of an indicator or recognition of a pattern, similar to what caused the entry.
a. The parameters can be the same as those that caused the entry, but in the other direction.
b. The parameters can be different for exit than for entry.
c. Some other indicator can be used.
2. By the price reaching a profit target.
3. By the time in the trade reaching a maximum holding period.
4. By the price falling back to the level of a trailing stop.
5. By the price falling back to the level of a maximum-loss stop.

In my experience and research, exits caused by 1, 2, or 3 are preferable. Exits caused by 5 are worst.

This is quite interesting as initially, I thought exit 4 and 5 was the best to minimise lost. But after further research and with my limited experience, I become more favourably biased toward exits caused by 1 and 2.

My "belief", yet untested, that the premature exit of an entry will degrade the whole system because you never left your winning trade to run its course because in "theory", they should have a higher probability of having a higher MFE.

That is, successful trades often rarely goes against you too far. And the use of trailing stops may limit risk, but will also limit potential returns that might reduce the overall performance of the system.

 

[tech/a]

Howard.
Your comments make complete sence.

I now realise that its is impossible for you and others speaking on such a topic to cover every aspect of your opinions and thoughts in a few paragraphs.
Like a good method it takes time to build the story so to speak.

Thanks again.

 

[tech/a]

Temjin.

I agree with your observations and I to have pondered and tested similar thoughts (although I dont have an edge ratio).

In line with 1 & 2 and also 3 being caught in trades which wallow in small profit to drawdown of initial capital has a detrimental effect.
This also raises another question
When does an entry lose its validity You enter a trade on a signal off it goes then retraces below your initial entry but above all stops.The entry signal is now gone and you are in a trade which has failed to continue given its original condition/s.
Should we stay in that trade in the hope it will continue in our direction? At what point is the entry signal invalid and if it is what are we doing holding the trade if its NOT in profit?

Ideally staying in trades which continue to accumulate open profit is what we attempt to do.
Trailing stops in my view should be widened as a trade matures in profit BUT with time and normal exit criteria still in place.

This is where system design falls into a catagory of balance.You wont (well I havent) find perfect settings for every parameter. There will always be a give and take.
Howards concept of objective function (Although we all do this to a degree have never seen it spelled out specifically) is a key element in what I or you would accept in our systems. Quite possibly different objectives/different settings/different money Management in the SAME system.

 

[Chorlton]

Hello All,

After reading through the recent posts, can somebody explain what "edge ratio" is and how it should be applied to a system?


Many Thanks....

 

[julius]

BingK,

System performance rarely has a linear response to changes in the input - hence the 'prickly' appearance the optimization graphs posted earlier.: A small shift in the underlying market state can be the straw that breaks the camels back.

It might be possible to find out more if you could post the trading stats from the system.

From what I know automated trading is fairly common within the institutions, but mostly as a tool to stagger orders to minimise market impact. I have read about a few of hedge funds that primarily trade 'black box' systems but no specifics unfortunately.

 

[Temjin]

Hello All,

After reading through the recent posts, can somebody explain what "edge ratio" is and how it should be applied to a system?


Many Thanks....

Curtis Faith invented the "Edge Ratio" quite recently when he posted the idea and formula in his own company's software forum on Oct 2006. The latest book, "Way of the Turtles" also have a small section to explain this Edge Ratio and its application with examples.

P.S: He initially called it the Exclusion Ratio, but it was named as Edge Ratio in his book.

More information is available on this thread at the TradingBlox forum.

http://www.tradingblox.com/forum/viewtopic.php?t=3310

Here is the formula he used.

I developed a new ratio (new to me at least) I call the Excursion Ratio which is defined as:

Average Volatility-Adjusted Maximum Favorable Excursion
-------------------------------------------------------------------
Average Volatility-Adjusted Maximum Adverse Excursion

It is computed by:

1) Computing the MFE and MAE for each trade.
2) Dividing each by the ATR at entry to adjust for volatility and normalize across markets
3) Summing these separately and dividing by the number of trades to get the average
4) Dividing Average Volatility-Adjusted MFE by Volatility-Adjusted MAE

The most interesting of this ratio is that his initial hypothesis was that on average, a random entry should never have an edge of over 1.0 over a given period of time. His testings has shown evidences that it is such a case.

This too confirmed my initial hypothesis that market prices after random entries would exhibit no tendency to move in any particular direction.

Definitely an interesting indicator to look at and something I would spend more time in.

How much of an edge does one need to make a profit? I don't know if I could answer that question.

Though in recent threads/posts, there are claims that you can still make money (in theory) with a random entry and a random exit. Dr Van Tharp believed that it is still possible to create a profitable system with a random entry using well defined exit and money management strategies.

 

[tech/a]

Though in recent threads/posts, there are claims that you can still make money (in theory) with a random entry and a random exit. Dr Van Tharp believed that it is still possible to create a profitable system with a random entry using well defined exit and money management strategies.

Now THAT I do believe is possible.

 

[weird]

Gary Smith - How I trade for a living,

When you short, you are bucking the 200-year upward bias in the stock market. I don't like to bet against such
odds. George Soros once told Victor Niederhoffer that he had lost more money on shorting stocks than on any
other speculative activity. According to Niederhoffer, selling short stock is a ticket to the poorhouse. I'll
leave shorting to the perennial bears. They seem to have some sadistic compulsion for failure anyway.

Random selection 'may' work, considering the upward bias of the market.

If there were over 1000 bulls running up a hill, and a mate and I had a wager for the average number of bulls we selected to get to the top first, I would think that if I was given an additional selection criteria where I could remove the lame and also the bulls running in the opposite direction ... then I may have a slight advantage if he only had a random choice.

To me, the success of random selection in a long only system, only reinforces the upward bias of the market ... that is a very good thing to know, however looking over 200 years of the stock market can tell you that as well.

BTW, don't get hung up on TT as being 'the' example of a trend trading system. I personally was not over excited when I tested it on the ASX300 ... however this was not the constituent list it was originally designed for.

 

[Nick Radge]

According to Niederhoffer, selling short stock is a ticket to the poorhouse

He should know...he blew up shorting S&P 500 puts.

 

[tech/a]

BTW, don't get hung up on TT as being 'the' example of a trend trading system. I personally was not over excited when I tested it on the ASX300 ... however this was not the constituent list it was originally designed for.

Cant agree more its "AN" example of a longterm trend following system.
Personally I think as a system its pretty simple.

 

[howardbandy]

Hi Julius --

From what I know automated trading is fairly common within the institutions, but mostly as a tool to stagger orders to minimise market impact.

I have read about a few of hedge funds that primarily trade 'black box' systems but no specifics unfortunately.

Not only do hedge funds use automated trading to stagger orders and hide the size of their orders, but you can use that software, too.
http://www.advisorpage.com/modules.php?name=News&file=article&sid=709

Hedge funds may trade automated systems that appear to be black boxes to us, but the details of those systems are well understood by them.

Thanks,
Howard

 

[theasxgorilla]

My "belief", yet untested, that the premature exit of an entry will degrade the whole system because you never left your winning trade to run its course because in "theory", they should have a higher probability of having a higher MFE.

There is a lot involved in this assumption. Many (like myself) who think in terms of 1R losses and multi-R wins and 50/50 or lower win/loss percentages will say, "but it's necessary to let your winners run their course in order to achieve the positive expectacy of the system". Yes, from that point of view it is. But what if your system could hop off a winning trade and into another equal probability winning trade? Maybe you could avoid large open equity drawdowns? Maybe you reduce market exposure and Max DD for an acceptable decrease on another performance variable like CAGR?

I find that MAE/MFE analysis is best done when time-capped eg. measured daily out to say 100 days. Measuring the MAE vrs MFE on something like QBE for the last five years probably isn't that useful as providing you bought sometime after June 2002 it doesn't matter what entry method you used, the MAE/MFE ratio is going to be stupendously high and skew the results.

ASX.G