Drawdowns

tech/a said:

Surely you not saying ALL trend following systems take or SHOULD take every signal?

If so I cant DISAGREE more.

Are these tests in your book.
IE how to set them up and how to apply them?


Thanks,
Tech

Click to expand...

If I have developed a mechanical trading system, tested and validated it, I can only expect the real-time results to approximate the out-of-sample results if I take all the signals. If I skip some signals, I have turned that system into a discretionary system. There is no way to validate a discretionary system. So, my answer depends on whether the system is mechanical or discretionary. If it is mechanical, I must take ALL the signals. If it is discretionary, I may skip whichever ones will turn out to be losers.

Yes. Statistical testing can be done using any number of observations. (Only out-of-sample observations may be used. In-sample results have no predictive value. But I've said that enough that you all know it by now.) The ability of statistical tests to differentiate between alternatives is related to the number of observations in each alternative, and to the mean and standard deviation of the observations. More observations make the decisions easier and quicker -- that is, they give an indication that a system is broken faster. It will probably take at least ten (perhaps considerably more, depending in part on the mean and standard distribution of the trades) closed trades to have reliable statistical evidence that a system is broken. But yes, the methods are explained in the book.

------

Just a point about statistical tests of trading data.

A common question to ask is "Is the mean profit (insert your own objective function in place of profit) for trading system A greater than zero?" Or, "Is the mean profit for trading system A greater than the mean profit for trading system B?"

It is not unusual for the standard deviation to be as great as, or greater than, the mean. If so, it is very difficult to have confidence that the mean is different than zero. The two-standard deviation bands around the mean include zero.


Thanks,
Howard
www.quantitativetradingsystems.com

stevo said:

Trend-following systems are designed to take every signal, losing on most trades (70% or so), winning on a few (30% or so), with small losers and big winners.

I think that this is dependent on the markets traded. I cannot take all the signals given when I scan 500 stocks using a long term system since I don't have that much money. If I was only following 20 stocks would definately be able to take all signals.

I am reading Way of the Turtle at the moment and it looks like it will offer some good insights in terms of drawdowns and trend following systems.

Click to expand...

Hi Stevo --

If you know of, or find, trend following systems that work for the majority of common stocks, please let us in on the techniques.

Works means that the system is mechanical, and that optimization, out-of-sample testing, and validation procedures have been followed and the systems pass.

It also means that the system is profitable (or fill in your own objective function) for most common stocks, not mutual funds or exchange traded funds. (Funds, particularly sector funds, are easy.) If the portfolio being traded is selected in some way, the selection process must pass validation procedures.

And it means that the system is reasonably profitable with acceptable drawdowns for long (and, if possible, short) positions in all market conditions. Staying flat in adverse conditions is acceptable. A system that is profitable for short positions is even more valuable.

I will be happy to code up, test, and distribute results for candidate systems.

Thanks,
Howard
www.quantitativetradingsystems.com

howardbandy said:

If I have developed a mechanical trading system, tested and validated it, I can only expect the real-time results to approximate the out-of-sample results if I take all the signals. If I skip some signals, I have turned that system into a discretionary system.

Click to expand...

I'm not sure I understand this.
Wouldn't your testing take into account that you don't trade all the signals based on how you've setup your position sizing, etc.?
So your testing results would also be based on the fact that it didn't/couldn't trade all buy signals. Otherwise you would only ever be able to test that mechanical systems are profitable assuming unlimited funds which wouldn't be very useful.

Howard, my query is similar to that of rub92me.

My understanding was that you could only take positions if you have capital to take the trade. On some days, and at some points during the trading/testing period, its possible that a trader may get more entry triggers than he has capital for, so some discretion must be applied here. This discretion cannot be tested and this is exactly why we apply montecarlo analysis when backtesting. And the minimum profit from the montecarlo applies to the poor bastard that picked all the dogs and the worst possible trading route whilst trading the system.

Systems which by their design take only a low amount of trades, and consequently have less number of possible trade permutations and combinations, have extreme profit/drawdown figures somewhere acceptably close to the mean.

Such a system would be ideal to trade in the sense that you would have a very good idea of how it would perform in the future in terms of profit/drawdowns as opposed to knowing you will return somewhere between two values very far apart.

I suspect Howard is a futures trader.
I'll respond later when I'm not so damned busy!

Any form of discretion applied to a mechanical system invalidates that "mechanical tag".

It's Snake Pliskin said:

Any form of discretion applied to a mechanical system invalidates that "mechanical tag".

Click to expand...

Well if all mechanical systems were designed so that there was enough capital to buy every signal and at no point during the trading timeframe a single decision (e.g. between 2 stocks) had to be made, then there would be no need for monte carlo analysis.

nizar said:

Well if all mechanical systems were designed so that there was enough capital to buy every signal and at no point during the trading timeframe a single decision (e.g. between 2 stocks) had to be made, then there would be no need for monte carlo analysis.

Click to expand...

I guess that highlights the "imperfect nature of system design and mechanical system reality".

It's Snake Pliskin said:

Any form of discretion applied to a mechanical system invalidates that "mechanical tag".

Click to expand...

I would see the restriction of capital more as a filter rather than 'discretion'. Or do you mean that in the strictest sence mechanical systems are limitless automata, and only useful as a theoretical construct?

rub92me said:

I would see the restriction of capital more as a filter rather than 'discretion'. Or do you mean that in the strictest sence mechanical systems are limitless automata, and only useful as a theoretical construct?

Click to expand...

In my opinion it's also important to consider what it actually means to have your capital 100% invested. It means that the opportunity factor presented to your system by the market is sufficient for you to be fully invested with the maximum number of positions that you deem is ideal for your system's money management and position sizing strategy. This is a good thing. In equities it seems that the leaders break first and trend longest. If you load up on these theoretically you should get the longest trends. At the end of a trend, a la what we saw a few weeks back, all stocks turn at once and typically trigger exits in concert.

Hi Howard

I think that it comes down to your definition of common stocks. As you have alluded to ETF's or mutual funds have different trend characteristics to the S&P500. Other markets such as the ASX 300 or the lower end of the Russell 2000 also exhibit different trend characteristics and in my experience produce quite acceptable results using weekly based trend following systems with appropriate index and position sizing based filters.

I quite enjoyed your book and was struck by the fact that you preferred shorter timeframes. This, apart from a personal preference, might I may suggest, possibly reflect the characteristics of a market such as the S&P 500 rather than other markets with different trend characteristics.

Your posting is greatly appreciated by all of us I am sure. My path along system tetsing has been five years so far and I am sure it will continue for many more. I enjoy all the discussion about system testing and thank you for your contributions.

Cheers

Shane

I could be wrong but I think there are wires crossed .

If Howard designs tests and trades Futures systems then all his comments fall into place.

If you design a Futures system with ONE entity you'll take EVERY trade.
Thats what the system will do.It will take a trade exit it then take another.
You'll optimise (or Howard will) his parameters constantly on that entity.

Where everyone is getting in aknot is that most of us here are Stock traders who trade stock systems.
Our capital base wont allow us to take every single trade triggered by the system.
Thats fine as our systems testing shows us that regardless of which combination of trades we take in a portfolio of X stocks we will still have a positive expectancy (And all the other Numbers) within the upper and lower limits returned by our MonteCarlo analysis of our system tested over many 1000s of Portfolio combinations,Times and number of constituents.

To us(Me) optimising every possible portfolio varient seems pointless and testing has shown (Preliminary anyway) that doing so STILL only returns figures within those parameters found in my MonteCarlo testing of one of my methods anyway.--Hardly conclusive but unexpected--by me anyway.

I can and have tested the possiblility of trading EVERY signal generated by Techtrader and the return had I the finances of Bill Gates is rediculous and un applyable due to being able to buy more than the shares on offer in any one day.

So in summary I think we are talking similar NOT the same and therefore all is NOT applicable in every circumstance.

But I could be wrong maybe not in my mind but in perhaps others like Howard.
I'll still buy the book! There is much to be learnt.

Greetings --

Thanks for all the responses to my comments on "take every signal."

I do trade futures and I do trade common stocks. I agree that there are differences between them. But I don't think those differences are the difficulty we are having in getting together on this discussion.

I define a trading system as a combination of a trading model and a market or group of markets to be traded using that model. Usually there is no confusion and model and system can be used interchangeably. I think that is causing the difficulty in this discussion.

If I am trading just a single issue in an account, say five-year treasury notes and nothing else, a mechanical trading system will have me long, short, or flat. The system has been designed, tested, and validated using the treasury note price data, and I am satisfied that it is tradable. If what I am trading is a futures contract, the system will also specify the number of contracts to hold. While flat, the funds in that account earn money market interest, but nothing else. The system is a trading model and the treasury note series.

If I am trading a group of common stocks, the group must have been selected by some process. Selection of tickers to include must be subject to the same procedures as selection of parameters for a model. The system is a trading model that will be applied to all issues in the group, plus the price and volume series for the issues in that group.

Focusing on the system that is made up of a model and a group of common stocks. During the process of developing the system, I will see that some days there are more buy signals than I have funds to cover. The system must recognize that and handle it as part of the system. There must be some mechanical way of assigning a score to each individual signal, then taking positions in those stocks with the highest scores.

AmiBroker does this in a two step process, all within one system. The model is run over all the tickers in the group. Say the group consists of 20 stocks, each a candidate for a signal to buy, sell, or be flat. I decide ahead of time, and build into my system, how many positions I wish to hold at any given time; say it is five. Assume I am trading using end-of-day data and taking positions at the next day's open. At the end of each day, I run the system. If there are sell signals for stocks I own, those will be sold at the next open. If there are buy signals for stocks I do not own, say there are seven buy signals, each individual signal will be assigned a number called its PositionScore. If there are positions available after the sales, say three, then orders will be placed to buy three stocks -- the three chosen from among the seven are the three with the top PositionScore values. The system knows how to do this and all I see (unless I want to see the deeper details) are the Sell and Buy orders for those tickers that the system chooses.

If I want to change the stocks in the group, or add stocks to the group, that has the same implication as changing the length of a moving average. The stocks in the group are as much a component of the trading systems as the length of the moving average, and must be chosen using good system design and validation techniques.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

Howard.

All makes perfect sense.
Some of us dont have amibroker so some of us dont have the ability to rank with Amibroker.
We can and some do using other methods/filters whatever you wish to call them.
Even so not all trades triggered as a buy are taken as a trade with or without Amibroker. Evidently amibroker ranks the best buy triggered on that day by some means possibly very different to that which I may chose,dont know.

I also know from my testing that it doesnt matter which trade I take from the group triggered (I dont select a stock from a group of triggered trades at random either) that regardless of which is chosen, I can expect a return of figures within tested parameters.

Thanks for replying.
tech

Thanks for the explanation Howard - looks like we're on the same page again, semantics aside.

Nick Radge said:

Donchian style systems are too simplistic and is why they've failed over time. You need to go beyond simplistic. Anyone can bang out a breakout system in minutes these days. Some research on the way Campbell & Co have changed over time will lead to some great insights on how to improve the quality of a trend following system. I think you'll find that price and volume are now almost secondary inputs.

I should also say that many of these top fund managers being quoted look at risk adjusted returns and not necessarily absolute returns like many of us are attempting to achieve. This portends directly to the drawdown equation being discussed herein. A 25-yo who has the maturity and patience to look at a trend following method over the longer term should have greater tolerance for drawdowns than say someone who is older and more risk adverse.

Nick

Click to expand...

Hi Nick --

I agree that simplistic systems, particularly when applied to long term holding periods, are unlikely to work well. They are too easily implemented. If it turns out that one is trading profitably, there are a lot of people who will notice, begin trading that system, and quickly remove the inefficiency that system recognizes.

There are only a few inputs available for trend followers -- prices for the tradable issue (including patterns, breakouts, moving averages, and indicators), prices for intermarket issues, and seasonality (time of year). Volume data for futures is available, but is reported a day late. Commitment of Traders data is available, but is reported several days late. Government and non-government agency reports are reported many days late and are subject to later revision (if you believe them at all).

I completely agree with you about selecting systems based on highest absolute returns. When running optimization searches and walk-forward tests, using absolute return as the objective function is seldom successful. Some form of risk-adjusted return is almost always better.

[About Campbell -- Campbell describes themselves as medium to long term trend followers. Annual Performance for their US$11 billion futures fund lists these results:
2002 1.23%
2003 20.41%
2004 6.96%
2005 11.02%
2006 5.48%
2007 (thru July) -6.02%

The data comes from:
http://www.autumngold.com/Advisor/CTAProfile.php?op=profile&id=23

Am I missing something?]

Thanks,
Howard

tech/a said:

Howard.

Some of us dont have amibroker so some of us dont have the ability to rank with Amibroker.

Click to expand...

Don't have AmiBroker???

IMO, AmiBroker is spectacular. It has (almost) all the features I want in a trading systems development platform, has outstanding support from its developer, and is very reasonable in price. The features missing from the current version are being filled in very rapidly. Version 5.00 was released yesterday. Download a trial version at no cost at:
http://www.amibroker.com/

Sales pitch over.

What development platforms are forum members using? For those who have tried AmiBroker and are not using it, what features does it lack?

Thanks,
Howard

I use metastock/tradesim.
Amibroker i did consider it, but decided the formula language would be too hard to learn.
We dont all have degrees in computer science

nizar said:

I use metastock/tradesim.
Amibroker i did consider it, but decided the formula language would be too hard to learn.
We dont all have degrees in computer science

Click to expand...

LOL

Ami is easier than metamumbojumbo + Tradesim.

When I started Systems testing Amibroker wasnt around.
Tradesim came on board as a serious tool about the same time Amibroker arrived.So I have been using Metastock and Tradesim.

Similar to Nizar I was/ am reluctant to put in the time to learn a new language. However As Tradesim can be linked to Amibroker and the power of Amibroker it makes sense to have it in the toolkit particularly as its so cheap.

If you dont want to learn the code Pay someone to code up!.

Howard
Any further comments on my observation/comments?

Agree with Wayne metaspeak isnt the easiest to get your head around and Mine is only partially around it.