System Robustness

howardbandy said:

The way to begin is to decide what kind of system you, personally -- very personally -- want. Define the characteristics of that system. How many trades a year (minimum or maximum), what is the minimum percentage of trades that should be winners, what is the minimum win to loss ratio, what is the maximum percentage system drawdown, and so forth. Combine everything that is important into one objective function. That objective function has a single value. Every trading system over every ticker over every time period can be evaluated using this objective function with the result that every alternative has a single number associated with it -- the objective function score.

Click to expand...

Hi Howard,

Excellent discussion in progress, many thanks for your contributions to date. I have one further question relating to the allocation of the "objective function score". All of the system parameters used in evaluating a system must have a certain weighting towards the calculation of this objective function score. The weighting of each parameter would be dependent on the preferences of the trader.

Given that these parameters all present their values in different magnitudes (some in %, some in Dollar Values, some with really large numbers, some really small numbers etc etc), do you have any suggestion on how all of these system variables can be "normalised" (for one of better word) so that the final objective score is not overly skewed one way or another ? Presumably, you would also require various weights to be allocated to the different parameters as well, which could result in a lengthy mathematical formula used to calculate a SINGLE value which you would use as a basis for comparing systems with. Do you have suggestions as to the formulas other may be using for calculating the score ??

Hi Bingk6 --

You are correct -- the objective function is created by combining all of the factors that are important, with each factor weighted according to its importance.

There are several methods for creating weighted objective functions.

One that I like is to first list the important features -- even if they are thought to have only minor importance.

Go down the list of features and determine what metric already exists for that feature. For example, average holding period is already in days or bars; compound annual rate of return is already in percent, and so forth.

Continuing with the list, for each feature decide what value should get full marks and what value should get no credit. Draw a graph showing what value gets what percentage of full marks. Full marks get a value of 1.0, no marks get a value of 0.0. I have pasted an image I use in the book, Quantitative Trading Systems, below as an example. The feature being scored is exposure -- full marks (1.00) for anything between 10 and 20 percent, linearly dropping to half marks (0.50) for anything over 40 percent. To use the graph, lookup the exposure for the run you are scoring -- go across the chart until you come to that value, then go up until you hit the line -- that is the score for this feature for this run.

At this point each feature has been evaluated and a chart drawn showing how the marks will be assigned.

Still with the list, imagine that you have 100 dollars to allocate to the entire objective function. Go down the list and decide how many dollars each of the features is worth.

The resulting objective function is determined by calculating what mark a feature earned and multiplying by its dollar allocation. Add them all up. The result will be a score between 0 and 100.

Now -- you probably didn't get it just right the first time -- I never do. Run some backtests. For each backtest, plot the equity curve, print out the statistical report, and calculate the objective score. Spread the sheets on the floor and put them in order according to how you would rank them as you read the statistics and look at the equity curve. The ones you rank highest should have the highest objective function score. In fact, the objective function scores should go from highest to lowest with none out of order. If that is the case, your objective function is complete. If that is not the case, you will need to add other features or re-weight the ones you have. Repeat until the order you prefer is reflected in the objective function score.

In AmiBroker, you can do exactly what I have just described. Then, you can tell AmiBroker that it should automatically compute the score for your custom objective function every time it does a backtest or optimization run, and include that number on the report.

For those of you who already have the book, Quantitative Trading Systems, the procedure is described in Appendix A, starting on page 317.

Thanks,
Howard
www.quantitativetradingsystems.com

Hi Tech/a and Nick --

This will be interesting.

I was working as a research analyst for a Commodity Trading Advisor firm in the US. The firm specialized in trend following systems. They developed some of the first mechanical and computerized methods, and were very successful until the mid 1990s. Almost all of the trading was commodity futures -- treasury notes, oil, and the like. At one point, we were approached by a large fund who wanted us to manage an account for them in which we were to trade common stocks using trend following techniques. We could not get it to work. Breakouts did not work, moving average crossovers did not work.

By work, I mean develop, test and validate the models using good modeling techniques. Optimizing over the symbol space does not count. Taking advantage of an overall rising market does not count.

Fast forward to 2007. Trend following works great on market sectors, sector mutual funds, and sector exchange traded funds. In fact, I describe several methods for building models based on the sector, then trading correlated equities.

So -- If you are willing to discuss the details (if not, I understand completely) -- what common stock tickers do you find trade well using trend following techniques, and what entry methods do you find work?

Thanks,
Howard
www.quantitativetradingsystems.com

Howard.

Given conditions I'm sure you can prove that following trend trading will fail,just as given certain conditions you can also prove that trend trading is profitable.

I have watched my son prove Black is White mathematically,something I'm sure helps one doing his Doctorate in Physics.

But a cursory glance at a 100 yr chart of the DJIA or the ASX reveals a steady climb.

Taking advantage of an overall rising market does not count.

Click to expand...

Just as all you need to know is that your system will get you out BEFORE a 1987 event
You need to Know your system will GET YOU IN a Y2000 bull run.

Below is a link to a very rudimentary and simple TREND FOLLOWING system that has been traded live for 5 yrs. Its served me well.
All entries and exits are there if interested.
We (Those who took part in design and testing) are a long way from expert.

http://www.thechartist.com.au/forum/ubbthreads.php?ubb=postlist&Board=4&page=1

Nick, Howard, Tech et al;

Apologies if any of this has been covered earlier in the thread,

I think the distinction between curve fitting and optimization is worth noting,
though accurately distinguishing one from the other is certainly beyond me.

Finding the best parameters over an arbitrary length of time and then expecting them to hold true for a multitude of conditions that may be encountered in the future is IMO a fools game, but optimizing chosen variables over a specified time period with the expectation that the performance of these variables should decay into the future is perhaps a more feasible approach.

I am aware there has been a fair amount research into this area by various academics in this field, but it would be fair to say the people who are profitably using this method prefer not to disclose. Bastards

From what I have read, machine learning applications (genetic algorithms, etc) are used to dynamicaly evaluate and update variables (or perhaps even overhaul the whole model) to optimize the next periods performance, based on a previous number of periods. Considering the calibre of individuals who subscribe to 'market cycles' and similar, I don't find it that hard to swallow.

Interested to hear others thoughts on this area.

Every time any trading system makes a profitable trade, the market it trades becomes more efficient and more difficult to trade profitably.

Click to expand...

I completely agree, though could this apply more to swing trading systems than trend following? What effect would the frequency of trades have on the effective life of a system?

Also, to Nick & Howard, what has your experience been with short term trading systems? Most of the discussion in this area seems to be on medium to longer time frame systems, have you seen short term systems employed profitably?

Julius I dont think specifically.

I think the distinction between curve fitting and optimization is worth noting,
though accurately distinguishing one from the other is certainly beyond me.

Finding the best parameters over an arbitrary length of time and then expecting them to hold true for a multitude of conditions that may be encountered in the future is IMO a fools game, but optimizing chosen variables over a specified time period with the expectation that the performance of these variables should decay into the future is perhaps a more feasible approach.
I am aware there has been a fair amount research into this area by various academics in this field, but it would be fair to say the people who are profitably using this method prefer not to disclose. Bastards

From what I have read, machine learning applications (genetic algorithms, etc) are used to dynamicaly evaluate and update variables (or perhaps even overhaul the whole model) to optimize the next periods performance, based on a previous number of periods. Considering the calibre of individuals who subscribe to 'market cycles' and similar, I don't find it that hard to swallow.

Click to expand...

Like you I have pondered this topic at leangth from a logic rather than mathamatical view. My thoughts were very similar to your own which I have highlighted in italics.

In the end I have come to the following conclusion (Not saying its right or wrong just my conclusion).

If I chose a set of parameters and assign variables to them that are fixed,test and accumulate results,I can expect that the re application of them on forward data will return results within the Devaition parameters returned from any MonteCarlo analysis,provided the data doesnt step outside the boundaries of that which it was tested against.

If I optimise parameters and or Variables then I am un realistically expecting results to perform as well as they have on PAST optimised results working forward.What I have in reality is a set of parameters that are chosen by optimisation rather than "randomness" which when re applied forward will not be the optimised "Best" results looking BACK after a period of forward trading.
They are no better than random.

I have infact tested this to my best ability by optimising over a period then applying the optimised parameters over another forward period.The results are not as good as the optimised original results.
re optimising THOSE results return very different variables than those selected
from the initial optimisation.

An endless futile search for the very best selected variables.
You just wont find them as they will always alter through time.
The best you can get is an expected performance based on the "given" parameters selected for testing---wether they be optimised or randomly selected.Derived by allocation of parameters and variables tested against a data set which return a likley set of performance results deviating from the mean high and low.

If acceptable you trade if not you keep searching.

I know the explaination is circular but so is the exercise.
but hey I'm happy for my LOGIC to be challenged!

I havent got much to contribute but keep the excellent discussion coming guys. Im thoroughly enjoying it

tech/a said:

If I optimise parameters and or Variables then I am un realistically expecting results to perform as well as they have on PAST optimised results working forward.What I have in reality is a set of parameters that are chosen by optimisation rather than "randomness" which when re applied forward will not be the optimised "Best" results looking BACK after a period of forward trading.
They are no better than random.!

Click to expand...

IMO, the reason for performing optimisation is to get some sort of feel as to what has performed best in the past. That is not to say that one can expect the same level of performance going forward using the optimised parameter values, but it is nonetheless a start. There is absolutely nothing available in any past data that would indicate what the future performance is likely to be. So for me, the only kind of "edge" (if you can even call it that) is to trade a system that I know has performed well in the past, rather than one with random parameter settings. In my view, extracting what has worked in the past is pretty much all that is up for grabs when looking at past data and there are no better alternatives than that.

The key really is to extract these optimised parameter value from an in-sample set of data and then to verify it using out of sample data. By out of sample data verification, I mean performing any MonteCarlo analysis you deem necessary etc etc using the optimised parameter settings on out of sample data. If the out of sample testing shows good robustness in the figures and are relatively close to the optimised figures, then you may well have a very decent system. On the other hand, if the out of sample testing shows results that are very poor, then there is a real problem with the system and its back the drawing board.

That, in a nutshell, is my perception of the role that optimisation plays, it is merely a starting step which would hopefully lead to the formulation of a robust system that is better than random.

howardbandy said:

The resulting objective function is determined by calculating what mark a feature earned and multiplying by its dollar allocation. Add them all up. The result will be a score between 0 and 100.

Click to expand...

Hi Howard,

That is a good methodology, I'll give it a shot. Thanks

Bingk6

Similar to my thinking.
but is it really better than random.
Logic says it should be.
But no real reason why it will be.

I dont have amibroker so dont have the facilities to find optimum variables over a portfolio.
Id be interested in what they are for T/Trader and then test the results over data and with tradesim.All I need is the optimum values.
My suspicion is that the edge if any wouldnt equate to much.

anyone help out?

Interested in Nicks take.

tech/a said:

anyone help out?

Click to expand...

Sure. What do you think of these paramaters?

Parameters:

Short Breakoout, 2 -> 50 days, increment 1
Long Breakout, 20 -> 200 days, increment 1

Fast MA -> 10 -> 100, increment 1
Slow MA -> 50 -> 350, increment 1

ETA, 64 hours. I have a Core Duo damn it! Might be my clunky code implementation, but I've stripped it back. It wants to do 2+ million iterations, which might also explain it. If we trim the paramaters or increase the increments I expect it could speed up exponentially.

ASX.G

bingk6 said:

IMO, the reason for performing optimisation is to get some sort of feel as to what has performed best in the past. That is not to say that one can expect the same level of performance going forward using the optimised parameter values, but it is nonetheless a start. There is absolutely nothing available in any past data that would indicate what the future performance is likely to be. So for me, the only kind of "edge" (if you can even call it that) is to trade a system that I know has performed well in the past, rather than one with random parameter settings. In my view, extracting what has worked in the past is pretty much all that is up for grabs when looking at past data and there are no better alternatives than that.

The key really is to extract these optimised parameter value from an in-sample set of data and then to verify it using out of sample data. By out of sample data verification, I mean performing any MonteCarlo analysis you deem necessary etc etc using the optimised parameter settings on out of sample data. If the out of sample testing shows good robustness in the figures and are relatively close to the optimised figures, then you may well have a very decent system. On the other hand, if the out of sample testing shows results that are very poor, then there is a real problem with the system and its back the drawing board.

That, in a nutshell, is my perception of the role that optimisation plays, it is merely a starting step which would hopefully lead to the formulation of a robust system that is better than random.

Click to expand...

I agree wholly with the above.

theasxgorilla said:

Sure. What do you think of these paramaters?

Parameters:

Short Breakoout, 2 -> 50 days, increment 1
Long Breakout, 20 -> 200 days, increment 1

Fast MA -> 10 -> 100, increment 1
Slow MA -> 50 -> 350, increment 1

ETA, 64 hours. I have a Core Duo damn it! Might be my clunky code implementation, but I've stripped it back. It wants to do 2+ million iterations, which might also explain it. If we trim the paramaters or increase the increments I expect it could speed up exponentially.

ASX.G

Click to expand...

ASX.G Amibroker is not dual core aware by design.

R0n1n said:

ASX.G Amibroker is not dual core aware by design.

Click to expand...

I know, but at least one of the cores should be able to pump harder that this, it's 2007 don't u know !

Such small increments are rediculous over longer timeframes.

for breakouts 5 days is ample and M/A's 10.

Then we have liquidity,Closing price,Stop % and Position sizing.

What software do you use HOWARD

theasxgorilla said:

I know, but at least one of the cores should be able to pump harder that this, it's 2007 don't u know !

Click to expand...

lol... tell me about it... I have a dual core XEON 3.4 ghz with 4 gb ram and still Amibroker can gun it to its limits.

tech/a said:

Then we have liquidity,Closing price,Stop % and Position sizing.

Click to expand...

A lot of stuff to look at, right??

This is a massive undertaking. And when you optimise more than two variables at a time you lose the ability to see 3d optimisation charts that can help you see hot spots where variable placement is likely to be most robust.

The objective measure is not a bad idea, but it can tend to make the assessment of a system quite one dimensional. Looking at the equity curve probably helps. But I think there is an element of common sense and introspect that comes into deciding how to optimise and select a set of variables.

A combination of what Ed Seykota refers to as the steam-roller and the hunt-and-peck approach.

ASX.G

The comments below are my interpretation of Howard Bandy's posts to date on this thread.

If you disagree with my conclusions then please let me know so that I can establish whether I am running with the pack or running in the opposite direction and consequently about to get trampled!

For simplicity purposes I will relate to Australian stocks only.

a)

"An in-sample data set that is used to select the parameters for the trading system"

Click to expand...

- 20 FPO's from the Materials Sector for the years 1998 & 1999.


b)

"An out-of-sample data set that is used very frequently"

Click to expand...

- the same 20 FPO's from the Materials Sector for the years 2000 & 2001.
NB 1) The period for the two samples does not have to be the same length and 2) the choice leaves room to utilise additional in-sample & out-of-sample data sets before going live.

c) "

Search through the in-sample data set as much as you wish. Look for values of the parameters that maximize the value of the objective function; add rules and filters."

Click to expand...

- Optimise the parameters (all) that will result in the highest objective
function value - now I can see why you chose Amibroker!!
NB This would not be referred to as curve fitting unless curve fitting was incorporated into your objective function.

d)

"Beginning with the first in-sample period (the oldest in-sample period), find the optimum parameters, then test over the associated out-of-sample period, and record the results. Move on to the second in-sample period, optimize, test the second out-of-sample period, and record the results. After all the in-sample periods have been processed, concatenate the results from the out-of-sample periods. If those results are satisfactory, your confidence is increased."

Click to expand...

- If you are now confident with your model (system) then go live?

e)

"Eventually the characteristics of the market change and the model (system) is out-of-synch with the market. Maybe the market will return to its earlier state, but usually it will not, and a new model is needed, so we must reoptimize. That is, we must perform the next walk-forward iteration."

Click to expand...

- now I have a problem! If the model is not achieving the same results as
before and I have been using, as my data set, all the shares in the Materials
Sector then where do I find my "out-of-sample data set" unless I stop trading
this model for some time? Dynamic optimisation must use an in-sample data set.

That's me done for now and I look forward to any feedback.

tech/a said:

What software do you use HOWARD

Click to expand...

AmiBroker for charting, backtesting, and optimization. In my opinion, it is head and shoulders the best software available to individuals and small companies. Tomasz Janeczko has done a masterful job. Version 5.0 will be released in a few months and will probably have some very powerful extensions to aid trading system validation.

http://www.amibroker.com/

I have been using Excel spreadsheet for some Monte Carlo analysis. I anticipate less need for that when AmiBroker 5 is released.

I use Excel to format output for presentations -- particularly charts.

Thanks,
Howard

rnr said:

The comments below are my interpretation of Howard Bandy's posts to date on this thread.

That's me done for now and I look forward to any feedback.

Click to expand...

Hi rnr --

I believe that I said the out-of-sample data should be used very INfrequently.

Thanks,
Howard