# Parameter optimization and identifying plateaus



## rx2 (17 January 2012)

Hi,

I am currently developing my first mechanical trading system with AmiBroker and in the process of testing. I am coming to realise, that I am likely to spend more effort testing the system than I have building it.

I am at the stage of wanting to do some walk forward testing on the system and currently pretty green in this area. I have been reading up on WF testing, have read Howard's book, Quant. Trading, ordered Robert Pardo's book The Evaluation and Optimization of Trading Strategies, and also ordered Howard's other book Modeling Trading System Performance + also been reading a lot of forum material.

While I have defined an objective function, one thing that struck me tonight is ideally the objective function should incorporate a value that represents the selected parameter set's stability level. 

Everywhere I have read, there is a consistent message that you should use a set of parameters that sits on a plateau rather than a single sharp peak. This makes perfect sense (I think). However, if you are doing an automated walk forward analysis in AmiBroker, how can this be done? and is what I am proposing make sense? I have 5 parameters in my system and a relatively small space ~1.2Million, so I am using the covariance matrix adaptation evolution strategy. Any thoughts on finding plateaus and stable parameters?

Cheers


----------



## howardbandy (18 January 2012)

Hi Rx --

You are experiencing the "curse of dimensionality."  I wrote a brief Posting about that on my blog, entitled "Optimization and the Curse of Dimensionality."
http://www.blueowlpress.com/WordPress/blog/

The difficulty in finding plateaus is the increase in the number of test runs that must be made to evaluate neighboring points.

Assume I have a one-parameter system that uses integer values.  From a single optimization run I find an optimum at some value, say 12.  In order to test the smoothness of the objective function surface (it is only a curve for one dimension), I might decide to use a 5 point average.  The problem changes from evaluating single results to evaluating the average of sets of 5 single results.  The best value may some point other than 12, and that is the purpose of doing this test.  In order to get the 5 point average, the results from each individual test must be stored and later averaged and sorted.  AmiBroker currently does store and sort, but not average.

As an aside, consider the objective function.  I have written extensively on the importance of designing the objective function before serious testing begins.  One of the most powerful features of AmiBroker is that it allows developers to design and code their own custom objective functions.  For example, see my book "Quantitative Trading Systems", or my blog article "Managing Subjectivity with Objective Functions."
http://www.blueowlpress.com/WordPress/articles/managing-subjectivity-with-objective-functions/ 
AmiBroker (and other platforms) computes several metrics that can be used as objective functions and reports all of them following an optimization run.  If the objective function is not well defined, the system developer will often sort the optimization results on the values of various optimization functions after the runs have been completed.  Making all of the possible objective function values available adds considerable overhead.  But ignore that for the time being.

Returning to the trading system, expand the number of variables from 1 to 2.  The search is now for a plateau -- a flat area on the objective function surface above a two dimensional parameter space.  The best spot on the surface is the best average of 25 test runs -- 5 in each dimension.  

Expanding the number of variables to 3 requires computing and sorting the best average of 125 test runs.

And so forth.  

The number of runs is the number of parameters raised to the power of the number of neighbor points.  As the number of runs increases, the amount of time needed to sort the results increases.  For large parameter spaces, sorting may take longer than the runs took.  

Note that asking for an average of, say, 5 adjacent values for each of n parameters requires exhaustive calculation of those 5 points.  Non-exhaustive techniques might be used to select the parameter values for each individual run, but it still requires calculation of n^5 to arrive at the objective function value for that single point.

There are methods, such as evolutionary operation, that search a single dimension at a time.  This reduces the number of runs considerably, but requires an iterative process to sequentially find the best in each dimension -- repeating the process several times until a stable solution is found.  If the system being evaluated is well behaved, the optimal solution will be found.  Unfortunately, financial trading systems are often not well behaved.

My recommendation is to not make the search for the best solution a quest that prevents finding one or more adequate solutions.

Keep in mind that the solution found is based on a static set of data -- the historical price and volume data.  Future data will be different.  As the logic of the system and the data it is processing move in and out of synchronization, the parameter values that will be used for trading will require adjustment.  Make the system development process easy enough and fast enough to be able to rerun the parameter selection routines as required.

Thanks for listening,
Howard


----------



## rx2 (18 January 2012)

Thanks Howard for the very informative reply. There is a lot of information there to digest. I'm at work at the moment so have limited time, but one thing I am trying to clearly is understand is your paragraph below.




howardbandy said:


> Hi Rx --
> 
> Assume I have a one-parameter system that uses integer values.  From a single optimization run I find an optimum at some value, say 12.  In order to test the smoothness of the objective function surface (it is only a curve for one dimension), I might decide to use a 5 point average.  The problem changes from evaluating single results to evaluating the average of sets of 5 single results.  The best value may some point other than 12, and that is the purpose of doing this test.  In order to get the 5 point average, the results from each individual test must be stored and later averaged and sorted.  AmiBroker currently does store and sort, but not average.




When you talk about using a 5 point average, what do you exactly mean. For example; I run an optimization run in AB for a single parameter that has a range of 0-100 with a step of 1. From the run I will sort the results by objective function (highest first). To work out the 5 point average are you suggesting taking the 5 optimized parameter values corresponding with the 5 highest ranking objective functions and averaging these? I assume you would also want to look at the standard deviation as well to understand volatility of the space?

Taking it a step further - you could then apply the same method described above but instead of doing only a single pass starting with the highest objective function, you could multiple passes each time starting with the highest objective + n? The benefit of this is that you are able to find flat surface areas.

I am not a maths wizard or statisician - but hopefully this makes some sense.

And once again - thanks for your thoughts.


----------



## rx2 (18 January 2012)

rx2 said:


> When you talk about using a 5 point average, what do you exactly mean.




Pondering a 5 point average a little more I think I know what you were describing. Were you suggesting to take a 5 point average of the objective function using the parameter space as the x-series i.e. parameter space is 0-100 in steps of 1.


----------



## Gringotts Bank (18 January 2012)

5 parameters sounds like an awful lot to optimize.  I would have thought you'd end up with something curve fitted no matter what.

Isn't there a huge drop off in performance when you view the walked forward results?


----------



## rx2 (18 January 2012)

Gringotts Bank said:


> 5 parameters sounds like an awful lot to optimize.




Is 5 an awful lot? I had thought it was a relatively good number, but happy to hear what others think is reasonable. I have no other constants in my system that relate to generating buy and sell signals.

For the purpose of illustration only, here is the basic structure of my system. P1-P5 are the parameters I am optimizing.


X1 = SOMEFUNCTION(C, P1)
X2 = SOMEFUNCTION(C, P2)

BUY = X1 > P3
BUY = BUY AND X2 > P4 
SELL = X2 < P5




Gringotts Bank said:


> Isn't there a huge drop off in performance when you view the walked forward results?



When performing walk-forward testing there is a considerable difference between the IS and OOS equity curve results in terms of total return, but the OOS equity is still smooth and consistent, and drawndown comparable.


----------



## Gringotts Bank (18 January 2012)

rx2 said:


> When performing walk-forward testing there is a considerable difference between the IS and OOS equity curve results in terms of total return, but the OOS equity is still smooth and consistent, and drawndown comparable.




^^ that's all that matters (as you know).  I could never achieve that using multiple param optimizations.  Everything fell apart if I added more than two.  Guess it comes down to wise choice of parameters.  

If you don't mind sharing, are you using more math/trig type functions, or OHLC relationships or indicators?


----------



## rx2 (18 January 2012)

Gringotts Bank said:


> If you don't mind sharing, are you using more math/trig type functions, or OHLC relationships or indicators?




Its a short term mean reversion system. There is nothing fancy / complex to it; using basics such as mean / stddev / zscore etc.... and I also incorporate some basic volume analysis as well.


----------



## howardbandy (19 January 2012)

Hi Rx --

Using the 5 value average ---

Assume I am looking for the best length of a moving average.  When I test lengths from 1 to 100, 12 is best -- but that is a single point and it might be an isolated peak.  To find the best plateau, I average (using some weighting and averaging scheme) the 100 individual results in groups of 5 adjacent values.  1-5, 2-6, 3-7, ... 96-100.     

Two examples of weighting and averaging schemes:
1.  Equally weight each result, then divide by the number of results -- simple arithmetic average.
2.  Weight the center more heavily.  Perhaps 1,2,3,2,1.  Multiple each result by the weight and divide by the sum of the weights.

Thanks,
Howard


----------



## howardbandy (19 January 2012)

Hi Rx --

When evaluating out-of-sample results from walk forward runs --

If the OOS results appear to be satisfactory -- smooth growth with low drawdown -- even though they are only a fraction as profitable as the in-sample results, proceed to the analysis phase to see whether the profit potential and drawdown risk fit your personal risk tolerance and your business plan.  If they do, do not worry about the low ratio between OOS and IS results.

No matter how good your system is, you will be resynchronizing the model to the data periodically throughout the life of the system.  You get an idea of the OOS performance relative to the IS performance by evaluating the walk forward results.  Some will have a high ratio, some low.  A low ratio may be the result of fewer trading opportunities in that particular OOS period, or a shift in the data moving the sweet spot away from the logic coded in your model.  You have no way of knowing what the characteristics of the future data series will be.  The best you can hope for is a high degree of confidence.  You get confidence by practicing.  Every walk forward step is a practice step.

Thanks,
Howard


----------

