Reoptimisations

[nizar]

Just wanted to have a discussion on a phrase which has popped up a few times.

From my understanding, re-optimisation is when, after trading the system, you decide to optimise the system again based on more recent data.

When is reoptimisation required?

Is it after you suspect the market dynamics have changed?
eg. increase in volatility?

Why is it required?

Is it an attempt to increase the life-span of the system?

I'd be interested if any of the systems-based traders here reoptimise their systems and if they do then why and if they dont then why not.

Any discussion would be much appreciated.

 

[_ExY_]

Great topic

 

[theasxgorilla]

My own un-pro thoughts on this are that if you have a longer term trend following system, and it's designed to handle certain market conditions better than others, that with re-optimisation you risk adapting the system to market conditions to which is fundamentally unsuited, only to find that when the truly suitable market conditions return you've optimised the system away from being able to take proper advantage of them.

 

[weird]

A product and forum which may relate to this topic,

http://www.biocompsystems.com/products/Dakota/

as mentioned in Howard's book.

While the forum is free, when I joined to check it out, the owner needed to ok you via email before you're allowed in.

If you did not get enough of those terms, 'future leak', and 'out of sample', from Howard's book, you may fill up more on hearing those terms used within that product's literature and forum perhaps.

 

[nizar]

Thanks guys.

Anymore responses from anyone?

Nick, tech/a, Stevo, Howard Bandy, etc, I'd be keen to hear your thoughts on this topic please.

 

[Synergy]

I think totally understanding your system is the key here, and being able to read then relate it to market conditions.

For me, I'd only want to change a system if i noticed a change in market conditions and had a reasonable idea as to how that would effect the system. The system is obviously designed with a theory or idea behind it so randomly changing that idea to suit a shorter period would be risky.

Something that might be worth looking at is running long term trends for the key variables in the system so you could constantly keep an eye on the behaviour of the market. Graphing average volitility for the systems universe or $ traded/day etc could signal variables in the system that can be changed with a fair amount of confidence. The system could then be optimized over a smaller period without the uncertainty of 'by chance' results.

 

[howardbandy]

Hi Nizar, and all --

Reoptimization.

The premise of technical analysis is that we can recognize some characteristics of market inefficiencies in the historical data that precede profitable trading opportunities; build a trading system model to automatically give buy and sell signals based on those characteristics; and that those characteristics and inefficiencies persist long enough beyond the period of time used to develop the system / model to allow us to make a profit.

The markets we are modeling are dynamic. The models we are building are static.

The systems / models are developed on in-sample data, then traded on out-of-sample data. As long as the market and the model remain in synchronization, the results will be profits.

Assuming that you have developed the model using a walk forward process, you will have observed many transitions from in-sample to out-of-sample, and you will know what to expect at the beginning of each out-of-sample period.

The period of time between the steps in the walk forward process is the period of time between reoptimizations in the system development process. This is convenient, but not required. But you May reoptimize any time you wish to.

Reoptimization is Required whenever the trading system becomes unprofitable.

Note that there is no way to estimate the period of time a newly reoptimized system will remain profitable. It does not depend on the length of time in the in-sample period, or any other easily measured characteristic of the model or the market. It depends solely on the market and model remaining in sync. Clever models can remain in better synchronization, and for longer periods, than naive models.

Eventually, every trading system gets out of sync with its market. If the two are simply out of sync, reoptimization will fix the problem. Eventually, every trading system fails; reoptimization will not fix the problem. So, you need to understand what typically happens at the beginning of each out-of-sample period and be on the lookout for the signs that the system has failed, rather than is simply out of sync.

Thanks,
Howard

 

[Tambourine Man]

It might not be necessary to reoptimize a system daily, but it will not harm the system in any way. Optimized results will not reflect real-life returns, but it is the best guess one can make.

Just be careful not to overadapt the system with to many parameters or too many tests...

/Tambourine Man

 

[Tambourine Man]

From one of my favourite books:

The optimization paradox states that parameter optimization results in a system is more likely to perform well in the future but less likely to perform as well as simulation indicates.

Thus, optimization improves the likely performance of the system while decreasing the predictive accuracy ofthe historical simulation metrics.

Many traders shy away from optimizing systems out of fear of overoptimization or curve fitting a system, but proper optimization is always desirable.

/Tambourine Man

 

[howardbandy]

Hi T.M., and all --

Who wrote the text you quoted? It does not mention whether they are talking about in-sample or out-of-sample. That distinction is The critical point when it comes to optimization.

"From one of my favourite books:

The optimization paradox states that parameter optimization results in a system is more likely to perform well in the future but less likely to perform as well as simulation indicates.

Thus, optimization improves the likely performance of the system while decreasing the predictive accuracy ofthe historical simulation metrics.

Many traders shy away from optimizing systems out of fear of overoptimization or curve fitting a system, but proper optimization is always desirable."

You can do anything you want to in the in-sample period -- try out different logic, change values for parameters, whatever. Whenever you are making a choice from two or more alternatives, you might as well look at thousands. There is nothing special about an automatic optimizer -- it just makes those thousands easier to compute than by hand.

The results achieved over the in-sample data will be good -- they are always good -- we do not stop playing around until they are good. The in-sample results have no value in terms of estimating the likely performance out-of-sample. No value.

So --- do whatever you want to in-sample, then check out-of-sample results before deciding to trade or go back to the drawing board.

On the other hand, there Is something special about an automatic optimizer working in conjunction with automatic walk forward analysis. Done properly, the walk forward analysis will give an excellent indication of how the system will perform on real trades with real money.

Thanks for listening,
Howard

 

[nizar]

Now thats what Im talking about guys!!

Thanks.

 

[julius]

Howard,

Note that there is no way to estimate the period of time a newly reoptimized system will remain profitable. It does not depend on the length of time in the in-sample period, or any other easily measured characteristic of the model or the market.

Would you be able to expand on this? What about trading frequency / timeframe ?

 

[howardbandy]

Hi Julius --

I have not found anything that indicates the length of time that a system will remain in synchronization with the market.

Systems that have long holding periods and trade infrequently are much harder to statistically validate than systems that trade frequently and have short holding periods. You'd expect this, because the longer a system is in a position, the greater the likelihood that the market will change in some way. But even intra-day systems can get out of sync with their markets.

In my experience, the only way to tell if a system is out of sync is to compare the real time performance and equity curve with the historical out-of-sample performance. The best OOS to compare with are actual trading results, but the OOS results from an automated walk forward process can also be used.

If it appears (by eye or by application of statistical tests) that the system is out of sync, stop trading it, reoptimize it, and watch the next signaled trades. But do not take them until you are confident that the system is back in sync.

Thanks,
Howard

 

[Temjin]

Excellent thread guys, and welcome back Nizar. Where have you been? Email us!

Howard, what are you views on avoiding unnecesary frequent re-optimisations to unprofitable systems? How do you know when a system become terminally unprofitable?

My personal belief is that the more simple (less degree of freedom) a system is and the more markets it has been tested with and is profitable under whatever testing criteria, the more robust the system is. And obviously, result in the reduction of the chance to re-optimise the system parameters again.

Do you propose any other solutions to this?

 

[howardbandy]

Excellent thread guys, and welcome back Nizar. Where have you been? Email us!

Howard, what are you views on avoiding unnecesary frequent re-optimisations to unprofitable systems? How do you know when a system become terminally unprofitable?

My personal belief is that the more simple (less degree of freedom) a system is and the more markets it has been tested with and is profitable under whatever testing criteria, the more robust the system is. And obviously, result in the reduction of the chance to re-optimise the system parameters again.

Do you propose any other solutions to this?

Hi Temjin --

If the system still works, and you have determined the length of the in-sample period that puts the system and the market in synchronization, then the system can be reoptimized as often as desired. Every day, if you wish. (But, be careful of how the optimizer and walk forward engine handle trades that cross the in-sample out-of-sample boundary.)

There is a mechanical answer to your question. When you performed the walk forward analysis, you selected some period of time as the length of the out-of-sample period. In the most common method of walking forward, that is also the period between reoptimizations. Continuing to use that period will set up a regular schedule of reoptimizations.

If the out-of-sample results are regularly better at the beginning of the OOS period than at the end of it, then set the length of the OOS period to be shorter and reoptimize more often.

Simple systems -- that is, systems with a few parameters -- are less likely to become curve-fit to the noise in the data. But they are also less capable of adapting to changes in the data.

Complex systems -- that is, systems with more parameters -- are more likely to become curve-fit to the noise in the data. But they are also more capable of adapting to changes in the data.

So, we have a classical trade-off -- simple system versus complex system. It goes along with the trade-off of long in-sample period versus short in-sample period. For the time series data we are dealing with, there is no easy way to define the relationship between the number of parameters and the number of data points. A lot depends on the cleverness of the algorithm and a lot depends on the variability of the signal component of the data that the system is trying to isolate. You will have to run some tests.

/////////

How can you tell when a system is terminally broken?

If the system has been working, and has been successfully reoptimized through use of an automated walk forward process, then reoptimizing should put the system back in sync with the data. If it does not, then the system is either out of sync or broken. There is no way to tell which without waiting to observe what happens on future trades.

So, I recommend the following procedure:
1. Develop the system using sound modeling techniques, including out-of-sample testing.
2. Perform walk forward analysis to get several looks at the transition from in-sample development to out-of-sample testing / trading.
3. Reoptimize whenever desired, or whenever the actual results fall below expectations (as measured by the out-of-sample results).
4. When the out-of-sample results (actual trades or paper trades) of the most recent reoptimization continue to fall below expectations, do not trade the system. Watch it, and reoptimize it as desired, but do not trade it until the OOS results return to acceptable levels.

Thanks for listening,
Howard