Walk forward optimisation of the periods

[opulence]

Has anyone come across software that you can optimise the best lengths to set your optimisation period and test period to? Or is this essentially undoing all the hard work you did you use out of sample data by curve fitting your out of sample samples?

 

[tech/a]

Re: Walk forward optimisation of the periods.

Has anyone come across software that you can optimise the best lengths to set your optimisation period and test period to? Or is this essentially undoing all the hard work you did you use out of sample data by curve fitting your out of sample samples?

Curve fitting.
You could simply re set the optimisation after each period length has expired.
You'll more than likely get a different period set each time. (nominally i would expect)

 

[opulence]

Re: Walk forward optimisation of the periods.

Yeah I thought that might be the case. So how do you go about setting the best ratio of optimisation to test period?

 

[the phantom]

Re: Walk forward optimisation of the periods.

If you back tested over a 20 year period to see a nice overall equity curve, but saw that it flat-lined or declined over a 6 - 12 - 18 months period at one time but then recovered during the overall entire period of 20 years, what would have been the optimal walk-forward test period to stop trading ?

Walk-forward testing by itself ignores a lot of system testing and trading realities.

Walk-forward testing to me falls into the usual "beginners cycle" of wanting to be right.

Instead of concentrating on walk-forward testing, the concentration should be less on optimisation and more on basic principles that work to begin with.

However these are harder to find, so easier to lead people on back testing, and over optimisation the **** of something and everything.

 

[opulence]

Re: Walk forward optimisation of the periods.

If you back tested over a 20 year period to see a nice overall equity curve, but saw that it flat-lined or declined over a 6 - 12 - 18 months period at one time but then recovered during the overall entire period of 20 years, what would have been the optimal walk-forward test period to stop trading ?

So what your saying is that walk forward testing here would have falsely made you stop trading (type 1 error) over the course of the 20 years when in actual fact if you stuck it out you would have been fine over the duration?

Walk-forward testing by itself ignores a lot of system testing and trading realities.

Walk-forward testing to me falls into the usual "beginners cycle" of wanting to be right.

Hmmm food for thought. I've read that book on universal principles of trading by Penfold which says very similar things and doesn't talk too highly of optimisation. So I'm not chasing being right but more the positive expectancy and being "the best loser" etc etc.

Instead of concentrating on walk-forward testing, the concentration should be less on optimisation and more on basic principles that work to begin with.

However these are harder to find, so easier to lead people on back testing, and over optimisation the **** of something and everything.

The trouble is, to me it seems any time you code something there will always be parameters that can be varied giving different results. Even if you're not an indicator trader, coding patterns essentially boils down to formulaic versions of the patterns with tons of parameters.

Surely nothing could work across all values for the parameters?

 

[the phantom]

Re: Walk forward optimisation of the periods.

Great questions, and input, you are yards ahead. Go with it, and develop.

I'm O/S for a bit, prayer I come home safely ... surely South America is not that bad !!!! Internal joke, one relative is extremely concerned. Anyhow I am signing off soon, but agree and enjoy smart comments from yourself.

 

[opulence]

Re: Walk forward optimisation of the periods.

Great questions, and input, you are yards ahead. Go with it, and develop.

I'm O/S for a bit, prayer I come home safely ... surely South America is not that bad !!!! Internal joke, one relative is extremely concerned. Anyhow I am signing off soon, but agree and enjoy smart comments from yourself.

Awww now you've given me a big head! Thanks Phantom.

 

[howardbandy]

Re: Walk forward optimisation of the periods.

Hi Opulence --

In my opinion walk forward testing is a critical step in validating that a trading system is likely to be profitable in the future.

I have posted extensively on this subject on Aussie Stock Forums. Search using my name.

I have published a book that covers the design, testing, and validation of trading systems that might help you. You can read about it on its website:
http://www.quantitativetradingsystems.com/

A second, more advanced, book that covers trading system performance in much more detail will be out in a couple of months:
http://www.modelingtradingsystemperformance.com/

Whatever else you do, do not begin trading a system until you have performed rigorous out-of-sample testing, preferably including walk-forward testing. In-sample results always over estimate performance and under estimate the probability of bankruptcy.

Thanks for listening,
Howard

 

[tech/a]

Re: Walk forward optimisation of the periods.

Whatever else you do, do not begin trading a system until you have performed rigorous out-of-sample testing, preferably including walk-forward testing. In-sample results always over estimate performance and under estimate the probability of bankruptcy

Howard
Has been my experience that when optimising this can be the case---however I have found that more the case of knowing what skews a system towards a positive expectancy is more important in systems design than optimising parameters to force profitability---which in turn leads --often---to breakdown in walk forward testing. Dave (The Phantom---blown his cover!) has experienced similar.

Question
Do you advise optimising all parameters in a system?
How often would you recheck and adjust optimisation as the system is traded?

 

[Gringotts Bank]

Right now, we have the perfect set of data to test systems on.

We have the low volatility bull run from 2003-7, followed by a massive crash, followed by a high volatility whipsaw fest, followed by a massive run in spec stocks, followed by a relatively dull period in the market. All these conditions packed into a recent time frame makes it the ideal time to test a system.

IMO, the only thing you need to do is come up with something that gives you a steep and smooth equity curve....everything else is fluff. All the jargon about CAR/MDD, expectancy, K ratios, Sharpe ratios etc only describe how smooth and steep your curve. Make it simple and leave the stats jargon out of it!

Also not a fan of optimization. But I have to admit that's only because I've never been able to make it work too well. My best optimized system used RSI into oversold territory on ASX200 stocks.

 

[tech/a]

Right now, we have the perfect set of data to test systems on.

We have the low volatility bull run from 2003-7, followed by a massive crash, followed by a high volatility whipsaw fest, followed by a massive run in spec stocks, followed by a relatively dull period in the market. All these conditions packed into a recent time frame makes it the ideal time to test a system.

When youve been around long enough you'll realise that perfect has a way of being stuffed up by something a little less than perfect!

IMO, the only thing you need to do is come up with something that gives you a steep and smooth equity curve....everything else is fluff. All the jargon about CAR/MDD, expectancy, K ratios, Sharpe ratios etc only describe how smooth and steep your curve. Make it simple and leave the stats jargon out of it!

A simplistic view.
Without stats you have nothing to measure with or to compare against.

Also not a fan of optimization. But I have to admit that's only because I've never been able to make it work too well. My best optimized system used RSI into oversold territory on ASX200 stocks.

An intreguing topic as it makes sense to use variables which have a relationship to best parameters rather than just a random variable.
I can see where a single data set like Futures contract would benifit but alwas wondered about a portfolio of stocks for instance.
Do you optimise each individual stock or the Portfolio it self.
You could make a singular index of your portfolio and optimise that against the portfolio itself.

Ah so many thoughts so little time.


Evolution of a trader: FA > TA > advanced TA > ANN > TA > sentiment > psychology.---BOREDOM

 

[Gringotts Bank]

I guess '% annual return' is a useful stat. That's about it for me.

That reminds me: there is one exception to just developing a 'smooth and steep' curve, and that is a system that has a high % annual return with massive 'smooth' drawdowns. Such a system can be traded effectively by drawing a low lag MA through the equity curve and using that to start/stop trading activity.

I notice the latest stocks & commodities magazine has a free low lag MA. I've plotted it against my expensive Jurik indicator and it's almost as good.

 

[tech/a]

Personally I hardly use indicators.
Certainly shorter term methods use price and volume only---no oscillators.

 

[opulence]

Futures contract would benifit but alwas wondered about a portfolio of stocks for instance.
Do you optimise each individual stock or the Portfolio it self.
You could make a singular index of your portfolio and optimise that against the portfolio itself.

Ah so many thoughts so little time.


Evolution of a trader: FA > TA > advanced TA > ANN > TA > sentiment > psychology.---BOREDOM

I guess it depends on why you have a portfolio. If it's to diversify risk then if you were to optimise for each individual stock then surely you would undo your diversification. I.e if you were long two stocks that had a perfectly negative correlation you would be fine but if you were long and short respectively because of differences in parameters then you would either make a heck of a lot of money or loose it.

 

[waza1960]

I thought I would bump this thread as I have been wondering how relevant Walk Forward testing is to good systems design.
Although I am into Systems Trading intensively I feel that WF is just adding a comfort zone to my testing i.e if I get good results using WF then I should feel more confident in the results while I'm sure this is partly true I think it is dangerous to layer WF onto optimisation (confirming my biases) .
The biggest problem I have with WF is that you have to have a reasonable time frame for adequate testing and you are basically using parameters that worked well in the past but in my experience don't give the optimal result when live trading in the present.
For example for short term trading on time frames< 4hrs I don't optimise/ backtest longer then 18mths and I am optimising up until the present period which in my view gives me the best parameter settings in tune with the market.
I feel you can get much better results by optimising and backtesting intelligently

I believe that live testing is the only true test of system profitablility and I do respect Howard Bandy's work ( I have all his books) and wanted to believe that WF is a great tool but I can't see it at the moment but willing to change my mind.

 

[howardbandy]

Hi Waza --

Isn't The question whether I, as a trader about to take a trade signaled by my system, have confidence that the trade will be good? That the real trades I take will have the same performance as the best estimate of trading performance I could make during the system design, testing, and validation?

We know that in-sample results (and all backtests and optimizations are in-sample) over-estimate profit and under-estimate risk. We do not know by how much until the validation phase of system development.

The walk forward process does several things that can be done in no other way I know of, including:

Each walk forward steps gives an example of the transition from development phase where logic rules and parameter values are established by closely examining the in-sample data to the trading phase where the rules are tested on data that was not used during development.

Gives the system an opportunity to resynchronize itself with current data conditions.

Allows the developer to create a baseline of trading performance. The out-of-sample trades are the best estimate of future performance. This "best estimate" set of data can be used to analyze profit potential and drawdown risk, and compare later real trade performance to determine system health. The best estimate set is also used to determine the position size that will maximize account growth while holding drawdown to a limit in keeping with the risk tolerance of the trader.

----

No one is required to perform validation of their trading system before trading it. But the market will be performing an out-of-sample test for you. In my opinion, walk forward validation is not a dangerous layer added to optimization. It is an essential phase of system development.

I know how difficult it is to develop systems that pass walk forward validation. If the walk forward results are not good, that is an indication that the system has been fit to the specific in-sample data and is unlikely to perform well when actually traded.

Omitting the walk forward validation may be dangerous to your wealth.

Thanks for listening,
Howard

 

[Wysiwyg]

Re: Walk forward optimisation of the periods.

Personally I hardly use indicators.
Certainly shorter term methods use price and volume only---no oscillators.

Oxymoron? Indicate = "suggest as a desirable or necessary course of action." If not random action then something in the price/volume data must influence action!

I guess it depends on why you have a portfolio. If it's to diversify risk then if you were to optimise for each individual stock then surely you would undo your diversification. I.e if you were long two stocks that had a perfectly negative correlation you would be fine but if you were long and short respectively because of differences in parameters then you would either make a heck of a lot of money or loose it.

Are there any statistics (a large sample to prove a fact) that show "diversification" has any merit in achieving (assuming above bank interest) desirable outcomes?

The biggest problem I have with WF is that you have to have a reasonable time frame for adequate testing and you are basically using parameters that worked well in the past but in my experience don't give the optimal result when live trading in the present.

The reason (isn't there one for everything we do?) is that the future price movement is unknown to most and I really mean most because "most" of us are reacting emotionally and the contagion from such is exploited every minute of every day. I don't know if anyone is immune to this.

I believe that live testing is the only true test of system profitablility

Does this not confirm that the strategy only works in the time period tested? As Howard states it is all about confidence to trade the system so inevitably there has to be a "period" when "testing" gives the user confidence and confidence varies from individual to individual (see human traits - optimism/pessimism)

We know that in-sample results (and all backtests and optimizations are in-sample) over-estimate profit and under-estimate risk. We do not know by how much until the validation phase of system development.

G'day Howard. Waza seems to be absent of recent times here and there are realities that I face which need to be aired for anyone to comment. Why do past data extrapolations over estimate profit and underestimate risk? Obviously real time participation of the system developer will alter the price action of the issue traded and availability of units at "expected" prices will alter (sometimes considerably) from the design phase. Is there anything less obvious? What are the real reasons? Anyone?

The walk forward process does several things that can be done in no other way I know of, including:

Each walk forward steps gives an example of the transition from development phase where logic rules and parameter values are established by closely examining the in-sample data to the trading phase where the rules are tested on data that was not used during development.

Honestly I do not get this. There is no way to "walk forward test" a system than in present going forward. I don't understand how testing historical data for one period and then testing an historical period in front of it is somehow different. Change the test period = change the test result either 2 periods ago or 200000 periods ago

I know how difficult it is to develop systems that pass walk forward validation.

That is because profitable edges are rare (if not a temporary illusion) and generally speaking profitability is in position sizing and post trade management. Preferably maximising every trade profit and minimising every trade loss. A mechanical system can neither see nor hear and at times is a severe lag.

Walk-forward testing by itself ignores a lot of system testing and trading realities.Walk-forward testing to me falls into the usual "beginners cycle" of wanting to be right.

As was mentioned it is a best estimate for a starting point. Even successful traders are refining their trading to trade with higher probability than the mechanical system will ever do. A fine line there.

Instead of concentrating on walk-forward testing, the concentration should be less on optimisation and more on basic principles that work to begin with.

To enter a trade then some form of optimisation must have been practiced! The entry point in the traders mind must have been the conclusion of some "perceived optimum" moment. True?

At the end of the day you have to be synchronous with the market in which you participate. It's just the future that remains (to "most" of us) uncertain.

 

[howardbandy]

Greetings --

I am not sure I can add anything new -- just try restating what I have already said with a different perspective.

In-sample results are always good. Some models are intended to memorize. Some are intended to learn. Trading systems require models that have learned features in the data that precede profitable trades. The Only way to tell whether learning has happened is to supply the model with data that has not been used in fitting process and check the accuracy of the trades that are identified.

Traditional trading system development platforms (AmiBroker, Ninja, TradeStation) produce models that are decision trees. At the beginning of the development process, a decision tree model is not well fit to the data. In-sample testing and adjusting of rules and parameters improves the fit of the model to the data and the profitability of the resulting trades. The more attention given to fitting the data to the model, adjusting rules and parameters to give good in-sample results, the closer the model comes to memorizing. We can think of the trades as signals hidden among the price and volume data. Strong signals produce profitable trades -- most of the strong signals have already been discovered and are incorporated in trading applications of large firms -- hedge funds, banks, etc. What remains are weak signals among a noisy background.

As we search for models that fit the data more closely, we improve in-sample performance then test out-of-sample. Any adjustment to the model based on out-of-sample results causes the previously out-of-sample data to become in-sample data in an expanded fitting process. That iteration contaminates the out-of-sample data. At some point, we decide we have found a good system and want to trade it. The results we achieved during development are in-sample plus contaminated out-of-sample. Both sets of results include trades that result from recognition of the genuine signal as well as trades that result from fit to the random noise in the data.

Signals appear to be unique conditions -- particularly the impulse signals we see as Buy and Sell. They form a distribution. Think mean, variance, kurtosis, skew. Use the entire distribution to the extend you can. At very least, think beyond mean.

We rely on the distribution of signals in the future data being similar to the distribution of signals in the development (in-sample and out-of-sample) data. That is the point when we discuss the need for stationarity. The future must resemble the past if we want the trades in the futures to resemble the trades in the past.

As I, and essentially all professional data scientists, regularly state -- in-sample results have no value in estimating future performance. They Always underestimate risk and overestimate profit. We do not stop developing until the fit is good and the trades are profitable. If validation is done using a single pass at previously unexamined data, the trades in the validation period are reasonable estimates of future performance to the extent that the future resembles the past. If there have been several uses of the validation data, it suffers from overfitting just as the formal in-sample data.

Trading systems are designed to recognize inefficiencies in the price series and give signals that allow traders to make a profit by buying and selling at the inefficient prices. In so doing, each profitable trade removes some of the inefficiency the system was developed to recognize and profit from. Eventually enough traders will recognize that inefficiency and remove so much of it that small traders cannot make profits greater than costs of operation -- data subscriptions, salaries, slippage, commission, taxes, etc. Eventually the profitability -- particularly the risk-normalized profitability -- of every system deteriorates and results are poorer than even the best out-of-sample results.

 

[Gringotts Bank]

Greetings --



Trading systems are designed to recognize inefficiencies in the price series and give signals that allow traders to make a profit by buying and selling at the inefficient prices. In so doing, each profitable trade removes some of the inefficiency the system was developed to recognize and profit from. Eventually enough traders will recognize that inefficiency and remove so much of it that small traders cannot make profits greater than costs of operation -- data subscriptions, salaries, slippage, commission, taxes, etc. Eventually the profitability -- particularly the risk-normalized profitability -- of every system deteriorates and results are poorer than even the best out-of-sample results.

Expert stuff, as always.

I still struggle to develop a half reasonable micro cap momentum strategy. It looks like there's so much money there in Aussie micros, but I can't seem to grab it! Could you provide a tip please?
Is it an area of the market in which fundamentals insight is more likely to yield profits?

 

[howardbandy]

Expert stuff, as always.

I still struggle to develop a half reasonable micro cap momentum strategy. It looks like there's so much money there in Aussie micros, but I can't seem to grab it! Could you provide a tip please?
Is it an area of the market in which fundamentals insight is more likely to yield profits?

My research and experience suggests that not every issue available for trading can be safely and profitably traded. I think there are several criteria that must all be met:
1. The issue itself must be volatile enough to provide profit potential, but not so volatile that intra-trade drawdowns are excessive.
2. There must be a signal component to the data that can be identified by some model. That is, some model must be able to give reliable buy and sell signals.
3. The resulting trades must have a risk profile that is acceptable when compared with the trader's personal risk tolerance.
4. The reward must be worth the risk.

I have posted a presentation on YouTube that expands these thoughts:
https://www.youtube.com/watch?v=Vw7mseQ_Tmc&feature=youtu.be

In summary -- maybe your search is difficult because the solution is not possible.

P.S. Those of you who have heard me speak or read my material already know my thoughts regarding use of fundamental information.
In summary, they are:
1. Issues of reporting schedule and revisions make the data unusable.
2. The fundamental information available outside the boardroom of the company is biased in an unknowable way and is of no value.

Best regards,
Howard