# Montecarlo vs. Walk forward testing



## waza1960 (29 October 2009)

I would appeciate it if someone could explain the advantages/disadvantages/differences of the above thanking you in advance


----------



## TradeSim (1 November 2009)

I wrote something about that here:-

http://www.compuvision.com.au/Examples/MonteCarloVsOptimization.htm


----------



## howardbandy (1 November 2009)

waza1960 said:


> I would appeciate it if someone could explain the advantages/disadvantages/differences of the above thanking you in advance




Hi Waza --

Walk forward testing is an automated process of in-sample search for good logic and parameters, followed by an out-of-sample test on the best alternative.  It is the only method that will give an unbiased estimate of the future performance of a trading system.

Monte Carlo analysis is valuable in many aspects of trading system development, but is not an alternative to walk forward testing.

A search of the Aussie Stock Forum threads for my name, howardbandy, will return several posts related to this topic.

Thanks,
Howard


----------



## TradeSim (1 November 2009)

howardbandy said:


> Hi Waza --
> 
> Walk forward testing is an automated process of in-sample search for good logic and parameters, followed by an out-of-sample test on the best alternative.  It is the only method that will give an unbiased estimate of the future performance of a trading system.
> 
> ...




Hello Howard,

In my opinion your optimization and walk forward testing on a portfolio of a securities is totally flawed because it relies on ranking of trades and this always assumes  that this trade ranking always produces the optimum selection of trades. In practice this seldom occurs so therefore you cannot possibly optimize a portfolio trading system with this methodology no matter how many times you walk it forward.

Regards
David


----------



## waza1960 (1 November 2009)

Thanks for the input guys.
 I did not mean to start off an argument over which was the best testing method rather more of an explanation of the differences.
Maybe they are better suited to different scenarios, will check on the posts thanks for the tip Howard.
I have already bought Tradesim although I have not had the time yet to really explore its potential.
I also went to the ATAA conference in Melbourne and I would like to thank Howard for his presentation which I really enjoyed (also bought his two books there)looking forward to the third book and getting my teeth into Amibroker.


----------



## TradeSim (2 November 2009)

Just to clarify. We added a new parametric sweep feature to the Enterprise Edition. In this mode we step through a series of trade parameters and run a Monte Carlo analysis for each step. Then we plot the resulting histograms on a common axis for direct comparison.

For example we ran a parametric sweep on the Equis Bollinger Band trading system by stepping through the trading capital from $1,000 to $10,000 in $500 increments. The resulting twenty Net Profit histograms are plotted below on a common axis. As you can see it is virtually impossible to characterize the results of each stepping with a single metric as is done with a conventional optimization routine. A conventional optimization procedure would just run 20 back tests and plot just 20 values on a graph whereas the parametric sweep runs 20 x 5,000 or 100,000 back tests and displays the results in terms of a statistical spread overlayed on a common axis !! All of this important information disappears when using a conventional optimization procedure which attempts to over simplify something that is inherently complex.

Regards
David


----------



## theasxgorilla (2 November 2009)

TradeSim said:


> Hello Howard,
> 
> In my opinion your optimization and walk forward testing on a portfolio of a securities is totally flawed because it relies on ranking of trades and this always assumes  that this trade ranking always produces the optimum selection of trades. In practice this seldom occurs so therefore you cannot possibly optimize a portfolio trading system with this methodology no matter how many times you walk it forward.
> 
> ...




Hi David,

I have a really hard time understanding the relevance of Monte Carlo testing and really get the impression that it is not all it is made out to be.

If there are no situations where multiple trades are presented, and therefore no possibility of trade selection generating another path through the data, isn't Monte Carlo testing completely worthless?

ASX.G


----------



## TradeSim (2 November 2009)

theasxgorilla said:


> Hi David,
> 
> I have a really hard time understanding the relevance of Monte Carlo testing and really get the impression that it is not all it is made out to be.
> 
> ...




There are different types of Monte Carlo analysis. The one TradeSim deals with is the one inherent when trading a portfolio of securities. In this case there are many permutations and combinations of trades that can be taken which satisfy your entry and position size requirements. This in turn generates variance in the system so it is important to understand the way the results are distributed by repeated simulation of the system. However forcing an outcome based on a certain ranking criteria is not really understanding the nature of the system because it only deals with one of many outcomes.

Of course when you are dealing with one security this variance does not exist so a different form of Monte Carlo analysis using time series shuffling or price synthesis is used to analyse the system.

regards
David


----------



## theasxgorilla (2 November 2009)

TradeSim said:


> There are different types of Monte Carlo analysis. The one TradeSim deals with is the one inherent when trading a portfolio of securities. In this case there are many permutations and combinations of trades that can be taken which satisfy your entry and position size requirements. This in turn generates variance in the system so it is important to understand the way the results are distributed by repeated simulation of the system.




This is the part I don't get.  Let's say a given system trades end-of-day on a portfolio of securities, and when tested on a given data sample only generates at most one trade per day.  How do you get the variance?  Assuming the start-date of the in-sample testing doesn't change, and the system itself doesn't change, it seems to me there is only one path through the data.


----------



## TradeSim (2 November 2009)

theasxgorilla said:


> This is the part I don't get.  Let's say a given system trades end-of-day on a portfolio of securities, and when tested on a given data sample only generates at most one trade per day.  How do you get the variance?  Assuming the start-date of the in-sample testing doesn't change, and the system itself doesn't change, it seems to me there is only one path through the data.




It is a possible but highly improbable scenario with a portfolio of securities and long period of time 

regards
david


----------



## theasxgorilla (3 November 2009)

TradeSim said:


> It is a possible but highly improbable scenario with a portfolio of securities and long period of time




The other factor is how selective the entry criteria are.  The system I trade is very selective, so I don't think the likely variance is that large.

What I think is more important is to test system start-up and to increment the start dates of in/out-of-sample data to experience the difference between starting up in say October 2002 and October 2007... and at any other time in between.

I can run a Monte Carlo test with the same start date, and conclude that for 10,000 runs I only saw a max DD of 20%, but if I start the same system up just before a major turning point in the market, then max DD blows out to over 30%.  Ouch!

I make these comments because in the past I've seen people brandish Monte Carlo test stats like something that gives them the confidence to go trading their system, but for me, this seems insufficient.  I seem to have been able to regularly demonstrate a back-test run that trades outside of Monte Carlo stats simply by launching a system at a turning point in the market.  And since none of us can know when that is, IMO it's best to test all those worst case scenarios from the past... since that's the best we've got to work with.


----------



## TradeSim (3 November 2009)

theasxgorilla said:


> The other factor is how selective the entry criteria are.  The system I trade is very selective, so I don't think the likely variance is that large.
> 
> What I think is more important is to test system start-up and to increment the start dates of in/out-of-sample data to experience the difference between starting up in say October 2002 and October 2007... and at any other time in between.
> 
> ...




You should always include periods of major downturns in your back testing no matter which back testing methodology you use. This will stress test your system. A back test is only as good as the data you feed into it.

regards
david


----------



## Sir Burr (4 November 2009)

Interesting subject - would be a fantastic thread if two top gurus debated the subject!

Cheers.


----------

