# Monte Carlo analysis question...



## theasxgorilla (31 March 2008)

Hi Everyone,

I'm looking for a assistance in my understanding of Monte Carlo analysis.

My question, specifically is, can/does Monte Carlo testing  shuffle the chronological ordering of trades?  If you were testing on the last 10 years of data, can a Monte Carlo simulation string together trades that were from 2001, then 1999, then 2007, then 2004?

Or does the path through the data have to be chronological?

ASX.G


----------



## bingk6 (31 March 2008)

Hi Gorilla,

I cannot comment on the theoritical or acceptable definition and/or usage of Monte Carlo as I don't know enough about it. However my view is that a system that is able to maintain performance irrespective of whether the trades are executed in Chronological order or not is superior to one that depends on the trades being executed in chronological order.

This means that a system should be tested not just for a walk forward but a walk back as well and this means that the system should be able to perform during OOS periods *BOTH* subsequent to and prior to the in-sample period.

Off course, if there is a piece of software that is able to to jumble the order of trades across x different permutations, in additions to GP's random skipping of trades, in addition to the more traditional "trade selection when signals exceed position allocations" then these battery of tests should give the user more confidence about the ability of the system.

Amibroker can perform the last two easily, but the first is a challenge. Presumbly one would need to store all the signals into variables in the custom backtest procedure and then randomly issue the trades. Could run into problems with Amibroker not accepting trades that are not in time sequence. COuld be a real challenge. Anyway sorry for digressing, it is after all 3:30 in the morning!!


----------



## howardbandy (31 March 2008)

Greetings --

In my opinion, performing Monte Carlo re-ordering of trades in the in-sample period gives No information about the likelihood of the future profitability of a trading system.

Also, running a test over data that precedes the in-sample period will over-estimate the likelihood of the future profitability.  Perhaps just a little; perhaps enough to "change the sign" and suggest that the system will be profitable when it will not; and there is no way of knowing without testing on data that follows the in-sample period.

The tests that give the best estimate of the future performance are those made on data that is more recent than the in-sample data and that has not been used during the system development.  

Thanks for listening,
Howard


----------



## theasxgorilla (31 March 2008)

howardbandy said:


> In my opinion, performing Monte Carlo re-ordering of trades in the in-sample period gives No information about the likelihood of the future profitability of a trading system.




Agreed.  Does anyone know if this is how it is implemented in TradeSim?


----------



## theasxgorilla (31 March 2008)

bingk6 said:


> However my view is that a system that is able to maintain performance irrespective of whether the trades are executed in Chronological order or not is superior to one that depends on the trades being executed in chronological order.




I'm actually of the opinion that chronological order matters a lot.  Particularly when your have a universe of stocks that you are planning to trade.  It's bedtime...I'll explain more later.


----------



## nizar (31 March 2008)

theasxgorilla said:


> Agreed.  Does anyone know if this is how it is implemented in TradeSim?




Monte Carlo analysis, as per TradeSims definition, is to identify different PATHS of trades.

Chronological order is not modified. You can see this from the Trade Log.

TradeSim only takes the trade if you would take the trade in real-life. Its a trading simulator.

Thats why monte carlo analysis in TradeSim is such an important tool for systems design in terms of analysing systems that by their design, cannot take all the trades that trigger.

Because if you just run a single simulation, you have no idea if the PATH that the software took was the very best, very worst, or just the average. Because for such systems there are MULTIPLE PATHS that you could have took, and each of these would have resulted in a different outcome.

Monte Carlo analysis will give you the performance (percent winners, number of trades, profit, drawdown) of the best portfolio (PATH), the worst, the average, and standard deviations from thousands of portfolios.


----------



## nizar (31 March 2008)

In response to the original post of this thread.

Hi Everyone,

I'm looking for a assistance in my understanding of Monte Carlo analysis.

My question, specifically is, can/does Monte Carlo testing  shuffle the chronological ordering of trades?  If you were testing on the last 10 years of data, can a Monte Carlo simulation string together trades that were from 2001, then 1999, then 2007, then 2004?

No. Is there a purpose of this sort of simulation?
It would be utterly useless.

Or does the path through the data have to be chronological?

Yes. The path must be chronological. Monte carlo analysis simulates different paths of trades.


----------



## howardbandy (1 April 2008)

Greetings all --

Monte Carlo simulation does have a place in trading system development.  The technique of Monte Carlo is one of modifying some aspect of an experiment, model, simulation, etc, by adding a random component.  The results of the modified runs are analyzed with the hope of leaning something about the original runs.

There are many ways that Monte Carlo can be used to advantage in trading systems.  Here are three examples:

One way is to add random noise to the input data -- the open, high, low, close, volume, open interest, and so forth.  When the trading system is re-run on the now noisier data, it will give an idea of how sensitive the system is to small changes in the data.

Another way is to add random noise to the values of the parameters -- the lookback lengths, buy and sell levels, profit targets, and so forth.  This can give information about how sensitive the system is to its parameters -- that is, how robust the system is with respect to its parameters.

Another way is to reorder closed trades.  This can be done either by using exactly the trades that were produced by the trading system, or by creating trades from a statistical distribution that has characteristics similar to those from the trading system.  The reordered trades allow the system developer to see what might have happened if the winning and losing trades happened in a different order.  For example, if the losing trades happened early in the run, would the trading account have been drawn down too far to allow continued trading.

My cautions include the following:

In-sample results have no value in determining the likely future performance, so reordering in-sample trades is a waste of time and cannot give any additional information.

It is possible that reordering out-of-sample trades will give some information about characteristics of the equity curve, but not always.  The trading system may have logic built into it that tests the equity curve before placing  buy and sell orders.  Reordering those trades would not be a legitimate use of Monte Carlo analysis.   

In my opinion, it is more important to perform rigorous out-of-sample testing than it is to perform Monte Carlo analysis, although properly done Monte Carlo analysis can improve the confidence that the developer has in his or her system.

When Monte Carlo analysis is performed, the developer must carefully think through the process and the interpretation of the results.

Thanks for listening,
Howard


----------



## tech/a (1 April 2008)

> In my opinion, it is more important to perform rigorous out-of-sample testing than it is to perform Monte Carlo analysis, although properly done Monte Carlo analysis can improve the confidence that the developer has in his or her system.




Howard Ive seen you mention this many times.

I keep asking myself.

"Once Ive done an out of sample test it becomes "in sample"
There is no guarentee that future out of sample tests are going to perform the same,good or bad."

Really then all testing can offer is an indication that any set of conditions and parameters can perform similarly given a similar environment.

In sample
Out of Sample
Optimised
Montecarlo Analysis
Various markets
Maximised testing periods
Massive data inputs
Various environments

Doesnt matter.
Change the environment and youll change the results.
You'll never have a definative answer to 
"Is my system likley to be profitable after testing indicates that it could be."

Not to say testing is a pointless exercise.


----------



## theasxgorilla (1 April 2008)

nizar said:


> Yes. The path must be chronological. Monte carlo analysis simulates different paths of trades.




Okay, cool.

Any idea how Tradesim does this?  Does it ignore trades at random, or randomise delaying the start date of the testing period?

ASX.G


----------



## theasxgorilla (1 April 2008)

howardbandy said:


> *Another way is to reorder closed trades.*  This can be done either by using exactly the trades that were produced by the trading system, or by creating trades from a statistical distribution that has characteristics similar to those from the trading system.  The reordered trades allow the system developer to see what might have happened if the winning and losing trades happened in a different order.  For example, if the losing trades happened early in the run, would the trading account have been drawn down too far to allow continued trading.




Is this what I am talking about?  Put another way, mucking with the chronological ordering of the (closed) trades taken by the system.


----------



## howardbandy (1 April 2008)

Hi Tech, and all --

You are correct -- there are no guarantees.  The best we can do is follow good modeling and simulation practices, and develop as much confidence as possible in both our trading system and the development process.  

It is particularly important to understand and be confident about the transition from development to trading.  In my opinion, the best -- and as far as I know, the only -- way to get that understanding is to practice.  That is, to observe a lot of transitions from in-sample development to out-of-sample testing.  An easy way to do this, and to do it using good techniques, is an automated walk forward process. 

Some practices are better than others.  Of course, any one can develop his or her trading systems any way he or she wishes.  

What I try to raise awareness of and caution against are those practices that are known to over-estimate (often very significantly) the likelihood of acceptable performance in the future -- particularly when the trades are made with real money.  Some of the techniques described in the literature, posted on discussion boards, and practiced by some developers are very dangerous to their wealth, or to the wealth of the clients they hope to attract.  

Full apologies in advance to anyone on the ASF board who takes offense -- none is intended.  And I do not imply that anyone here is knowingly deceiving anyone.  The biggest risk for deception is probably that of the trading system developer to him or herself.

Thanks for listening,
Howard


----------



## tech/a (1 April 2008)

> What I try to raise awareness of and caution against are those practices that are known to over-estimate (often very significantly)




Howard while I have your book and if these practices are listed there I will look them up.
If not and if scattered throughout your work,is it possible to perhaps list some of these common mistakes (practices) that as developers we are likely to mislead ourselves with?

Single tests of a set of conditions comes to mind.
Incorrect use of optimisation another.
Insufficient data
Perhaps some which are not so obvious.

Thanks.


----------



## nizar (1 April 2008)

theasxgorilla said:


> Okay, cool.
> 
> Any idea how Tradesim does this?  Does it ignore trades at random, or randomise delaying the start date of the testing period?
> 
> ASX.G




No, it does not ignore trades at random.

There is another function within TradeSim to ignore a percentage of trades.

As to Monte Carlo analysis:

There are certain points during the course of trading where a decision MUST be made because there is insufficient available capital to take all the trades.

For example. Week 1.
12 signals.
You can afford 8.

Which ones?

Monte carlo analysis simulation #1. Will pick the first 8 trades.
Monte carlo analysis simulation #2. Picks trades 1,2,3,7,8,9,11,12.

A few weeks down the track.

You have enough capital for 3 stocks.
Five signals pop up.

Monte carlo analysis as per TradeSim's definition, will take every possible PATH of trades that had been possible if you were trading the system in real life.

Techs way of putting it is very good.

Monte Carlo analysis is like giving your system and the same available starting capital to 20,000 people, telling them to come back in 10 years time, and then collating the results. You then have the mean, and also the positive and negative outliers, and the standard deviations for profit, max.DD, number of trades, percent winners, etc.

With this information, you will have a good idea of how the system can be expected to perform, and more importantly, the range of distribution of each parameter.


----------



## nizar (1 April 2008)

Also, just to add.

I dont think monte carlo analysis and walk forward analysis is not an area where you pick one or the other.

I did both.

The initial 6 year testing period (in which the system was optimised) was 1998-2003. Then I did monte carlo runs in 2 year blocks on the out-of sample data.


----------



## theasxgorilla (2 April 2008)

nizar said:


> Monte Carlo analysis is like giving your system and the same available starting capital to 20,000 people, telling them to come back in 10 years time, and then collating the results. You then have the mean, and also the positive and negative outliers, and the standard deviations for profit, max.DD, number of trades, percent winners, etc.




Okay, so when presented with multiple trades and insufficient capital to take them all, TradeSim will use some random function to decide which trades to take.  Am I getting it?

So the value of this kind of Monte Carlo analysis must be dependent upon enough of these junctures existing, in order to get the multiple paths you talk about.

Will TradeSim report if it repeats a path, or will it stop testing if it's exhausted all possible paths?  How many paths are there using a garden variety LTTF system and 2 years worth of data on the ASX300?

Probably a bit difficult to answer that last question decisively.  I'm just exploring the question.  I keep seeing this 20,000 figure thrown up, but wonder if there are 20,000 paths through such a data set...and if there is, how much they can reasonably be expected to vary by.

ASX.G


----------



## howardbandy (2 April 2008)

Greetings all --

I understand the discussion has been about how TradeSim handles a situation of having more signals than available funds, and how Monte Carlo techniques can be used to simulate the results.

I do not want to sidetrack your thread, but to point out that the AmiBroker trading system platform handles that situation in a deterministic manner and avoids the need for Monte Carlo techniques in this circumstance.

AmiBroker gives the system developer the option of deciding which trades to take within the trading system code.  A system variable called PositionScore is used to break ties.  

For every bar and for every ticker symbol, compute not only the signals and indicators that will be used to enter a new trade, but also compute PositionScore.  If there are more signals at a given time than there are "slots", take positions in those that have the highest PositionScore.  

Good choices for computing PositionScore are the same that would be made if the trader could evaluate all of the potential entries at the time of the trade.  Some might want to buy the issues that have been strongest recently, others may want to buy those that have been weakest, others may want to buy those that belong to industries that are not already owned, and so forth.

Thanks for listening,
Howard


----------



## Shane Baker (2 April 2008)

Hi ASX

TradeSim takes the trades alphabetically in the normal backtest unless instructed otherwise. You can rank the trades similar to, but not as flexibly as Amibroker by using the  ExtFml( "TradeSim.SetVariableTradeRank",); function.Personally I like to see the LTTF system with and without this function used in the MC stats as I like to make sure that the very basic premise of the LTTF system holds up to scrutiny. 

I am not sure it is important to know whether any paths have been repeated as I am mainly interested in MC testing to look at the extremes of results rather than the mean or median results. I might suggest that two years for a LTTF data set is too short and a much longer period will produce more meaningful results (including assessing performance over various market condictions).

Cheers

Shane


----------



## nizar (2 April 2008)

Hi ASX.G

Okay, so when presented with multiple trades and insufficient capital to take them all, TradeSim will use some random function to decide which trades to take.  Am I getting it?

Yes!!

So the value of this kind of Monte Carlo analysis must be dependent upon enough of these junctures existing, in order to get the multiple paths you talk about.

On the ball, son!

Will TradeSim report if it repeats a path, or will it stop testing if it's exhausted all possible paths?  How many paths are there using a garden variety LTTF system and 2 years worth of data on the ASX300?

It will test as many portfolios as you instruct it to.
I normally choose between 5,000-20,000.
For 2 years, I would use 5,000.
For 6 years of more, I did 20,000.
Even though I could select any number I want, More than 20,000 it just takes too long.

Probably a bit difficult to answer that last question decisively.  I'm just exploring the question.  I keep seeing this 20,000 figure thrown up, but wonder if there are 20,000 paths through such a data set...and if there is, how much they can reasonably be expected to vary by.

What I look for in my monte carlo results is a tight distribution of results.
Ideally, there shouldnt be much different between the max. and min. profits and drawdown (low standard deviation from the mean).

I don't know about whether 20,000 is excessive or not. BUT during the first couple of months of trading my LTTF, I was getting between 22-28 signals thrown every week, even though I could by no more than 10. And I bought them all (maybe foolishly) in the first week. So say in the first 8 weeks you were getting between 22 and 28 signals and you could buy only 10 from all of them (obviously if you got stopped out you could buy again). But from such a small sample of 2 months there were probably hundreds of possible paths.

Also, when testing on only 2 years of data, you must remember to close out all open trades at the end of the testing period, because this is possibly where most of the profit will be.


----------



## MRC & Co (2 April 2008)

tech/a said:


> "Once Ive done an out of sample test it becomes "in sample"
> There is no guarentee that future out of sample tests are going to perform the same,good or bad."




"In sample" means historical data yeh?  Whereas "out of sample" means forward testing?  

Just trying to clear up a few of these definitions.


----------



## Chorlton (2 April 2008)

MRC & Co said:


> "In sample" means historical data yeh?  Whereas "out of sample" means forward testing?
> 
> Just trying to clear up a few of these definitions.




Be aware that "Out-of-Sample" can also be historical data as long as the System under development has NOT yet been exposed to it.

As soon as this does happen, it will then become part of the "in-Sample"


----------



## tech/a (2 April 2008)

MRC
Yes and yes to Chorlton.


----------



## MRC & Co (2 April 2008)

Ah k, yes logical, thanks guys.


----------



## barney (2 April 2008)

Hi guys, I'm probably out of my depth here cause I don't have any testing software yet, but am interested for future reference .............

Question re testing a system etc.  ............. Has anyone done any "testing" of their various systems over time periods when the "Patterns" of the Index or Stock you are testing is (or as close as you can get to it) is similar to the current "pattern" .............. ie. Does the testing prove more successful under these conditions when the "ducks" are lined up in a similar fashion so to speak ............. Just a thought.   It would seem to me that testing a system in a similar "environment" would carry a lot more weight if it showed positive expectancy, rather than just a random time frame ............. any thoughts?

EDIT:  Just to clarify that ............ It would mean for example, you may test your data from say the year 2000-2001 or any time frame you thought applicable, because the market patterns were behaving in a similar pattern to the current time of testing .... hope that makes sense.


----------



## theasxgorilla (2 April 2008)

howardbandy said:


> I do not want to sidetrack your thread, but to point out that the AmiBroker trading system platform handles that situation in a deterministic manner and avoids the need for Monte Carlo techniques in this circumstance.




Not sidetracking at all.  I simply want to know what something is and how it works 

Regarding the position score attribute in Amibroker...here is a wild idea...would it not be possible to randomise this through various iterations of a backtest and essentially simulate the Monte Carlo testing that TradeSim does...using optimisation of a dummy variable to generate each simulated portfolio?

ASX.G


----------



## GreatPig (2 April 2008)

theasxgorilla said:


> Regarding the position score attribute in Amibroker...here is a wild idea...would it not be possible to randomise this through various iterations of a backtest and essentially simulate the Monte Carlo testing that TradeSim does...using optimisation of a dummy variable to generate each simulated portfolio?



From what I've seen, that wild idea has been used many times by many people for quite some time.

However, IMHO, it may not provide a great deal of randomisation if there are not a large number of buy signals. PositionScore only determines the entry when there are multiple possibilities at a single bar, and if a trading system doesn't generate a lot of signals, having multiple at a single bar may be fairly rare. In that case, the variation in portfolios between backtests may be small, or due to the nature of the system, certain trades may always be taken, skewing the results if those trades generate large profits or losses.

GP


----------



## theasxgorilla (3 April 2008)

GreatPig said:


> However, IMHO, it may not provide a great deal of randomisation if there are not a large number of buy signals. PositionScore only determines the entry when there are multiple possibilities at a single bar, and if a trading system doesn't generate a lot of signals, having multiple at a single bar may be fairly rare. In that case, the variation in portfolios between backtests may be small, or due to the nature of the system, certain trades may always be taken, skewing the results if those trades generate large profits or losses.




I totally agree GP.  And my system is very selective in it's entry setups so there aren't a great deal of circumstances where it is going to have multiple paths to take...and so my thinking is the variation is going to be very low and therefore I don't think I can imply very much from Monte Carlo testing in this fashion.

Or does TradeSim do some other magic when it performs a Monte Carlo simulation?


----------



## theasxgorilla (3 April 2008)

nizar said:


> My question, specifically is, can/does Monte Carlo testing  shuffle the chronological ordering of trades?  If you were testing on the last 10 years of data, can a Monte Carlo simulation string together trades that were from 2001, then 1999, then 2007, then 2004?
> 
> No. Is there a purpose of this sort of simulation?
> It would be utterly useless.





I must admit, I still don't get it.

I thought that Monte Carlo testing or simulating was terminology used to describe the process of introducing some random factor into a function, such that you could produce different outcomes on each run-through.

I thought that Monte Carlo could be done in a variety of ways.  For example, ignoring a proportion of trades at random would result in varying backtest paths through data...this would be a way of running a Monte Carlo simulation.

Gathering every single possible trade from a data set and stringing them together in a random fashion to generate simulated equity curves could be another.  You might chose to obey or disregard chronological ordering.  If you disregard it you might chose to breakup trades into individual entities OR keep them together in chunks.  I believe the latter is intended to try and preserve the effects of the underlying market conditions...shock events, for example.

The purpose of this kind of testing might be, as Howard mentions, to see how an equity curve and associated measures like max DD and consective winners/losers might have looked if the sequencing of trades was different.  But with the consequence of breaking the relationship that the sequencing of trades had to market conditions.


----------



## elbee (3 April 2008)

Monte Carlo simulation is used in modelling when there is one or more variables in the model that cannot be easily quantified. Monte Carlo simulation assigns random variables to those unquantified variables so that the model can be run. Obviously this means a large number of runs with with different random numbers has to be carried out to get an indication of the possible range of results.

If all the variables in a model are quantifiable then there is no need for Monte Carlo simulation.

As I understand it (I have not used the software), the Monte Carlo function in TradeSim is to overcome a deficiency in MetaStock where it is not easy to specify which trades to take when there are multiple opportunities.

As Howard has pointed out this is overcome in Amibroker with the positionscore function.

Thus it is possible to precisely define a mechanical system or model in Amibroker and avoid the need for Monte Carlo simulation.

Assigning random numbers to the positionscore variable would appear to be using this function to do what it was designed to avoid.


----------



## stevo (3 April 2008)

Positionscore is not some magic bullet that makes Monte Carlo simulations a waste of time. It's another trading parameter that enables trades to be ranked in order of preference, based on some condition. The ranking could be as simple as using price to rank the order of trades, or even the alphabet. 

If you find that ranking based on the alphabet gives better results than taking random trades then you would pick stocks starting with "A" over those starting with "B"! I am too sure that this is approach is valid - most people would argue that the name of a stock should have no bearing on it's performance.

Comparing a ranking strategy against taking random signals could give some insight into the performance of the ranking strategy. Where does the ranking strategy sit in terms of the distribution of possible outcomes without the rankinig strategy? How sensitive is it to minor changes? What happens if you hardcode into the buy signal that stocks from M to Z are ignored etc

Ranking gives one path out of the thousands generated in a Monte Carlo. Potentially a bit like getting an infinite number of monkeys, giving them all typewriters and getting one to write poetry.

I am currently reading Howard's book and I am really getting a lot out of it -  just the first 3 chapters are making me think hard about the process of developing trading systems.

stevo


----------



## nizar (3 April 2008)

stevo said:


> Comparing a ranking strategy against taking random signals could give some insight into the performance of the ranking strategy. *Where does the ranking strategy sit in terms of the distribution of possible outcomes without the rankinig strategy? *How sensitive is it to minor changes? What happens if you hardcode into the buy signal that stocks from M to Z are ignored etc
> 
> Ranking gives one path out of the thousands generated in a Monte Carlo. Potentially a bit like getting an infinite number of monkeys, giving them all typewriters and getting one to write poetry.




ASX.G.
Stevo has explained monte carlo analysis and its purpose quite succintly here.



			
				elbee said:
			
		

> As I understand it (I have not used the software), the Monte Carlo function in TradeSim is to overcome a deficiency in MetaStock where it is not easy to specify which trades to take when there are multiple opportunities.
> 
> As Howard has pointed out this is overcome in Amibroker with the positionscore function.




In TradeSim, it is in fact very easy to specify which trades to take when there are multiple opportunities. This function is completely separate and has nothing to do with Monte Carlo analysis.

You can rank according to stock price, rate of change, volatility, etc.

Monte Carlo analysis is a simulation designed to see the possible range of results, and the "tightness" of their distribution, and like Stevo said, its only through a monte carlo where your ranking fits in relation to the mean (just picking signals randomly).

If you do not perform a monte carlo analysis, then its not possible to see what benefit or adverse effect your ranking has on the results.


----------



## nizar (3 April 2008)

GreatPig said:


> However, IMHO, it may not provide a great deal of randomisation if there are not a large number of buy signals. PositionScore only determines the entry when there are multiple possibilities at a single bar, and if a trading system doesn't generate a lot of signals, having multiple at a single bar may be fairly rare. In that case, the variation in portfolios between backtests may be small, or due to the nature of the system, certain trades may always be taken, skewing the results if those trades generate large profits or losses.
> 
> GP




I agree wholly.


----------



## nizar (3 April 2008)

Hi ASX.G

I thought that Monte Carlo testing or simulating was terminology used to describe the process of introducing some random factor into a function, such that you could produce different outcomes on each run-through.

Correct.

I thought that Monte Carlo could be done in a variety of ways.  For example, ignoring a proportion of trades at random would result in varying backtest paths through data...this would be a way of running a Monte Carlo simulation.

Correct.

Gathering every single possible trade from a data set and *stringing them together in a random fashion* to generate simulated equity curves could be another. 

You might chose to obey or disregard chronological ordering.  

The whole purpose of systems testing is to see how a system would perform in real-life. As such we try to make the backtests as realistic as possible so we can deduct meaningful conclusions from the results.

What purpose would disregarding chronological ordering serve?

If you disregard it you might chose to breakup trades into individual entities OR keep them together in chunks.  I believe the latter is intended to try and preserve the effects of the underlying market conditions...shock events, for example.

What do you mean by breaking up trades or keeping them in chunks?
Can we trade this way in real life?


----------



## theasxgorilla (3 April 2008)

nizar said:


> Stevo has explained monte carlo analysis and its purpose quite succintly here.




Yeah i know...I still can't stop thinking to myself, "that's all it is???".

As I've said, for a system that doesn't have that many situations where the ranking mechanism (position score) kicks in, I think that randomly ignoring a proportion of trades would be a more effective way of backtesting alternate paths.


----------



## tech/a (3 April 2008)

theasxgorilla said:


> Yeah i know...I still can't stop thinking to myself, "that's all it is???".
> 
> As I've said, for a system that doesn't have that many situations where the ranking mechanism (position score) kicks in, I think that randomly ignoring a proportion of trades would be a more effective way of backtesting alternate paths.





ASX

Really that's what it does.
Taking out a chunk in the middle 20% in or 80% in would basically be similar to starting the system without those chunks ever being included.
In real life they are there.

You could simply generate your system to start 6 mths later or a year earlier/later and finish 6 mths earlier or 6 mths later.That would give you 5 runs of the system.
You could then Montecarlo each of these then check and compare the results of each.---endless really.

But as I have pointed out you can test your brains out but if market conditions alter significantly beyond the parameters in which you have tested then there is a very good chance your system will be affected.
BOTH ways---positive affect or negative.

All we are doing is giving ourselves more confidence in our trading method and a blueprint with which to monitor performance.
Not to mention a structured plan with clear cut rules that if applied can have an expected positive expectancy---given the above caveat


----------



## stevo (3 April 2008)

theasxgorilla said:


> Yeah i know...I still can't stop thinking to myself, "that's all it is???".
> 
> As I've said, for a system that doesn't have that many situations where the ranking mechanism (position score) kicks in, I think that randomly ignoring a proportion of trades would be a more effective way of backtesting alternate paths.




Something that I have tried. You would have to watch out that a buy signal really is ignored and not just bought in the following periods. For example all the buy parameters are valid but a flip of the coin knocks out the trade today. Then  heads come up next month and the buy signal triggers. 

Also exactly what are you testing? A different set of trades with each run or a trader that does not follow a system precisely? Do you knock out 1/2 the trades or only 10%? You could even "optimise" the number of trades to ignore!

As tech mentions there are no guarantees in this business.

stevo


----------



## weird (3 April 2008)

Unfortunately I find it hard to explain Monte Carlo from a Metastock/Tradesim perspective, to those that have never used the tool/combination (Metastock could be replaced with another package if that package is able to produce the required output, such as Amibroker). 

I have seen some posts on competitive product mailing lists, to Metastock, where Tradesim sadly gets dismissed as a tool for making up for Metastock's system testing short comings. Its usefulness as a portfolio backtester, is just the tip of the iceberg.

For developing "portfolio based" stock trading systems, where there could be multiple buys signals, and there is some need to exclude trades due to position sizing constraints (that is Monte Carlo), this is a fantastic analysis tool.

By brute forcing the multiple possible combinations, it can show the range of potential outcomes, by summarizing the range of different possible paths constrained within the money management rules.

Another cool feature of Tradesim, is the easy ability to deselect any trade from the result, to remove outliners.

Positionscore while cool, does not easily allow one to remove the outliner effect.


----------



## Shane Baker (4 April 2008)

weird said:


> Unfortunately I find it hard to explain Monte Carlo from a Metastock/Tradesim perspective, to those that have never used the tool/combination (Metastock could be replaced with another package if that package is able to produce the required output, such as Amibroker).
> 
> I have seen some posts on competitive product mailing lists, to Metastock, where Tradesim sadly gets dismissed as a tool for making up for Metastock's system testing short comings. Its usefulness as a portfolio backtester, is just the tip of the iceberg.
> 
> ...




I agree. Each peice of software has some great features. The problem is that there isn't any one commercially available tool that allows traders to develop and stress test their systems fully (except for the market of course ) Tradesim has some great features, Amibroker has some great features, Mechanica, Wealth Lab has some great features but they all have some deficiencies in their design which is probably a result of their developer's bias and longevity in the market (what was good design and user interface 10 years ago may not be now....are you listening Equis?).

Ultimately a developer may use multiple pieces of software and data to develop and stress test their systems. Eventually it all comes down to the final walk forward testing with real money.

Cheers

Shane


----------



## stevo (4 April 2008)

Shane
The main limit I have found in Amibroker is my own programming skills!

You can also Positionscore in TradeSim by using ranking criteria column. But when a Monte Carlo is run the ranking is turned off so that trade decisions are randomised rather than ranked. If there are no "leftover" trades then Monte Carlo is going to tell you much, although it could give an insight to slippage if you randomise the entry and exit prices. 

I am very comfortable with TradeSim and Amibroker combo, even though some will tell me that Amibroker can do all that TradeSim can do. You can put any list of trades, in the right format, into TradeSim generated from any program (or even real trades) and do portfolio and position size testing.

It's the jump from testing to walking forward with real money that many find quite hard. A system test of 10 years worth of trades takes 1-2 minutes, but real life is much slower - I guess that it is "real". 

You don't suffer the drawdowns in a system test. Mybe we need an electric shock add-in to simulate the pain of losses and give more direct feedback on a system's performance. Then slow down the test during the painful periods and speed it up when it's making money!

stevo


----------



## weird (4 April 2008)

[O/T from MC] Just some thoughts on forward walk, that is on testing on 'unseen data' ...

From reading about forward walk, those testing systems, may simply take this to mean to test on a defined period, as example 2000-2006, and then once you have completed this then test on unseen data, example 2006-2008. However others may also read this as 'optimize the parameters' as much as one can, then test how well those optimized parameters hold over another period. The emphasis here is on 'optimize'. While we all 'optimize' to a degree, some take it much further than others.

Some system designers and traders, will actually take the concept of 'forward walk' even another step further,  and continue to walk forward test and continually adapt parameters through optimization as they trade (that is using the results of this type of testing as a form of system feedback),attempting to adapt to the changing market. 

Some of these type of systems, may be very basic, such as a moving average cross-over. This approach to system design of optimization/walk-forward feedback, re-optimization, in my humble opinion is too focused on re-optimization of a system (which potentially could be beating a very dead horse), and not really determining why something may or not work in its raw form, now or in the future.


----------



## nizar (4 April 2008)

weird said:


> [O/T from MC] Just some thoughts on forward walk, that is on testing on 'unseen data' ...
> 
> From reading about forward walk, those testing systems, may simply take this to mean to test on a defined period, as example 2000-2006, and then once you have completed this then test on unseen data, example 2006-2008. However others may also read this as 'optimize the parameters' as much as one can, then test how well those optimized parameters hold over another period. The emphasis here is on 'optimize'. While we all 'optimize' to a degree, some take it much further than others.
> 
> ...




Tend to agree with this.


----------



## Shane Baker (6 April 2008)

stevo said:


> Shane
> 
> 
> It's the jump from testing to walking forward with real money that many find quite hard. A system test of 10 years worth of trades takes 1-2 minutes, but real life is much slower - I guess that it is "real".
> ...




That would reduce the number of people designing and testing systems for sure


----------



## tech/a (6 April 2008)

> It's the jump from testing to walking forward with real money that many find quite hard.




The larger majority dispense with the testing bit and just step up to the plate with idea 101 and get straight into forward testing---urr trading.


----------



## Trembling Hand (6 April 2008)

tech/a said:


> The larger majority dispense with the testing bit and just step up to the plate with idea 101 and get straight into forward testing---urr trading.




I will put my hand up and say I have NEVER back tested.


----------



## nizar (6 April 2008)

Trembling Hand said:


> I will put my hand up and say I have NEVER back tested.




Any specific reason for not backtesting?

I guess with the average length of your trades being a few seconds, it wouldn't take long to just forward trade the method. A week would probably be several hundred trades and enough to tell you the essential statistics of the system.

The only problem with that is that you are initially putting yourself at risk of trading a system without knowing beforehand what it will perform like.


----------



## Trembling Hand (6 April 2008)

nizar said:


> Any specific reason for not backtesting?




Like most punters I just started trading. 



nizar said:


> The only problem with that is that you are initially putting yourself at risk of trading a system without knowing beforehand what it will perform like.




Yes my testing was very expensive.


----------



## tech/a (6 April 2008)

Like TH I did the very same thing.
Lost a motzza in the first 6 mths.

Travelling Nizars path I think fast tracked the learning not only of whats useless and whats not and what are the key ingredients of a successful method but how to construct one.

It also lead me to the path of being able to trade in a discretionary manner profitably once these keys were known.
Mind you I'm far more confident in trading larger $ value with tested methods than discretionary "Ideas".

Personally I have found Consistency comes from a structured format which has the key elements and a blueprint from which to work.

My discretionary ideas give rise to my systems in the end.
True there are some discretionary and successful methods which are for me at least to complex to turn into a trading system.

These I don't discard----- however its these which have the least/poorest consistency.

Like most here they will have found that you can be right more often over shorter periods,the difficulty is staying right long enough to collect a good profit often enough.

As Radge says its
"Being wrong isn't the problem its how long we STAY WRONG that is."


----------



## theasxgorilla (6 April 2008)

Shane Baker said:


> I am not sure it is important to know whether any paths have been repeated as I am mainly interested in MC testing to look at the extremes of results rather than the mean or median results. I might suggest that two years for a LTTF data set is too short and a much longer period will produce more meaningful results (including assessing performance over various market condictions).




Thanks Shane.  I agree, 2 years is not sufficient IMO either.

I am interested in the distrubution of results with a random function introduced into a backtest.  I like to see the distribution bunch and shift in a favourable direction with refinement of parameters.


----------



## theasxgorilla (6 April 2008)

elbee said:


> Assigning random numbers to the positionscore variable would appear to be using this function to do what it was designed to avoid.




Perhaps, but this is why I like Amibroker.  When you code at the low-level of the customer backtester it's so flexible that with some lateral thinking you can implement all kinds of things that the designer might never have thought of when he created the framework.


----------



## howardbandy (7 April 2008)

theasxgorilla said:


> Not sidetracking at all.  I simply want to know what something is and how it works
> 
> Regarding the position score attribute in Amibroker...here is a wild idea...would it not be possible to randomise this through various iterations of a backtest and essentially simulate the Monte Carlo testing that TradeSim does...using optimisation of a dummy variable to generate each simulated portfolio?
> 
> ASX.G




Hi Gorilla --

Yes, you can assign a random number generated within AmiBroker to PositionScore.  

PositionScore = Random;  // uses a random seed

or 

PositionScore = Random (13331); // starts a sequence that is repeatable


Or -- ignoring PositionScore tells AmiBroker to use alphabetical order.

Thanks,
Howard


----------



## howardbandy (7 April 2008)

weird said:


> [O/T from MC] Just some thoughts on forward walk, that is on testing on 'unseen data' ...
> 
> 
> 
> Some system designers and traders, will actually take the concept of 'forward walk' even another step further,  and continue to walk forward test and continually adapt parameters through optimization as they trade (that is using the results of this type of testing as a form of system feedback),attempting to adapt to the changing market.




Hi Weird --

You can do that.

But making decisions to change the trading system based on the out-of-sample results changes those previously out-of-sample results into in-sample results.  If you do not have another set of never-used data to make a final out-of-sample test, then the system is just as curve-fit / over-fit as any other optimized-but-never-tested system.

You may get lucky and have profitable trades from it.  But the way you will learn is by using real money.  Why not set aside some data for a no-cost out-of-sample test?  If that is profitable, go ahead and trade the system -- if that is not profitable, you have saved yourself from trading losses. 

If you are thinking about designing adaptive systems, build the adaption into the logic so the program senses the changes itself without the need for parameter modification after the fact.  You can test whether this works by running out-of-sample tests.  

Look at the work of John Ehlers for examples of adaptive logic.  He has published some algorithms for detecting the length of the dominant cycle on each and every bar.  That cycle length is then used in indicators such as stochastic or RSI.  Whenever the dominant cycle changes, the program logic uses the new value, with no need for the programmer / trading system designer to set anything.  

Thanks,
Howard


----------



## grantk (7 April 2008)

Thought I might post my  on how I do Monte Carlo testing as I know it means different things to different people.
I use AmiBroker with an extremly large starting capitial and simple fixed position sizing to generate a list of every possible trade the system could generate over the choosen period.  The list of trades is then exported to a csv file.

I wrote a little java app that performs two variations on monte carlo testing.
It imports the csv and stores all of the trades then:

1.  randomizes the paths that could have been taken in trading during this period using a set number of runs (I usually use 5 thousand) This is the same approach that Nizar has detailed.

2.  It also then randomizes the order of the trades and performs test number 1 randomizing the trade order on each iteration.
I believe this helps give the sample data a bit more of a random feel as future data will be, while still keeps the integrity of the data.

Both tests then give the averages, outliers etc.  and both tests just add a little bit more information with which to judge how the system may perform with real money on the line tommorrow.


----------



## GreatPig (8 April 2008)

howardbandy said:


> PositionScore = Random



Just as a note, if you want to use the Mersenne Twister RNG, use:

PositionScore = mtRandomA();

not:

PositionScore = mtRandom();

The former returns an array of random numbers, the latter only a single value. This is stated in the help, but as any good programmer knows, you only read the manual when all else fails. 

GP


----------



## Chorlton (8 April 2008)

grantk said:


> Thought I might post my  on how I do Monte Carlo testing as I know it means different things to different people.
> I use AmiBroker with an extremly large starting capitial and simple fixed position sizing to generate a list of every possible trade the system could generate over the choosen period.  The list of trades is then exported to a csv file.
> 
> I wrote a little java app that performs two variations on monte carlo testing.
> ...





Hello Grantk,

Apologies if I misunderstand your approach, but if you are using a "very large" starting capital with simple fixed position sizing then, as you say, you will be generating a list of EVERY possible trade.

If this is the case, then can I ask what benefit there is in carrying out Monte Carlo testing on this list as every trade will still be taken?  Basically, the only thing you are assessing is the order in which they are taken.

My understanding on the benefit of Monte Carlo Analysis was to assess the impact to a system if every trade could NOT be taken, due to lack of capital, etc.

Just interested in your thoughts...

Cheers,

Chorlton


----------



## grantk (8 April 2008)

Chorlton said:


> Hello Grantk,
> 
> Apologies if I misunderstand your approach, but if you are using a "very large" starting capital with simple fixed position sizing then, as you say, you will be generating a list of EVERY possible trade.
> 
> ...





Ah, it would of made more sense if I had explained it properly.  The list of full trades is passed into the monte carlo application, which also has settings for position sizing and capital.  These are set before the monte carlo tests are run....


----------



## Chorlton (8 April 2008)

grantk said:


> Ah, it would of made more sense if I had explained it properly.  The list of full trades is passed into the monte carlo application, which also has settings for position sizing and capital.  These are set before the monte carlo tests are run....




Ah... now that makes sense


----------



## weird (8 April 2008)

Thanks Howard, but I don't really understand your reply though. Including or excluding some data with a once overall look or individual private viewing looks, it is still having a look. It is still optimizing.

 Most people look at Equity curves when backtesting systems, if the curve stops having the nice angle up, on any period ... then they usually discard the system ... this is testing on full data ... your 'other' data would fall within this equity curve picture, and would cause the system designer to discard the entire system seeing these very bad spots .

Is the wanting to test on unseen data , because one has totally optimized the parameters on the seen data ?


----------



## weird (9 April 2008)

The problem I see is that the parameters for testing on the 'seen data' will be optimized until they meet similar results on the 'unseen data'.


----------



## howardbandy (9 April 2008)

Greetings all --

A trade that will be made tomorrow will be an out-of-sample trade.

Question:  What is the best (only?) way to learn what might happen when your trading system goes from development on in-sample data to trading on out-of-sample data?  
Answer:  Observe as many transitions from in-sample to out-of-sample as possible.  

Every out-of-sample test, and every step of the walk forward testing, gives one more observation of that transition.

If the system designer never performs a truly out-of-sample test, on data that has never been seen before, that person has no idea what to expect when real trading begins with real money.

Remember, results from in-sample testing have No value as predictors of out-of-sample performance.  None!  Not even if the order of the in-sample trades are changed, or the position sizes are changed, or any other manipulations are made.  And no matter how steep and smooth the in-sample equity curve looks.  None!

Techniques that apply and work well to modeling non-time series data and to stationary time series data do not work well on the non-stationary time series data that we have.

In all of my education, research, and experience, I know of no way to learn what might happen when real trading begins other than to use out-of-sample testing.  

Think of this as preparing for an exam where the only question is "what is likely to happen when real trading of this new trading system begins?"  The market will score this exam, and it is not on a curve.

It is your choice --  get a free look at what might happen out-of-sample by using a portion of the historical data -- get a whole series of free looks by using walk forward testing -- or skip the testing, pay your money, and make a real trade.  

Even with the very best technique, there are no guarantees.  The best we can hope for is a high level of confidence.

Thanks for listening,
Howard


----------



## smersh (3 December 2008)

Hi all,

There has been no mention here (or I did not spot it) of using _synthetic market data_ for Monte Carlo testing.

I use randomly generated data sets to stress-test my systems. In particular I find it valuable to uncover software bugs that might exist in my systems that never show up when using real historical data because of the limited price paths available.

I hope it is not bad form or against the rules of this forum to mention my own product that I use to generate data sets which I then import into my off-the-shelf backtester (either Amibroker or RightEdge).

By testing a system over hundreds of "potential" markets I investigate the outliers in the system reports. I look for either unusually under/over performing "stocks" or individual trades and investigate why they occurred. This investigation sometimes leads to finding bugs in my systems.

Hope this helps.


----------



## GumbyLearner (3 December 2008)

As an IT graduate, I dont really care about preditable results...what is more concerning is market data that hasn't been released on the public record!!!

Too many assumptions for my liking, similar to what my Old English Form 5/6 Teacher used to tell me, "Dont assume, because if you do it makes an ass out of u and me"

At the same time, to rely on data that has no reported foundation or basis is a fallacy in itself! 

http://en.wikipedia.org/wiki/Deterministic_algorithm


----------



## howardbandy (10 December 2008)

smersh said:


> Hi all,
> 
> There has been no mention here (or I did not spot it) of using _synthetic market data_ for Monte Carlo testing.
> 
> ...




Hi Smersh --

Synthetic data might have some use in debugging a trading system Platform (of course, you could use real data to do that anyway and avoid the step of creating synthetic data), but it has no value in testing a trading system itself.  The markets are very close to efficient, and the tradable signal portion of the data is well hidden among the noise of the price and volume data.  If I knew enough to be able to create synthetic data that realistically tested my trading system, then I would already know the characteristics of the signal, and I would not need a test.

Thanks,
Howard


----------



## Wysiwyg (26 January 2010)

Link to a Monte Carlo Simulation product that has a free trial for those interested in venturing down this path.

http://www.riskamp.com/



> About RiskAMP
> 
> RiskAMP is a Monte Carlo simulation engine that works with Microsoft Excel ®. The RiskAMP Addin
> adds comprehensive probability simulation to spreadsheet models and Excel ® applications. The
> ...


----------



## Superboot (17 February 2010)

Hi all,

Can I ask how others create an 'in sample' data set for testing system concepts? I am using amibroker and have ~20 years of historical ASX data. I don't fancy creating a watch list of random shares... or is that the only way??

Thanks in advance 
S


----------



## tech/a (17 February 2010)

Its endless.
Depends on what you want to design a system around.

Suggestions which will be just a few.
Stocks which made their highest high in the last 3 mths
Or lowest low
Or trade a volume over 1 million a day for the last mth.
OR the ASX 50,100,200,300,500 full list.
All stocks starting with A


----------



## Superboot (17 February 2010)

Thanks tech/a,

I new to to amibroker so was trying to workout the easiest way to create a large watchlist. For others who may be in a similar boat, you can bulk add symbols from the results pane of the Automatic Analyzer - you can select multiple symbols, right click and add to a watch list. You will need to write some simple AFL code to list all the shares.

Hope this helps others. Also if there is a better way, I would be keen to hear 

Cheers
S


----------

