Monte Carlo analysis question...

[Chorlton]

"In sample" means historical data yeh? Whereas "out of sample" means forward testing?

Just trying to clear up a few of these definitions.

Be aware that "Out-of-Sample" can also be historical data as long as the System under development has NOT yet been exposed to it.

As soon as this does happen, it will then become part of the "in-Sample"

 

[tech/a]

MRC
Yes and yes to Chorlton.

 

[MRC & Co]

Ah k, yes logical, thanks guys.

 

[barney]

Hi guys, I'm probably out of my depth here cause I don't have any testing software yet, but am interested for future reference .............

Question re testing a system etc. ............. Has anyone done any "testing" of their various systems over time periods when the "Patterns" of the Index or Stock you are testing is (or as close as you can get to it) is similar to the current "pattern" .............. ie. Does the testing prove more successful under these conditions when the "ducks" are lined up in a similar fashion so to speak ............. Just a thought. It would seem to me that testing a system in a similar "environment" would carry a lot more weight if it showed positive expectancy, rather than just a random time frame ............. any thoughts?

EDIT: Just to clarify that ............ It would mean for example, you may test your data from say the year 2000-2001 or any time frame you thought applicable, because the market patterns were behaving in a similar pattern to the current time of testing .... hope that makes sense.

 

[theasxgorilla]

I do not want to sidetrack your thread, but to point out that the AmiBroker trading system platform handles that situation in a deterministic manner and avoids the need for Monte Carlo techniques in this circumstance.

Not sidetracking at all. I simply want to know what something is and how it works

Regarding the position score attribute in Amibroker...here is a wild idea...would it not be possible to randomise this through various iterations of a backtest and essentially simulate the Monte Carlo testing that TradeSim does...using optimisation of a dummy variable to generate each simulated portfolio?

ASX.G

 

[GreatPig]

Regarding the position score attribute in Amibroker...here is a wild idea...would it not be possible to randomise this through various iterations of a backtest and essentially simulate the Monte Carlo testing that TradeSim does...using optimisation of a dummy variable to generate each simulated portfolio?

From what I've seen, that wild idea has been used many times by many people for quite some time.

However, IMHO, it may not provide a great deal of randomisation if there are not a large number of buy signals. PositionScore only determines the entry when there are multiple possibilities at a single bar, and if a trading system doesn't generate a lot of signals, having multiple at a single bar may be fairly rare. In that case, the variation in portfolios between backtests may be small, or due to the nature of the system, certain trades may always be taken, skewing the results if those trades generate large profits or losses.

GP

 

[theasxgorilla]

However, IMHO, it may not provide a great deal of randomisation if there are not a large number of buy signals. PositionScore only determines the entry when there are multiple possibilities at a single bar, and if a trading system doesn't generate a lot of signals, having multiple at a single bar may be fairly rare. In that case, the variation in portfolios between backtests may be small, or due to the nature of the system, certain trades may always be taken, skewing the results if those trades generate large profits or losses.

I totally agree GP. And my system is very selective in it's entry setups so there aren't a great deal of circumstances where it is going to have multiple paths to take...and so my thinking is the variation is going to be very low and therefore I don't think I can imply very much from Monte Carlo testing in this fashion.

Or does TradeSim do some other magic when it performs a Monte Carlo simulation?

 

[theasxgorilla]

My question, specifically is, can/does Monte Carlo testing shuffle the chronological ordering of trades? If you were testing on the last 10 years of data, can a Monte Carlo simulation string together trades that were from 2001, then 1999, then 2007, then 2004?

No. Is there a purpose of this sort of simulation?
It would be utterly useless.

I must admit, I still don't get it.

I thought that Monte Carlo testing or simulating was terminology used to describe the process of introducing some random factor into a function, such that you could produce different outcomes on each run-through.

I thought that Monte Carlo could be done in a variety of ways. For example, ignoring a proportion of trades at random would result in varying backtest paths through data...this would be a way of running a Monte Carlo simulation.

Gathering every single possible trade from a data set and stringing them together in a random fashion to generate simulated equity curves could be another. You might chose to obey or disregard chronological ordering. If you disregard it you might chose to breakup trades into individual entities OR keep them together in chunks. I believe the latter is intended to try and preserve the effects of the underlying market conditions...shock events, for example.

The purpose of this kind of testing might be, as Howard mentions, to see how an equity curve and associated measures like max DD and consective winners/losers might have looked if the sequencing of trades was different. But with the consequence of breaking the relationship that the sequencing of trades had to market conditions.

 

[elbee]

Monte Carlo simulation is used in modelling when there is one or more variables in the model that cannot be easily quantified. Monte Carlo simulation assigns random variables to those unquantified variables so that the model can be run. Obviously this means a large number of runs with with different random numbers has to be carried out to get an indication of the possible range of results.

If all the variables in a model are quantifiable then there is no need for Monte Carlo simulation.

As I understand it (I have not used the software), the Monte Carlo function in TradeSim is to overcome a deficiency in MetaStock where it is not easy to specify which trades to take when there are multiple opportunities.

As Howard has pointed out this is overcome in Amibroker with the positionscore function.

Thus it is possible to precisely define a mechanical system or model in Amibroker and avoid the need for Monte Carlo simulation.

Assigning random numbers to the positionscore variable would appear to be using this function to do what it was designed to avoid.

 

[stevo]

Positionscore is not some magic bullet that makes Monte Carlo simulations a waste of time. It's another trading parameter that enables trades to be ranked in order of preference, based on some condition. The ranking could be as simple as using price to rank the order of trades, or even the alphabet.

If you find that ranking based on the alphabet gives better results than taking random trades then you would pick stocks starting with "A" over those starting with "B"! I am too sure that this is approach is valid - most people would argue that the name of a stock should have no bearing on it's performance.

Comparing a ranking strategy against taking random signals could give some insight into the performance of the ranking strategy. Where does the ranking strategy sit in terms of the distribution of possible outcomes without the rankinig strategy? How sensitive is it to minor changes? What happens if you hardcode into the buy signal that stocks from M to Z are ignored etc

Ranking gives one path out of the thousands generated in a Monte Carlo. Potentially a bit like getting an infinite number of monkeys, giving them all typewriters and getting one to write poetry.

I am currently reading Howard's book and I am really getting a lot out of it - just the first 3 chapters are making me think hard about the process of developing trading systems.

stevo

 

[nizar]

Comparing a ranking strategy against taking random signals could give some insight into the performance of the ranking strategy. Where does the ranking strategy sit in terms of the distribution of possible outcomes without the rankinig strategy? How sensitive is it to minor changes? What happens if you hardcode into the buy signal that stocks from M to Z are ignored etc

Ranking gives one path out of the thousands generated in a Monte Carlo. Potentially a bit like getting an infinite number of monkeys, giving them all typewriters and getting one to write poetry.

ASX.G.
Stevo has explained monte carlo analysis and its purpose quite succintly here.

As I understand it (I have not used the software), the Monte Carlo function in TradeSim is to overcome a deficiency in MetaStock where it is not easy to specify which trades to take when there are multiple opportunities.

As Howard has pointed out this is overcome in Amibroker with the positionscore function.

In TradeSim, it is in fact very easy to specify which trades to take when there are multiple opportunities. This function is completely separate and has nothing to do with Monte Carlo analysis.

You can rank according to stock price, rate of change, volatility, etc.

Monte Carlo analysis is a simulation designed to see the possible range of results, and the "tightness" of their distribution, and like Stevo said, its only through a monte carlo where your ranking fits in relation to the mean (just picking signals randomly).

If you do not perform a monte carlo analysis, then its not possible to see what benefit or adverse effect your ranking has on the results.

 

[nizar]

However, IMHO, it may not provide a great deal of randomisation if there are not a large number of buy signals. PositionScore only determines the entry when there are multiple possibilities at a single bar, and if a trading system doesn't generate a lot of signals, having multiple at a single bar may be fairly rare. In that case, the variation in portfolios between backtests may be small, or due to the nature of the system, certain trades may always be taken, skewing the results if those trades generate large profits or losses.

GP

I agree wholly.

 

[nizar]

Hi ASX.G

I thought that Monte Carlo testing or simulating was terminology used to describe the process of introducing some random factor into a function, such that you could produce different outcomes on each run-through.

Correct.

I thought that Monte Carlo could be done in a variety of ways. For example, ignoring a proportion of trades at random would result in varying backtest paths through data...this would be a way of running a Monte Carlo simulation.

Correct.

Gathering every single possible trade from a data set and stringing them together in a random fashion to generate simulated equity curves could be another.

You might chose to obey or disregard chronological ordering.

The whole purpose of systems testing is to see how a system would perform in real-life. As such we try to make the backtests as realistic as possible so we can deduct meaningful conclusions from the results.

What purpose would disregarding chronological ordering serve?

If you disregard it you might chose to breakup trades into individual entities OR keep them together in chunks. I believe the latter is intended to try and preserve the effects of the underlying market conditions...shock events, for example.

What do you mean by breaking up trades or keeping them in chunks?
Can we trade this way in real life?

 

[theasxgorilla]

Stevo has explained monte carlo analysis and its purpose quite succintly here.

Yeah i know...I still can't stop thinking to myself, "that's all it is???".

As I've said, for a system that doesn't have that many situations where the ranking mechanism (position score) kicks in, I think that randomly ignoring a proportion of trades would be a more effective way of backtesting alternate paths.

 

[tech/a]

Yeah i know...I still can't stop thinking to myself, "that's all it is???".

As I've said, for a system that doesn't have that many situations where the ranking mechanism (position score) kicks in, I think that randomly ignoring a proportion of trades would be a more effective way of backtesting alternate paths.

ASX

Really that's what it does.
Taking out a chunk in the middle 20% in or 80% in would basically be similar to starting the system without those chunks ever being included.
In real life they are there.

You could simply generate your system to start 6 mths later or a year earlier/later and finish 6 mths earlier or 6 mths later.That would give you 5 runs of the system.
You could then Montecarlo each of these then check and compare the results of each.---endless really.

But as I have pointed out you can test your brains out but if market conditions alter significantly beyond the parameters in which you have tested then there is a very good chance your system will be affected.
BOTH ways---positive affect or negative.

All we are doing is giving ourselves more confidence in our trading method and a blueprint with which to monitor performance.
Not to mention a structured plan with clear cut rules that if applied can have an expected positive expectancy---given the above caveat

 

[stevo]

Yeah i know...I still can't stop thinking to myself, "that's all it is???".

As I've said, for a system that doesn't have that many situations where the ranking mechanism (position score) kicks in, I think that randomly ignoring a proportion of trades would be a more effective way of backtesting alternate paths.

Something that I have tried. You would have to watch out that a buy signal really is ignored and not just bought in the following periods. For example all the buy parameters are valid but a flip of the coin knocks out the trade today. Then heads come up next month and the buy signal triggers.

Also exactly what are you testing? A different set of trades with each run or a trader that does not follow a system precisely? Do you knock out 1/2 the trades or only 10%? You could even "optimise" the number of trades to ignore!

As tech mentions there are no guarantees in this business.

stevo

 

[weird]

Unfortunately I find it hard to explain Monte Carlo from a Metastock/Tradesim perspective, to those that have never used the tool/combination (Metastock could be replaced with another package if that package is able to produce the required output, such as Amibroker).

I have seen some posts on competitive product mailing lists, to Metastock, where Tradesim sadly gets dismissed as a tool for making up for Metastock's system testing short comings. Its usefulness as a portfolio backtester, is just the tip of the iceberg.

For developing "portfolio based" stock trading systems, where there could be multiple buys signals, and there is some need to exclude trades due to position sizing constraints (that is Monte Carlo), this is a fantastic analysis tool.

By brute forcing the multiple possible combinations, it can show the range of potential outcomes, by summarizing the range of different possible paths constrained within the money management rules.

Another cool feature of Tradesim, is the easy ability to deselect any trade from the result, to remove outliners.

Positionscore while cool, does not easily allow one to remove the outliner effect.

 

[Shane Baker]

Unfortunately I find it hard to explain Monte Carlo from a Metastock/Tradesim perspective, to those that have never used the tool/combination (Metastock could be replaced with another package if that package is able to produce the required output, such as Amibroker).

I have seen some posts on competitive product mailing lists, to Metastock, where Tradesim sadly gets dismissed as a tool for making up for Metastock's system testing short comings. Its usefulness as a portfolio backtester, is just the tip of the iceberg.

For developing "portfolio based" stock trading systems, where there could be multiple buys signals, and there is some need to exclude trades due to position sizing constraints (that is Monte Carlo), this is a fantastic analysis tool.

By brute forcing the multiple possible combinations, it can show the range of potential outcomes, by summarizing the range of different possible paths constrained within the money management rules.

Another cool feature of Tradesim, is the easy ability to deselect any trade from the result, to remove outliners.

Positionscore while cool, does not easily allow one to remove the outliner effect.

I agree. Each peice of software has some great features. The problem is that there isn't any one commercially available tool that allows traders to develop and stress test their systems fully (except for the market of course ) Tradesim has some great features, Amibroker has some great features, Mechanica, Wealth Lab has some great features but they all have some deficiencies in their design which is probably a result of their developer's bias and longevity in the market (what was good design and user interface 10 years ago may not be now....are you listening Equis?).

Ultimately a developer may use multiple pieces of software and data to develop and stress test their systems. Eventually it all comes down to the final walk forward testing with real money.

Cheers

Shane

 

[stevo]

Shane
The main limit I have found in Amibroker is my own programming skills!

You can also Positionscore in TradeSim by using ranking criteria column. But when a Monte Carlo is run the ranking is turned off so that trade decisions are randomised rather than ranked. If there are no "leftover" trades then Monte Carlo is going to tell you much, although it could give an insight to slippage if you randomise the entry and exit prices.

I am very comfortable with TradeSim and Amibroker combo, even though some will tell me that Amibroker can do all that TradeSim can do. You can put any list of trades, in the right format, into TradeSim generated from any program (or even real trades) and do portfolio and position size testing.

It's the jump from testing to walking forward with real money that many find quite hard. A system test of 10 years worth of trades takes 1-2 minutes, but real life is much slower - I guess that it is "real".

You don't suffer the drawdowns in a system test. Mybe we need an electric shock add-in to simulate the pain of losses and give more direct feedback on a system's performance. Then slow down the test during the painful periods and speed it up when it's making money!

stevo

 

[weird]

[O/T from MC] Just some thoughts on forward walk, that is on testing on 'unseen data' ...

From reading about forward walk, those testing systems, may simply take this to mean to test on a defined period, as example 2000-2006, and then once you have completed this then test on unseen data, example 2006-2008. However others may also read this as 'optimize the parameters' as much as one can, then test how well those optimized parameters hold over another period. The emphasis here is on 'optimize'. While we all 'optimize' to a degree, some take it much further than others.

Some system designers and traders, will actually take the concept of 'forward walk' even another step further, and continue to walk forward test and continually adapt parameters through optimization as they trade (that is using the results of this type of testing as a form of system feedback),attempting to adapt to the changing market.

Some of these type of systems, may be very basic, such as a moving average cross-over. This approach to system design of optimization/walk-forward feedback, re-optimization, in my humble opinion is too focused on re-optimization of a system (which potentially could be beating a very dead horse), and not really determining why something may or not work in its raw form, now or in the future.