Monte Carlo analysis question...

[theasxgorilla]

Hi Everyone,

I'm looking for a assistance in my understanding of Monte Carlo analysis.

My question, specifically is, can/does Monte Carlo testing shuffle the chronological ordering of trades? If you were testing on the last 10 years of data, can a Monte Carlo simulation string together trades that were from 2001, then 1999, then 2007, then 2004?

Or does the path through the data have to be chronological?

ASX.G

 

[bingk6]

Hi Gorilla,

I cannot comment on the theoritical or acceptable definition and/or usage of Monte Carlo as I don't know enough about it. However my view is that a system that is able to maintain performance irrespective of whether the trades are executed in Chronological order or not is superior to one that depends on the trades being executed in chronological order.

This means that a system should be tested not just for a walk forward but a walk back as well and this means that the system should be able to perform during OOS periods BOTH subsequent to and prior to the in-sample period.

Off course, if there is a piece of software that is able to to jumble the order of trades across x different permutations, in additions to GP's random skipping of trades, in addition to the more traditional "trade selection when signals exceed position allocations" then these battery of tests should give the user more confidence about the ability of the system.

Amibroker can perform the last two easily, but the first is a challenge. Presumbly one would need to store all the signals into variables in the custom backtest procedure and then randomly issue the trades. Could run into problems with Amibroker not accepting trades that are not in time sequence. COuld be a real challenge. Anyway sorry for digressing, it is after all 3:30 in the morning!!

 

[howardbandy]

Greetings --

In my opinion, performing Monte Carlo re-ordering of trades in the in-sample period gives No information about the likelihood of the future profitability of a trading system.

Also, running a test over data that precedes the in-sample period will over-estimate the likelihood of the future profitability. Perhaps just a little; perhaps enough to "change the sign" and suggest that the system will be profitable when it will not; and there is no way of knowing without testing on data that follows the in-sample period.

The tests that give the best estimate of the future performance are those made on data that is more recent than the in-sample data and that has not been used during the system development.

Thanks for listening,
Howard

 

[theasxgorilla]

In my opinion, performing Monte Carlo re-ordering of trades in the in-sample period gives No information about the likelihood of the future profitability of a trading system.

Agreed. Does anyone know if this is how it is implemented in TradeSim?

 

[theasxgorilla]

However my view is that a system that is able to maintain performance irrespective of whether the trades are executed in Chronological order or not is superior to one that depends on the trades being executed in chronological order.

I'm actually of the opinion that chronological order matters a lot. Particularly when your have a universe of stocks that you are planning to trade. It's bedtime...I'll explain more later.

 

[nizar]

Agreed. Does anyone know if this is how it is implemented in TradeSim?

Monte Carlo analysis, as per TradeSims definition, is to identify different PATHS of trades.

Chronological order is not modified. You can see this from the Trade Log.

TradeSim only takes the trade if you would take the trade in real-life. Its a trading simulator.

Thats why monte carlo analysis in TradeSim is such an important tool for systems design in terms of analysing systems that by their design, cannot take all the trades that trigger.

Because if you just run a single simulation, you have no idea if the PATH that the software took was the very best, very worst, or just the average. Because for such systems there are MULTIPLE PATHS that you could have took, and each of these would have resulted in a different outcome.

Monte Carlo analysis will give you the performance (percent winners, number of trades, profit, drawdown) of the best portfolio (PATH), the worst, the average, and standard deviations from thousands of portfolios.

 

[nizar]

In response to the original post of this thread.

Hi Everyone,

I'm looking for a assistance in my understanding of Monte Carlo analysis.

My question, specifically is, can/does Monte Carlo testing shuffle the chronological ordering of trades? If you were testing on the last 10 years of data, can a Monte Carlo simulation string together trades that were from 2001, then 1999, then 2007, then 2004?

No. Is there a purpose of this sort of simulation?
It would be utterly useless.

Or does the path through the data have to be chronological?

Yes. The path must be chronological. Monte carlo analysis simulates different paths of trades.

 

[howardbandy]

Greetings all --

Monte Carlo simulation does have a place in trading system development. The technique of Monte Carlo is one of modifying some aspect of an experiment, model, simulation, etc, by adding a random component. The results of the modified runs are analyzed with the hope of leaning something about the original runs.

There are many ways that Monte Carlo can be used to advantage in trading systems. Here are three examples:

One way is to add random noise to the input data -- the open, high, low, close, volume, open interest, and so forth. When the trading system is re-run on the now noisier data, it will give an idea of how sensitive the system is to small changes in the data.

Another way is to add random noise to the values of the parameters -- the lookback lengths, buy and sell levels, profit targets, and so forth. This can give information about how sensitive the system is to its parameters -- that is, how robust the system is with respect to its parameters.

Another way is to reorder closed trades. This can be done either by using exactly the trades that were produced by the trading system, or by creating trades from a statistical distribution that has characteristics similar to those from the trading system. The reordered trades allow the system developer to see what might have happened if the winning and losing trades happened in a different order. For example, if the losing trades happened early in the run, would the trading account have been drawn down too far to allow continued trading.

My cautions include the following:

In-sample results have no value in determining the likely future performance, so reordering in-sample trades is a waste of time and cannot give any additional information.

It is possible that reordering out-of-sample trades will give some information about characteristics of the equity curve, but not always. The trading system may have logic built into it that tests the equity curve before placing buy and sell orders. Reordering those trades would not be a legitimate use of Monte Carlo analysis.

In my opinion, it is more important to perform rigorous out-of-sample testing than it is to perform Monte Carlo analysis, although properly done Monte Carlo analysis can improve the confidence that the developer has in his or her system.

When Monte Carlo analysis is performed, the developer must carefully think through the process and the interpretation of the results.

Thanks for listening,
Howard

 

[tech/a]

In my opinion, it is more important to perform rigorous out-of-sample testing than it is to perform Monte Carlo analysis, although properly done Monte Carlo analysis can improve the confidence that the developer has in his or her system.

Howard Ive seen you mention this many times.

I keep asking myself.

"Once Ive done an out of sample test it becomes "in sample"
There is no guarentee that future out of sample tests are going to perform the same,good or bad."

Really then all testing can offer is an indication that any set of conditions and parameters can perform similarly given a similar environment.

In sample
Out of Sample
Optimised
Montecarlo Analysis
Various markets
Maximised testing periods
Massive data inputs
Various environments

Doesnt matter.
Change the environment and youll change the results.
You'll never have a definative answer to
"Is my system likley to be profitable after testing indicates that it could be."

Not to say testing is a pointless exercise.

 

[theasxgorilla]

Yes. The path must be chronological. Monte carlo analysis simulates different paths of trades.

Okay, cool.

Any idea how Tradesim does this? Does it ignore trades at random, or randomise delaying the start date of the testing period?

ASX.G

 

[theasxgorilla]

Another way is to reorder closed trades. This can be done either by using exactly the trades that were produced by the trading system, or by creating trades from a statistical distribution that has characteristics similar to those from the trading system. The reordered trades allow the system developer to see what might have happened if the winning and losing trades happened in a different order. For example, if the losing trades happened early in the run, would the trading account have been drawn down too far to allow continued trading.

Is this what I am talking about? Put another way, mucking with the chronological ordering of the (closed) trades taken by the system.

 

[howardbandy]

Hi Tech, and all --

You are correct -- there are no guarantees. The best we can do is follow good modeling and simulation practices, and develop as much confidence as possible in both our trading system and the development process.

It is particularly important to understand and be confident about the transition from development to trading. In my opinion, the best -- and as far as I know, the only -- way to get that understanding is to practice. That is, to observe a lot of transitions from in-sample development to out-of-sample testing. An easy way to do this, and to do it using good techniques, is an automated walk forward process.

Some practices are better than others. Of course, any one can develop his or her trading systems any way he or she wishes.

What I try to raise awareness of and caution against are those practices that are known to over-estimate (often very significantly) the likelihood of acceptable performance in the future -- particularly when the trades are made with real money. Some of the techniques described in the literature, posted on discussion boards, and practiced by some developers are very dangerous to their wealth, or to the wealth of the clients they hope to attract.

Full apologies in advance to anyone on the ASF board who takes offense -- none is intended. And I do not imply that anyone here is knowingly deceiving anyone. The biggest risk for deception is probably that of the trading system developer to him or herself.

Thanks for listening,
Howard

 

[tech/a]

What I try to raise awareness of and caution against are those practices that are known to over-estimate (often very significantly)

Howard while I have your book and if these practices are listed there I will look them up.
If not and if scattered throughout your work,is it possible to perhaps list some of these common mistakes (practices) that as developers we are likely to mislead ourselves with?

Single tests of a set of conditions comes to mind.
Incorrect use of optimisation another.
Insufficient data
Perhaps some which are not so obvious.

Thanks.

 

[nizar]

Okay, cool.

Any idea how Tradesim does this? Does it ignore trades at random, or randomise delaying the start date of the testing period?

ASX.G

No, it does not ignore trades at random.

There is another function within TradeSim to ignore a percentage of trades.

As to Monte Carlo analysis:

There are certain points during the course of trading where a decision MUST be made because there is insufficient available capital to take all the trades.

For example. Week 1.
12 signals.
You can afford 8.

Which ones?

Monte carlo analysis simulation #1. Will pick the first 8 trades.
Monte carlo analysis simulation #2. Picks trades 1,2,3,7,8,9,11,12.

A few weeks down the track.

You have enough capital for 3 stocks.
Five signals pop up.

Monte carlo analysis as per TradeSim's definition, will take every possible PATH of trades that had been possible if you were trading the system in real life.

Techs way of putting it is very good.

Monte Carlo analysis is like giving your system and the same available starting capital to 20,000 people, telling them to come back in 10 years time, and then collating the results. You then have the mean, and also the positive and negative outliers, and the standard deviations for profit, max.DD, number of trades, percent winners, etc.

With this information, you will have a good idea of how the system can be expected to perform, and more importantly, the range of distribution of each parameter.

 

[nizar]

Also, just to add.

I dont think monte carlo analysis and walk forward analysis is not an area where you pick one or the other.

I did both.

The initial 6 year testing period (in which the system was optimised) was 1998-2003. Then I did monte carlo runs in 2 year blocks on the out-of sample data.

 

[theasxgorilla]

Monte Carlo analysis is like giving your system and the same available starting capital to 20,000 people, telling them to come back in 10 years time, and then collating the results. You then have the mean, and also the positive and negative outliers, and the standard deviations for profit, max.DD, number of trades, percent winners, etc.

Okay, so when presented with multiple trades and insufficient capital to take them all, TradeSim will use some random function to decide which trades to take. Am I getting it?

So the value of this kind of Monte Carlo analysis must be dependent upon enough of these junctures existing, in order to get the multiple paths you talk about.

Will TradeSim report if it repeats a path, or will it stop testing if it's exhausted all possible paths? How many paths are there using a garden variety LTTF system and 2 years worth of data on the ASX300?

Probably a bit difficult to answer that last question decisively. I'm just exploring the question. I keep seeing this 20,000 figure thrown up, but wonder if there are 20,000 paths through such a data set...and if there is, how much they can reasonably be expected to vary by.

ASX.G

 

[howardbandy]

Greetings all --

I understand the discussion has been about how TradeSim handles a situation of having more signals than available funds, and how Monte Carlo techniques can be used to simulate the results.

I do not want to sidetrack your thread, but to point out that the AmiBroker trading system platform handles that situation in a deterministic manner and avoids the need for Monte Carlo techniques in this circumstance.

AmiBroker gives the system developer the option of deciding which trades to take within the trading system code. A system variable called PositionScore is used to break ties.

For every bar and for every ticker symbol, compute not only the signals and indicators that will be used to enter a new trade, but also compute PositionScore. If there are more signals at a given time than there are "slots", take positions in those that have the highest PositionScore.

Good choices for computing PositionScore are the same that would be made if the trader could evaluate all of the potential entries at the time of the trade. Some might want to buy the issues that have been strongest recently, others may want to buy those that have been weakest, others may want to buy those that belong to industries that are not already owned, and so forth.

Thanks for listening,
Howard

 

[Shane Baker]

Hi ASX

TradeSim takes the trades alphabetically in the normal backtest unless instructed otherwise. You can rank the trades similar to, but not as flexibly as Amibroker by using the ExtFml( "TradeSim.SetVariableTradeRank",); function.Personally I like to see the LTTF system with and without this function used in the MC stats as I like to make sure that the very basic premise of the LTTF system holds up to scrutiny.

I am not sure it is important to know whether any paths have been repeated as I am mainly interested in MC testing to look at the extremes of results rather than the mean or median results. I might suggest that two years for a LTTF data set is too short and a much longer period will produce more meaningful results (including assessing performance over various market condictions).

Cheers

Shane

 

[nizar]

Hi ASX.G

Okay, so when presented with multiple trades and insufficient capital to take them all, TradeSim will use some random function to decide which trades to take. Am I getting it?

Yes!!

So the value of this kind of Monte Carlo analysis must be dependent upon enough of these junctures existing, in order to get the multiple paths you talk about.

On the ball, son!

Will TradeSim report if it repeats a path, or will it stop testing if it's exhausted all possible paths? How many paths are there using a garden variety LTTF system and 2 years worth of data on the ASX300?

It will test as many portfolios as you instruct it to.
I normally choose between 5,000-20,000.
For 2 years, I would use 5,000.
For 6 years of more, I did 20,000.
Even though I could select any number I want, More than 20,000 it just takes too long.

Probably a bit difficult to answer that last question decisively. I'm just exploring the question. I keep seeing this 20,000 figure thrown up, but wonder if there are 20,000 paths through such a data set...and if there is, how much they can reasonably be expected to vary by.

What I look for in my monte carlo results is a tight distribution of results.
Ideally, there shouldnt be much different between the max. and min. profits and drawdown (low standard deviation from the mean).

I don't know about whether 20,000 is excessive or not. BUT during the first couple of months of trading my LTTF, I was getting between 22-28 signals thrown every week, even though I could by no more than 10. And I bought them all (maybe foolishly) in the first week. So say in the first 8 weeks you were getting between 22 and 28 signals and you could buy only 10 from all of them (obviously if you got stopped out you could buy again). But from such a small sample of 2 months there were probably hundreds of possible paths.

Also, when testing on only 2 years of data, you must remember to close out all open trades at the end of the testing period, because this is possibly where most of the profit will be.

 

[MRC & Co]

"Once Ive done an out of sample test it becomes "in sample"
There is no guarentee that future out of sample tests are going to perform the same,good or bad."

"In sample" means historical data yeh? Whereas "out of sample" means forward testing?

Just trying to clear up a few of these definitions.