Tend to agree with this.
Monte Carlo analysis question...
- Thread starter theasxgorilla
- Start date
Tend to agree with this.
- Joined
- 14 March 2006
- Posts
- 114
- Reactions
- 1
tech/a
No Ordinary Duck
- Joined
- 14 October 2004
- Posts
- 20,417
- Reactions
- 6,356
The larger majority dispense with the testing bit and just step up to the plate with idea 101 and get straight into forward testing---urr trading.
Trembling Hand
Can be found on the bid
- Joined
- 10 June 2007
- Posts
- 8,852
- Reactions
- 205
I will put my hand up and say I have NEVER back tested.
Any specific reason for not backtesting?
I guess with the average length of your trades being a few seconds, it wouldn't take long to just forward trade the method. A week would probably be several hundred trades and enough to tell you the essential statistics of the system.
The only problem with that is that you are initially putting yourself at risk of trading a system without knowing beforehand what it will perform like.
Trembling Hand
Can be found on the bid
- Joined
- 10 June 2007
- Posts
- 8,852
- Reactions
- 205
Like most punters I just started trading.
Yes my testing was very expensive.
tech/a
No Ordinary Duck
- Joined
- 14 October 2004
- Posts
- 20,417
- Reactions
- 6,356
Like TH I did the very same thing.
Lost a motzza in the first 6 mths.
Travelling Nizars path I think fast tracked the learning not only of whats useless and whats not and what are the key ingredients of a successful method but how to construct one.
It also lead me to the path of being able to trade in a discretionary manner profitably once these keys were known.
Mind you I'm far more confident in trading larger $ value with tested methods than discretionary "Ideas".
Personally I have found Consistency comes from a structured format which has the key elements and a blueprint from which to work.
My discretionary ideas give rise to my systems in the end.
True there are some discretionary and successful methods which are for me at least to complex to turn into a trading system.
These I don't discard----- however its these which have the least/poorest consistency.
Like most here they will have found that you can be right more often over shorter periods,the difficulty is staying right long enough to collect a good profit often enough.
As Radge says its
"Being wrong isn't the problem its how long we STAY WRONG that is."
Lost a motzza in the first 6 mths.
Travelling Nizars path I think fast tracked the learning not only of whats useless and whats not and what are the key ingredients of a successful method but how to construct one.
It also lead me to the path of being able to trade in a discretionary manner profitably once these keys were known.
Mind you I'm far more confident in trading larger $ value with tested methods than discretionary "Ideas".
Personally I have found Consistency comes from a structured format which has the key elements and a blueprint from which to work.
My discretionary ideas give rise to my systems in the end.
True there are some discretionary and successful methods which are for me at least to complex to turn into a trading system.
These I don't discard----- however its these which have the least/poorest consistency.
Like most here they will have found that you can be right more often over shorter periods,the difficulty is staying right long enough to collect a good profit often enough.
As Radge says its
"Being wrong isn't the problem its how long we STAY WRONG that is."
theasxgorilla
Problem solved... next bubble.
- Joined
- 7 December 2006
- Posts
- 2,343
- Reactions
- 1
Thanks Shane. I agree, 2 years is not sufficient IMO either.
I am interested in the distrubution of results with a random function introduced into a backtest. I like to see the distribution bunch and shift in a favourable direction with refinement of parameters.
theasxgorilla
Problem solved... next bubble.
- Joined
- 7 December 2006
- Posts
- 2,343
- Reactions
- 1
Perhaps, but this is why I like Amibroker. When you code at the low-level of the customer backtester it's so flexible that with some lateral thinking you can implement all kinds of things that the designer might never have thought of when he created the framework.
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Not sidetracking at all. I simply want to know what something is and how it works
Regarding the position score attribute in Amibroker...here is a wild idea...would it not be possible to randomise this through various iterations of a backtest and essentially simulate the Monte Carlo testing that TradeSim does...using optimisation of a dummy variable to generate each simulated portfolio?
ASX.G
Hi Gorilla --
Yes, you can assign a random number generated within AmiBroker to PositionScore.
PositionScore = Random; // uses a random seed
or
PositionScore = Random (13331); // starts a sequence that is repeatable
Or -- ignoring PositionScore tells AmiBroker to use alphabetical order.
Thanks,
Howard
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Hi Weird --
You can do that.
But making decisions to change the trading system based on the out-of-sample results changes those previously out-of-sample results into in-sample results. If you do not have another set of never-used data to make a final out-of-sample test, then the system is just as curve-fit / over-fit as any other optimized-but-never-tested system.
You may get lucky and have profitable trades from it. But the way you will learn is by using real money. Why not set aside some data for a no-cost out-of-sample test? If that is profitable, go ahead and trade the system -- if that is not profitable, you have saved yourself from trading losses.
If you are thinking about designing adaptive systems, build the adaption into the logic so the program senses the changes itself without the need for parameter modification after the fact. You can test whether this works by running out-of-sample tests.
Look at the work of John Ehlers for examples of adaptive logic. He has published some algorithms for detecting the length of the dominant cycle on each and every bar. That cycle length is then used in indicators such as stochastic or RSI. Whenever the dominant cycle changes, the program logic uses the new value, with no need for the programmer / trading system designer to set anything.
Thanks,
Howard
- Joined
- 4 July 2007
- Posts
- 11
- Reactions
- 0
Thought I might post my 
on how I do Monte Carlo testing as I know it means different things to different people.
I use AmiBroker with an extremly large starting capitial and simple fixed position sizing to generate a list of every possible trade the system could generate over the choosen period. The list of trades is then exported to a csv file.
I wrote a little java app that performs two variations on monte carlo testing.
It imports the csv and stores all of the trades then:
1. randomizes the paths that could have been taken in trading during this period using a set number of runs (I usually use 5 thousand) This is the same approach that Nizar has detailed.
2. It also then randomizes the order of the trades and performs test number 1 randomizing the trade order on each iteration.
I believe this helps give the sample data a bit more of a random feel as future data will be, while still keeps the integrity of the data.
Both tests then give the averages, outliers etc. and both tests just add a little bit more information with which to judge how the system may perform with real money on the line tommorrow.
on how I do Monte Carlo testing as I know it means different things to different people.I use AmiBroker with an extremly large starting capitial and simple fixed position sizing to generate a list of every possible trade the system could generate over the choosen period. The list of trades is then exported to a csv file.
I wrote a little java app that performs two variations on monte carlo testing.
It imports the csv and stores all of the trades then:
1. randomizes the paths that could have been taken in trading during this period using a set number of runs (I usually use 5 thousand) This is the same approach that Nizar has detailed.
2. It also then randomizes the order of the trades and performs test number 1 randomizing the trade order on each iteration.
I believe this helps give the sample data a bit more of a random feel as future data will be, while still keeps the integrity of the data.
Both tests then give the averages, outliers etc. and both tests just add a little bit more information with which to judge how the system may perform with real money on the line tommorrow.
GreatPig
Pigs In Space
- Joined
- 9 July 2004
- Posts
- 2,368
- Reactions
- 14
Just as a note, if you want to use the Mersenne Twister RNG, use:
PositionScore = mtRandomA();
not:
PositionScore = mtRandom();
The former returns an array of random numbers, the latter only a single value. This is stated in the help, but as any good programmer knows, you only read the manual when all else fails.
GP
- Joined
- 14 April 2007
- Posts
- 317
- Reactions
- 0
Thought I might post my
I use AmiBroker with an extremly large starting capitial and simple fixed position sizing to generate a list of every possible trade the system could generate over the choosen period. The list of trades is then exported to a csv file.
I wrote a little java app that performs two variations on monte carlo testing.
It imports the csv and stores all of the trades then:
1. randomizes the paths that could have been taken in trading during this period using a set number of runs (I usually use 5 thousand) This is the same approach that Nizar has detailed.
2. It also then randomizes the order of the trades and performs test number 1 randomizing the trade order on each iteration.
I believe this helps give the sample data a bit more of a random feel as future data will be, while still keeps the integrity of the data.
Both tests then give the averages, outliers etc. and both tests just add a little bit more information with which to judge how the system may perform with real money on the line tommorrow.
Hello Grantk,
Apologies if I misunderstand your approach, but if you are using a "very large" starting capital with simple fixed position sizing then, as you say, you will be generating a list of EVERY possible trade.
If this is the case, then can I ask what benefit there is in carrying out Monte Carlo testing on this list as every trade will still be taken? Basically, the only thing you are assessing is the order in which they are taken.
My understanding on the benefit of Monte Carlo Analysis was to assess the impact to a system if every trade could NOT be taken, due to lack of capital, etc.
Just interested in your thoughts...
Cheers,
Chorlton
- Joined
- 4 July 2007
- Posts
- 11
- Reactions
- 0
Ah, it would of made more sense if I had explained it properly. The list of full trades is passed into the monte carlo application, which also has settings for position sizing and capital. These are set before the monte carlo tests are run....
- Joined
- 14 April 2007
- Posts
- 317
- Reactions
- 0
- Joined
- 4 October 2005
- Posts
- 519
- Reactions
- 1
Thanks Howard, but I don't really understand your reply though. Including or excluding some data with a once overall look or individual private viewing looks, it is still having a look. It is still optimizing.
Most people look at Equity curves when backtesting systems, if the curve stops having the nice angle up, on any period ... then they usually discard the system ... this is testing on full data ... your 'other' data would fall within this equity curve picture, and would cause the system designer to discard the entire system seeing these very bad spots .
Is the wanting to test on unseen data , because one has totally optimized the parameters on the seen data ?
Most people look at Equity curves when backtesting systems, if the curve stops having the nice angle up, on any period ... then they usually discard the system ... this is testing on full data ... your 'other' data would fall within this equity curve picture, and would cause the system designer to discard the entire system seeing these very bad spots .
Is the wanting to test on unseen data , because one has totally optimized the parameters on the seen data ?
- Joined
- 4 October 2005
- Posts
- 519
- Reactions
- 1
The problem I see is that the parameters for testing on the 'seen data' will be optimized until they meet similar results on the 'unseen data'.
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Greetings all --
A trade that will be made tomorrow will be an out-of-sample trade.
Question: What is the best (only?) way to learn what might happen when your trading system goes from development on in-sample data to trading on out-of-sample data?
Answer: Observe as many transitions from in-sample to out-of-sample as possible.
Every out-of-sample test, and every step of the walk forward testing, gives one more observation of that transition.
If the system designer never performs a truly out-of-sample test, on data that has never been seen before, that person has no idea what to expect when real trading begins with real money.
Remember, results from in-sample testing have No value as predictors of out-of-sample performance. None! Not even if the order of the in-sample trades are changed, or the position sizes are changed, or any other manipulations are made. And no matter how steep and smooth the in-sample equity curve looks. None!
Techniques that apply and work well to modeling non-time series data and to stationary time series data do not work well on the non-stationary time series data that we have.
In all of my education, research, and experience, I know of no way to learn what might happen when real trading begins other than to use out-of-sample testing.
Think of this as preparing for an exam where the only question is "what is likely to happen when real trading of this new trading system begins?" The market will score this exam, and it is not on a curve.
It is your choice -- get a free look at what might happen out-of-sample by using a portion of the historical data -- get a whole series of free looks by using walk forward testing -- or skip the testing, pay your money, and make a real trade.
Even with the very best technique, there are no guarantees. The best we can hope for is a high level of confidence.
Thanks for listening,
Howard
A trade that will be made tomorrow will be an out-of-sample trade.
Question: What is the best (only?) way to learn what might happen when your trading system goes from development on in-sample data to trading on out-of-sample data?
Answer: Observe as many transitions from in-sample to out-of-sample as possible.
Every out-of-sample test, and every step of the walk forward testing, gives one more observation of that transition.
If the system designer never performs a truly out-of-sample test, on data that has never been seen before, that person has no idea what to expect when real trading begins with real money.
Remember, results from in-sample testing have No value as predictors of out-of-sample performance. None! Not even if the order of the in-sample trades are changed, or the position sizes are changed, or any other manipulations are made. And no matter how steep and smooth the in-sample equity curve looks. None!
Techniques that apply and work well to modeling non-time series data and to stationary time series data do not work well on the non-stationary time series data that we have.
In all of my education, research, and experience, I know of no way to learn what might happen when real trading begins other than to use out-of-sample testing.
Think of this as preparing for an exam where the only question is "what is likely to happen when real trading of this new trading system begins?" The market will score this exam, and it is not on a curve.
It is your choice -- get a free look at what might happen out-of-sample by using a portion of the historical data -- get a whole series of free looks by using walk forward testing -- or skip the testing, pay your money, and make a real trade.
Even with the very best technique, there are no guarantees. The best we can hope for is a high level of confidence.
Thanks for listening,
Howard
Hi all,
There has been no mention here (or I did not spot it) of using synthetic market data for Monte Carlo testing.
I use randomly generated data sets to stress-test my systems. In particular I find it valuable to uncover software bugs that might exist in my systems that never show up when using real historical data because of the limited price paths available.
I hope it is not bad form or against the rules of this forum to mention my own product that I use to generate data sets which I then import into my off-the-shelf backtester (either Amibroker or RightEdge).
By testing a system over hundreds of "potential" markets I investigate the outliers in the system reports. I look for either unusually under/over performing "stocks" or individual trades and investigate why they occurred. This investigation sometimes leads to finding bugs in my systems.
Hope this helps.
There has been no mention here (or I did not spot it) of using synthetic market data for Monte Carlo testing.
I use randomly generated data sets to stress-test my systems. In particular I find it valuable to uncover software bugs that might exist in my systems that never show up when using real historical data because of the limited price paths available.
I hope it is not bad form or against the rules of this forum to mention my own product that I use to generate data sets which I then import into my off-the-shelf backtester (either Amibroker or RightEdge).
By testing a system over hundreds of "potential" markets I investigate the outliers in the system reports. I look for either unusually under/over performing "stocks" or individual trades and investigate why they occurred. This investigation sometimes leads to finding bugs in my systems.
Hope this helps.
