This is a mobile optimized page that loads fast, if you want to load the real page, click this text.

Monte Carlo analysis question...

Jump to new

Tend to agree with this.
 

That would reduce the number of people designing and testing systems for sure
 
It's the jump from testing to walking forward with real money that many find quite hard.
Click to expand...

The larger majority dispense with the testing bit and just step up to the plate with idea 101 and get straight into forward testing---urr trading.
 
tech/a said:
The larger majority dispense with the testing bit and just step up to the plate with idea 101 and get straight into forward testing---urr trading.
Click to expand...

I will put my hand up and say I have NEVER back tested.
 
Trembling Hand said:
I will put my hand up and say I have NEVER back tested.
Click to expand...

Any specific reason for not backtesting?

I guess with the average length of your trades being a few seconds, it wouldn't take long to just forward trade the method. A week would probably be several hundred trades and enough to tell you the essential statistics of the system.

The only problem with that is that you are initially putting yourself at risk of trading a system without knowing beforehand what it will perform like.
 
Like TH I did the very same thing.
Lost a motzza in the first 6 mths.

Travelling Nizars path I think fast tracked the learning not only of whats useless and whats not and what are the key ingredients of a successful method but how to construct one.

It also lead me to the path of being able to trade in a discretionary manner profitably once these keys were known.
Mind you I'm far more confident in trading larger $ value with tested methods than discretionary "Ideas".

Personally I have found Consistency comes from a structured format which has the key elements and a blueprint from which to work.

My discretionary ideas give rise to my systems in the end.
True there are some discretionary and successful methods which are for me at least to complex to turn into a trading system.

These I don't discard----- however its these which have the least/poorest consistency.

Like most here they will have found that you can be right more often over shorter periods,the difficulty is staying right long enough to collect a good profit often enough.

As Radge says its
"Being wrong isn't the problem its how long we STAY WRONG that is."
 

Thanks Shane. I agree, 2 years is not sufficient IMO either.

I am interested in the distrubution of results with a random function introduced into a backtest. I like to see the distribution bunch and shift in a favourable direction with refinement of parameters.
 
elbee said:
Assigning random numbers to the positionscore variable would appear to be using this function to do what it was designed to avoid.
Click to expand...

Perhaps, but this is why I like Amibroker. When you code at the low-level of the customer backtester it's so flexible that with some lateral thinking you can implement all kinds of things that the designer might never have thought of when he created the framework.
 

Hi Gorilla --

Yes, you can assign a random number generated within AmiBroker to PositionScore.

PositionScore = Random; // uses a random seed

or

PositionScore = Random (13331); // starts a sequence that is repeatable


Or -- ignoring PositionScore tells AmiBroker to use alphabetical order.

Thanks,
Howard
 

Hi Weird --

You can do that.

But making decisions to change the trading system based on the out-of-sample results changes those previously out-of-sample results into in-sample results. If you do not have another set of never-used data to make a final out-of-sample test, then the system is just as curve-fit / over-fit as any other optimized-but-never-tested system.

You may get lucky and have profitable trades from it. But the way you will learn is by using real money. Why not set aside some data for a no-cost out-of-sample test? If that is profitable, go ahead and trade the system -- if that is not profitable, you have saved yourself from trading losses.

If you are thinking about designing adaptive systems, build the adaption into the logic so the program senses the changes itself without the need for parameter modification after the fact. You can test whether this works by running out-of-sample tests.

Look at the work of John Ehlers for examples of adaptive logic. He has published some algorithms for detecting the length of the dominant cycle on each and every bar. That cycle length is then used in indicators such as stochastic or RSI. Whenever the dominant cycle changes, the program logic uses the new value, with no need for the programmer / trading system designer to set anything.

Thanks,
Howard
 
Thought I might post my on how I do Monte Carlo testing as I know it means different things to different people.
I use AmiBroker with an extremly large starting capitial and simple fixed position sizing to generate a list of every possible trade the system could generate over the choosen period. The list of trades is then exported to a csv file.

I wrote a little java app that performs two variations on monte carlo testing.
It imports the csv and stores all of the trades then:

1. randomizes the paths that could have been taken in trading during this period using a set number of runs (I usually use 5 thousand) This is the same approach that Nizar has detailed.

2. It also then randomizes the order of the trades and performs test number 1 randomizing the trade order on each iteration.
I believe this helps give the sample data a bit more of a random feel as future data will be, while still keeps the integrity of the data.

Both tests then give the averages, outliers etc. and both tests just add a little bit more information with which to judge how the system may perform with real money on the line tommorrow.
 
howardbandy said:
PositionScore = Random
Click to expand...
Just as a note, if you want to use the Mersenne Twister RNG, use:

PositionScore = mtRandomA();

not:

PositionScore = mtRandom();

The former returns an array of random numbers, the latter only a single value. This is stated in the help, but as any good programmer knows, you only read the manual when all else fails.

GP
 


Hello Grantk,

Apologies if I misunderstand your approach, but if you are using a "very large" starting capital with simple fixed position sizing then, as you say, you will be generating a list of EVERY possible trade.

If this is the case, then can I ask what benefit there is in carrying out Monte Carlo testing on this list as every trade will still be taken? Basically, the only thing you are assessing is the order in which they are taken.

My understanding on the benefit of Monte Carlo Analysis was to assess the impact to a system if every trade could NOT be taken, due to lack of capital, etc.

Just interested in your thoughts...

Cheers,

Chorlton
 


Ah, it would of made more sense if I had explained it properly. The list of full trades is passed into the monte carlo application, which also has settings for position sizing and capital. These are set before the monte carlo tests are run....
 
grantk said:
Ah, it would of made more sense if I had explained it properly. The list of full trades is passed into the monte carlo application, which also has settings for position sizing and capital. These are set before the monte carlo tests are run....
Click to expand...

Ah... now that makes sense
 
Thanks Howard, but I don't really understand your reply though. Including or excluding some data with a once overall look or individual private viewing looks, it is still having a look. It is still optimizing.

Most people look at Equity curves when backtesting systems, if the curve stops having the nice angle up, on any period ... then they usually discard the system ... this is testing on full data ... your 'other' data would fall within this equity curve picture, and would cause the system designer to discard the entire system seeing these very bad spots .

Is the wanting to test on unseen data , because one has totally optimized the parameters on the seen data ?
 
The problem I see is that the parameters for testing on the 'seen data' will be optimized until they meet similar results on the 'unseen data'.
 
Greetings all --

A trade that will be made tomorrow will be an out-of-sample trade.

Question: What is the best (only?) way to learn what might happen when your trading system goes from development on in-sample data to trading on out-of-sample data?
Answer: Observe as many transitions from in-sample to out-of-sample as possible.

Every out-of-sample test, and every step of the walk forward testing, gives one more observation of that transition.

If the system designer never performs a truly out-of-sample test, on data that has never been seen before, that person has no idea what to expect when real trading begins with real money.

Remember, results from in-sample testing have No value as predictors of out-of-sample performance. None! Not even if the order of the in-sample trades are changed, or the position sizes are changed, or any other manipulations are made. And no matter how steep and smooth the in-sample equity curve looks. None!

Techniques that apply and work well to modeling non-time series data and to stationary time series data do not work well on the non-stationary time series data that we have.

In all of my education, research, and experience, I know of no way to learn what might happen when real trading begins other than to use out-of-sample testing.

Think of this as preparing for an exam where the only question is "what is likely to happen when real trading of this new trading system begins?" The market will score this exam, and it is not on a curve.

It is your choice -- get a free look at what might happen out-of-sample by using a portion of the historical data -- get a whole series of free looks by using walk forward testing -- or skip the testing, pay your money, and make a real trade.

Even with the very best technique, there are no guarantees. The best we can hope for is a high level of confidence.

Thanks for listening,
Howard
 
Hi all,

There has been no mention here (or I did not spot it) of using synthetic market data for Monte Carlo testing.

I use randomly generated data sets to stress-test my systems. In particular I find it valuable to uncover software bugs that might exist in my systems that never show up when using real historical data because of the limited price paths available.

I hope it is not bad form or against the rules of this forum to mention my own product that I use to generate data sets which I then import into my off-the-shelf backtester (either Amibroker or RightEdge).

By testing a system over hundreds of "potential" markets I investigate the outliers in the system reports. I look for either unusually under/over performing "stocks" or individual trades and investigate why they occurred. This investigation sometimes leads to finding bugs in my systems.

Hope this helps.
 
You must log in or register to reply here.

Similar threads

Basic chart analysis/chart reading 101 for beginners
Replies
10
Views
4K
Gunnerguy
G
L
Drawdown Monte Carlo Increase over time PROBLEM
Replies
14
Views
5K
rb250660
The Beginner Trader's Manual Guide
Replies
9
Views
20K
commoyo
H
Amibroker and Monte Carlo backtesting - I think I've cracked it
Replies
6
Views
9K
Harro26
H
Monte Carlo Simulation of trades in backtesting a la Van Tharp
2 3
Replies
42
Views
24K
abattia
A
Menu
Log in
Register
Cookies are required to use this site. You must accept them to continue using the site. Learn more...