Australian (ASX) Stock Market Forum

System Robustness

Jump to new
I'm only presenting my view/experiences.
I'm not here to convince anyone.

Nicks In Adelaide at the moment and not that well unfortunately.
The thought of Dinner with tech can be quite upsetting!
 
Hi Nizar,

Here is an interesting page on "Robustness".

http://www.turtletrader.com/robust-trading-system.html

On the next page following Robust, on curve-fitting:-

Trend Following parameters or rules work across a range of values. System parameters that work over a range of values are robust. If the parameters of a system are slightly changed and the performance adjusts drastically, beware. For example, if a system works great at 20, but does not work at 19 or 21 you have a system with poor robustness. On the other hand, if your system parameter is 50 and it also works at 40 or 60, your system is much more robust (and reliable).

That optimizing "addon" to Amibroker runs a system across all of your parameters giving a visual result of whether a slight change in parameters could turn the system to crap.

The big evil obtimizing/curve fitting, I agree 100% it's dangerous but if you test it on past data then moving forward and using those parameters in the following year (still using historical data) you can see what effect it
may have. That software (IO) DOES NOT pick the best case (curve-fitted) parameters either.

If you have tested a system I'm sure you would have run multiple simulations to try and find better and better results (optimizing). This is just getting straight to the point quickly and seeing if it works in the future and if any changes in parameters result in a system failure.

Also, least parameters are more in a system.
 
I guess it boils down to the our/ individuals definition of optimise.

To me that means testing a wide range of variables against an instrument in the attempt to find the variable which performs best.

The possibility that in a few weeks time the tested and selected optimised variable is no longer the best choice renders (to me atleast) the constant attempt to determine the best possible variable allocation pointless.

The situation is bad enough with a singular instrument (Stock/Future/Index etc) but then trying to apply it to a portfolio where the constituents are likely to move in an un correlated manner at any period in time---is (Again to me at least) a waste of time.

There is of course the arguement that the resultant variables chosen are no better or worse than those chosen by any other mean.
That the "Blueprint" if with an acceptable set of numbers can be traded just as any choice of variables with the same.

I'm yet to see a long term walked forward test either live or simulated which shows an appreciable improvement in a system over a reasonable period of time (500 periods of selected timeframe/2yrs on a daily).

But I'm willing to be educated.
 
Great Thread,

Sir Burr correct me if im wrong but your statement 'That software (IO) DOES NOT pick the best case (curve-fitted) parameters either' is not entirely correct.

Given IO is designed with two optimization modes, Swarm Optimization and Differential Optimization, (please view this for an explanation of swarm http://www.projectcomputing.com/resources/psovis/ also i don t know differential optimization that well perhaps someone could discuss this and mayb compare these two modes of optimization)and IO is generally in Swarm optimization.

Swarm optimization basically works by, you have a fitness space(like that in the link above), you spawn randomly a certain number of searchers throughout the space, then recursively,

access the value of each searcher(in the link above this means how high each searcher is at its current location),
the best values for each individual searcher is stored(storage of location and value of best),
the best value obtained by the whole swarm is stored,
and then each searcher is given a new location that is influenced by its own best value and the global best found so far

This loop is performed until some specific criteria is meet then the location of the best value is returned

Now what i think your trying to say, is that the solution that IO finds will be related to the 'curvature' of the surrounding landscape of the optimization space (also other parameters such as the number of particles(searchers), the manner in that the particles search, and also the nature of the search space itself, im sure theres more!) and by that the solutions that IO finds will be at the top of a hill such that hill will be reasonably rounded at the top. Also I think you are trying to point out that IO is unlikely to find maxima that simply stick straight out of the landscape (i.e in the link above this would be the equivalent to seeing something like a tall narrow 'stick like' shape pointing out of the landscape).

So to some extent your statement is true, IO is unlikely to find these stick-like hills, but beware this shouldn't be assumed and sometimes IO might get lucky and find these.
 
Tech, using metastock/tradesim, how do i get a results of a simulation without the 10 best champions (and/or 10 worst dogs) ?
 
This is how I do it.

A bit long winded.
Go to the trade records sheet.
Find those trades you wish to remove.(The Codes)
Record them on a pad.

Go to your Metastock exploration.
set up an exploration for the system
Go to securities and untick those you DONT wish
to be included.
re run the exploration and they wont be included.

ExY
Did you get an optimisation encyclopedia for Xmas!!
 
An alternative to Tech’s method is as follows:-

a) From the Trade Log Window note the unique trade numbers you wish to exclude from this simulation,

b) go to the Trade Database Window and remove the tick from the box of the trade numbers noted in a) above,

c) then press Start Simulation and the new results will now exclude those trades.
 
Thanks RnR and tech.

Is anything wrong with a system if the results are much different when the top10 winners and top10 losers are taken out??

Richard Dennis did say:
The worst mistake a trader can make is to miss a major profit opportunity. 95 percent of profits come from only 5 percent of the trades.
Click to expand...

Thoughts?

Surely techtrader was the same??
 
I think this is the essense of longer term trend following. Often, < 40% winners, and of those 40% that win < 20% win really big, eclipsing the 80% that meander or fail. Another approach for testing system resilience that I saw GP and Stevo experimenting with recently was to randomly skip a percentage of trades during a run. The more you can skip without affecting your stats the less reliant your system is on the opportunity factor provided by future market conditions...or that was my understanding.
 
theasxgorilla said:
I think this is the essense of longer term trend following. Often, < 40% winners, and of those 40% that win < 20% win really big, eclipsing the 80% that meander or fail. Another approach for testing system resilience that I saw GP and Stevo experimenting with recently was to randomly skip a percentage of trades during a run. The more you can skip without affecting your stats the less reliant your system is on the opportunity factor provided by future market conditions...or that was my understanding.
Click to expand...

Great idea.

Does anybody know how to do this in metastock/tradesim?
 
nizar said:
Great idea.

Does anybody know how to do this in metastock/tradesim?
Click to expand...


Never done it.
But would think you could put in different start and stop dates in Tradesim.
Ask stevo.

On taking out the best and worse Id only bother with say the top 3.
You just dont one a few trades skewing the results one way or the other radically.
 
theasxgorilla said:
Another approach for testing system resilience that I saw GP and Stevo experimenting with recently was to randomly skip a percentage of trades during a run. The more you can skip without affecting your stats the less reliant your system is on the opportunity factor provided by future market conditions...or that was my understanding.
Click to expand...

How can the information gained by this approach be anymore useful in testing the 'resilience' of the system?

This method seems to resemble monte carlo whereas instead of randomly selecting trades at random your simply rejecting them(is there any difference?!)
 
Guys, while I got the attention of all you Tradesim experts I might ask a quick question.

My mate has offered to test my system on his TradeSim if I can send him my backtest report as a trt file ...

Now Amibroker can export a csv file of the backtest, but I donno any other method of converting it to a trt file format.

Your help would be much appreciated.
 
Sorry cant help.
BUT all he needs is the formulas in M/S format and your test settings,universe of stocks and time periods.
 
I tried the Convert.exe (from Yahoo files) but it just created a 0 kb trt file.

With that procedure, where do I imbed it into the AFL ?
 

Similar threads

MovingAverage
Emergency Exit for Weekly System
2
Replies
27
Views
5K
MovingAverage
MovingAverage
Linus van Pelt
System optimisation: IS training vs. OOS testing, and Paper Trading
Replies
10
Views
2K
Garpal Gumnut
Garpal Gumnut
R
21 Lessons in Investing
Replies
6
Views
3K
divs4ever
divs4ever
S
Chasing participation
Replies
5
Views
4K
skyQuake
skyQuake
markhocky
Review: Trading Systems (a new approach...) - Jaekle and Tomasini
Replies
0
Views
5K
markhocky
markhocky
Top