System Robustness

[theasxgorilla]

But with that said -- what do you think about GPs results in which case the test over the ASX300 outperforms the one over the whole market?

Certainly, historical testing with a universe like the XAO or the ASX300 or ASX200 must be assumed to have some amount of survivorship bias. Interestingly, the ASX300 outperforms the XAO over the same 10 period, period. 9.24% to 8.82%. But I dont think this accounts for that big of a difference in the results

Unfortunately no one seems to have comprehensive data AND even if we had comprehensive data, we need to make the universe dynamic to test accurately. So, in lieu of all the aforementioned, IMO, the best we can do is use a random entry/exit system as a bench mark and aim to beat it. As I said initially, if your system can't beat a system that has no edge when tested on the same survivorship biased data, then your system also has no or a negative edge.

 

[R0n1n]

Here is the testing of my long term system on the Nasdaq 100. Your comments please...
I have attached the testing report as a spreadsheet as its much easier to read.

 

[nizar]

Here is the testing of my long term system on the Nasdaq 100. Yopur comments please...
I have attached the testing report as a spreadsheet as its much easier to read.

Hi R0n1n,

Looks the goods to me.
28%pa is nothing to scoff at.
Only 82 trades in 10 years, which is mad.
Your not doing much for 28%pa!!
Average holding time almost 2.5years for the winners.
50% winners is nice, but i would imagine R/R would be something special

Max.DD is 52%, but it looks like this is only through 1 run.

Have you done a montecarlo analysis on this system and how is it looking?

 

[R0n1n]

Hi R0n1n,

Looks the goods to me.
28%pa is nothing to scoff at.
Only 82 trades in 10 years, which is mad.
Your not doing much for 28%pa!!
Average holding time almost 2.5years for the winners.
50% winners is nice, but i would imagine R/R would be something special

Max.DD is 52%, but it looks like this is only through 1 run.

Have you done a montecarlo analysis on this system and how is it looking?

thanx for the review Nizar. I plan this system to be fairly long term and the pne I posted the results before to be short to medium term.
This is just the first version of the system, with heaps of modifications ( I am upto mod version S) . I haven't done a montecarlo analysis on it yet but plan to do it shortly.

 

[stevo]

Ronin
28% annualised gain sounds good but with 52% drawdown I personally would find it hard to trade - but we all have our own parameters and limitations that shape the way we trade. That's what makes the market!

Some thoughts;

One of the problems of comparing / testing systems is that position size can make such a big difference to results. For example tech's system had maximum number of open trades set to 100. So there is not really any limit to the number of trades that would be taken, pyramiding is set to "Yes" and position sizing is 2% risk. If trades were limited to 10 the results could be totally different. Not saying that there is anything wrong with tech's test. The number of positions would probably be self limiting, although since pyramiding is on it may be possible to have 20 plus trades at any one time even with 14.5% position size limit.

But the point is that position sizing alone can easily double / half returns and drawdown. So comparing systems can be difficult unless position sizing is kept constant for comparison purposes. One can tweak a system substantially playing with position sizing, as many of us have already discovered.

With AB I use an #include<TestParams.afl> statement and put position sizing strategy in a separate AFL file so I can be sure the position sizing strategy doesn't vary.

I really like the concept of a random system as a benchmark - if a system isn't better than random why trade it? Also the random system approach shows the natural bias of the markets over the last 10 years or so - and probably much longer.

A trader can use this natural bias in their favour. Adding some simple things, like ASX.G's simple momentum filter, or maybe a more dynamic stop, can narrow the range and slide the results up the X axis.

The idea of not optimising at all seems quite strange to me, but then it is probably easier to do in AmiBroker than Metastock. Everyone optimises, even by just looking at a chart and picking the values that they think will work best. ASX.G's already mentioned the power of AmiBroker's 3D optimisation charts. Walk forward testing is also a good approach to see how a system performs out of sample - it's even more valid when the walk forward is for real!

I usually test in AmiBroker using random entry and exit prices. Open price trades can be reasonably difficult to achieve, especially as trade size grows.

The last month has been a pretty good test of any long only system, as well as a humbling experience for us overconfident, over-optimised, mechanical system traders! A 10% to 20% drawdown doesn't look like much on paper , but give back thousands of dollars and you get a different perspective. I just think of the trades I will make over the next 20 years and everything falls into place.

regards
stevo

 

[It's Snake Pliskin]

Radge keeps saying it.

The solution is simple.
Know WHY your system returns profit.

Know why it works as a trend following system
OR
Know why it works as a short system
OR
Know why it works as a forward and reverse system
OR
Know why it works as a combination of systems.

Ever thought of investing some $$s in Radges "Building a System" course?

Yes good points tech.
Have you done his course?

 

[tech/a]

Yes good points tech.
Have you done his course?

No.

 

[R0n1n]

With AB I use an #include<TestParams.afl> statement and put position sizing strategy in a separate AFL file so I can be sure the position sizing strategy doesn't vary.

Stevo, very valid points. All taken onboard.

Can you out a sample position sizing AFL here or PM me please, it will save me some time in knocking one up( I am still learning AFL)

A few questions:

1) I'll probably start a thread on optimization, but how much should one optimise ? Will over optimization turn into curve fitting eventually ?

2) Once you have tested a system and done Monte Carlo analysis and are happy with it, do you just start trading it or you paper trade it in real time to see how it performs ? When do you switch it to production ?

3) With one of my systems I like to see the charts and the scans in weekly format as well. I use the code below to swith it to weekly (its Amibroker)

TimeFrame = Param("Weekly Timeframe? Y=1",0,0,1,1);

if( TimeFrame )	// switch to weekly
{
TimeFrameSet( inWeekly );
}

Is this right for backtesting as well ? I mean are the backtesting results are that of a weekly system ? (this code is at the very beginning of the system.)

4) Is it better to have multiple systems (for example a longterm, a short term CFD, a short term stock system) or just one system and tweak it according to times.

cheers,

Ronin.

 

[Temjin]

I've completely missed this thread, then it goes too long and I haven't had time to read through it.

A few questions:

1) I'll probably start a thread on optimization, but how much should one optimise ? Will over optimization turn into curve fitting eventually ?

Over optimisation = curve fitting. If you are getting unrealistic results for a specific set of data, and then get another set of results (and worst) on an off-sample data, then you have indeed over optimised your system.

How much one should optimise their system? The less optimisation you need to do, and the more robust the results are, the better.

2) Once you have tested a system and done Monte Carlo analysis and are happy with it, do you just start trading it or you paper trade it in real time to see how it performs ? When do you switch it to production ?

As a guide, you should either paper trade it first or UNDER trade for a few months to see if your system performs within your tested parameters. Then switch to full position sizing production.

4) Is it better to have multiple systems (for example a longterm, a short term CFD, a short term stock system) or just one system and tweak it according to times.

It is ALWAYS BETTER to have as many uncorrelated systems in your portfolio as possible. However, be aware of all trending systems tend to correlate with each other over time. Counter-trend systems are less correlated with the trending systems.

Remember all non-adaptive, parametric only systems will eventually fail over time. Unless you have a well developed plan and a good tweaking strategy that "re-adapt" your single system to current market conditions, you are best off to have all your systems to adapt itself automatically.

 

[Temjin]

Ronin
One of the problems of comparing / testing systems is that position size can make such a big difference to results......

But the point is that position sizing alone can easily double / half returns and drawdown. So comparing systems can be difficult unless position sizing is kept constant for comparison purposes. One can tweak a system substantially playing with position sizing, as many of us have already discovered.

You are completely right here.

Personally, I treat position sizing as an "adjustable power dial/knob" that can both proportionally increase both the return and risk of a system when need to be. If a certain power level of a system produced a historically tested risk that is unacceptable to the user, then he/she should reduce it proportionally. If the return at this acceptable risk level is unacceptable to the user, then he/she should further improve his system by either having better entries / exits (mainly exits), or be creative in money management techniques and even with the power dial itself.

Systems should be compared based on their return/risk ratio. There are many definition of return and risk, so how you define it is up to you.

 

[stevo]

Stevo, very valid points. All taken onboard.

Can you out a sample position sizing AFL here or PM me please, it will save me some time in knocking one up( I am still learning AFL)

A few questions:

1) I'll probably start a thread on optimization, but how much should one optimise ? Will over optimization turn into curve fitting eventually ?

2) Once you have tested a system and done Monte Carlo analysis and are happy with it, do you just start trading it or you paper trade it in real time to see how it performs ? When do you switch it to production ?

3) With one of my systems I like to see the charts and the scans in weekly format as well. I use the code below to swith it to weekly (its Amibroker)

TimeFrame = Param("Weekly Timeframe? Y=1",0,0,1,1);

if( TimeFrame )	// switch to weekly
{
TimeFrameSet( inWeekly );
}

Is this right for backtesting as well ? I mean are the backtesting results are that of a weekly system ? (this code is at the very beginning of the system.)

4) Is it better to have multiple systems (for example a longterm, a short term CFD, a short term stock system) or just one system and tweak it according to times.

cheers,

Ronin.

Have a look at http://www.amibroker.com/library/detail.php?id=545 for an idea of the components to use in backtesting.

Curve fitting is more likely to occur if you don't have enough data. By portfolio testing over a large number of stocks the data issue becomes less of an issue. Monte Carlo analysis also helps by showing the range of possible results, rather than just one good run.

I am hopeless at paper trading so I just trade real money to forward test a system. But I need to strongly believe that it will work before I put money on it - and I also need to be sure that I haven't done something dumb when backtesting.

I stay away from daily charts so I don't switch timeframes.

Multiple systems seem to be a good idea and the basic theory goes along the lines of having various uncorrelated systems working at the one time - just as long as they all make money over the long run. The problem is getting multiple systems that you would be happy to trade. I would struggle trading a short term system, yet others couldn't handle a long term system.

stevo

 

[nizar]

If you want to test a system over different timeframes, can you just go to the "Trade database Manager" window in TradeSim and adjust the dates at the bottom of the screen and run the simulation again?

OR

Do you have to run a separate metastock exploration each time you want to change the dates by adjusting this function:

ExtFml("Tradesim.SetStartRecordDate",1,09,1998);
ExtFml("Tradesim.SetStopRecordDate",19,05,2006);

Any ideas would be much appreciated.

 

[tech/a]

Change the dates as you have highlighted and run Metastock again.
That will create a database on those dates.

Ive often thought but not tried starting and stopping on different multiple dates.
IE dont trade May each year.
Sorry just thinking.

 

[stevo]

Ive often thought but not tried starting and stopping on different multiple dates.
IE dont trade May each year.
Sorry just thinking.

Good idea - take a break for one month a year!

stevo

 

[tech/a]

Here is the testing of my long term system on the Nasdaq 100. Your comments please...

Had a chance to comment.
Would like a Montecarlo sim on both.
As the Buy and hold would be dependant on what it was you bought and held?
96 Trades for buy and hold.
I'm not understanding.Once you buy you hold so how does a portfolio have 96trades?
Whats your position sizing model?

I'm sure all will be clearer once you explain.
I'm a bit thick!

 

[nizar]

R0n1n.

Does the equity curve for "buy and hold" mean the index??

 

[R0n1n]

sorry for the late reply guys. Here is the discription of Buy & Hold, it should clear some of the confusion that I caused:

The Buy & Hold Equity is represented as a blue line. The line is calculated by taking an equally-sized Position in each symbol of the WatchList at the start of the $imulation period, and holding the Positions until the end of the period. The $imulator uses real-world rules for even the Buy & Hold Positions. It bases the size of the Positions on the closing value of the first bar of data, and opens the Positions at the opening price of the next bar. Because of this you'll notice that the $imulator's Buy & Hold Exposure level is usually never exactly 100%, but is typically within 1% of 100% (no margin assumed).

The $imulator does not close Buy & Hold positions. The outstanding profit is based on closing price of the last bar, but no exit commission is applied.

 

[tech/a]

Ron.

Thanks
But I still dont get it.
If the buy and hold only took the same number of positions which were triggered in your system then OK.
But it took more?
I presume also that it takes positions in every stock (as a comparison) to those in your portfolio everytime a buy occurs.Even though you stop out or exit the buy and hold continues.
While I understand the comparison its practical application in the real world is not possible due to lack of capital.You couldnt replicate the results.You could only buy and hold the first X positions then your fully capitalised.

Yes I understand its a benchmarking method!
But----As such to me its of little value.

 

[Nick Radge]

I just think of the trades I will make over the next 20 years and everything falls into place.

If you wish to be a successful trader:

(1) Design a robust trend following system,
(2) Control the risk,
(3) heed Stevo's words of wisdom above.


"Next 1000 trades" is my mantra. Tattoo it on your forehead.

In my experience many people have held the "holy grail" without knowing it. They discard it at the first hiccup. This recent correction will be a great example of people who perhaps gave back some open profits and feel a little pained by it. The pain will make them look elsewhere for something that doesn't offer pain. Of course all trading systems will induce some level of pain. What makes a successful trader is one that accepts some pain but totally understands that over the longer term the positive expectancy will continue to roll on unabated. Control the pain by controlling risk, then allow "time in" the system to work its magic.

Snake,
My course does not cover this stuff. It's all about "discretionary" trading. I have never put anything down on system trading.

 

[howardbandy]

Greetings --

I have written one book about trading system design, testing, and validation -- Quantitative Trading Systems -- and have another planned -- Trading System Validation. I've read through the thread to date, and I'd like to add my two cents worth to the discussion of robustness.

1. In order to measure robustness, there must be a metric, often called an objective function. The metric is personal -- yours will probably be different than mine -- and it must be designed or chosen before the model development process begins in earnest. Some people use compound annual rate of return, others have quite complex objective functions that include terms for trading frequency, annual rate of return, drawdown, holding period, and so forth. The sensitivity of parameters to specific data can be addressed by perturbing the parameter values, measuring the associated objective value, and averaging. Sensitivity to outliers can be addressed by limiting the profit or loss associated with them. These, and other Monte Carlo techniques can be incorporated into the objective function.

There is a very important point about the objective function -- if two trading systems or alternatives are compared over a given set of data, the one that scores highest according to the objective function must be the one that the person developing the system prefers. If that is not the case, then there must be some factor that has not yet been included in the objective function. That factor must be identified, quantified, and incorporated into the objective function before proceeding. Further along in the trading system development process, we will not be able to see all the alternatives -- only the one alternative that scores highest will be used.

2. The data being used must be divided into at least two data sets -- an in-sample data set that is used to select the parameters for the trading system, and an out-of-sample data set that is used very infrequently -- preferably exactly one time -- to measure the performance of the system on data that has never been seen before. If the trading system is adjusted based on the results of the out-of-sample data, then that previously out-of-sample data has just become in-sample data, and a new out-of-sample data set must be used for validation.

3. Search through the in-sample data set as much as you wish. Look for values of the parameters that maximize the value of the objective function; add rules and filters. When you are satisfied with the in-sample results, test on the out-of-sample data. If the out-of-sample results are satisfactory, you have an indication that the system might be robust and might be tradable.

4. To increase your level of confidence, perform the walk-forward test. That is, select a length of time to use as the in-sample length and a length of time to use as the out-of-sample length. The only arrangement of the two periods that is practical is to have the out-of-sample immediately follow the in-sample. Start far enough ago that there is room for several out-of-sample periods before today. Beginning with the first in-sample period (the oldest in-sample period), find the optimum parameters -- optimum is always determined by the highest value of the objective function -- then test over the associated out-of-sample period, and record the results. Move on to the second in-sample period, optimize, test the second out-of-sample period, and record the results. After all the in-sample periods have been processed, concatenate the results from the out-of-sample periods. If those results are satisfactory, your confidence is increased.

5. Contrary to popular belief, there is no statistic or other measurement that can be taken from the in-sample results that give any indication of the likelihood of a system to be profitable in the future. Thirty closed trades is not enough -- 30,000 closed in-sample trades is not enough. The results achieved over the in-sample period are meaningless.

6. I'll say that again -- the result achieved over the in-sample period are meaningless. They are always good. We do not stop searching until they are good.

7. There is no guarantee that any trading system will be profitable in the future. The best we can hope for is a high degree of confidence. The only way to build that confidence is by repeating the optimization - out-of-sample testing process.

8. If you do not have acceptable test results on a truly out-of-sample data period, then out-of-sample testing begins tomorrow with real money.

Thanks for listening,
Howard
www.quantitativetradingsystems.com