Backtesting Advice

[Chorlton]

Hello All,


I'm currently trying to develop a Weekly LONG System for trading the UK Market (FTSE100), and would now like to begin initially backtesting some of my ideas.

The EOD data which I want to test is between 1999 and the Present Day.

Looking at the FTSE100 Index over this same period:-

Year 1999 & 2000, the index remained in a Range.
Year 2001 & 2002, the index fell sharply, and from Year 2003 onwards, the index has steadily risen.

I'm also aware that ideally I should have both an in-sample and out-sample for Backtesting purposes.

As a result my current assumption is:

1. To use the last 2-3 years (2005 onwards) for the out-sample, as these are the most recent years and therefore should be fairly close to the results of the system, when it is initially traded live.

With the remaining 6 years (1999-2004), I was going to break these down and test them in "2 year" blocks.

1) 1999 & 2000 : To assess how the system preforms during a Ranging Period.
2) 2001 & 2002 : To assess how the system performs during a BEAR Market
3) 2003 & 2004 : To assess how the system performs during a BULL Market.

Assuming each of the above tests delivers results which are deemed acceptable, I would then go on to backtest the system again using ALL the in-sample data (1999-2004).

If the results of this test also proved acceptable, I would then use the Out-Sample Data.

However, as this is my first system, I would welcome any comments from the more experienced System Developers on whether the above is the best method for achieving a reliable backtest result.

All comments welcome.....

Kind Regards,

Chorlton

PS. I've also posted this question on another forum, as I would to get as many comments as possible before decided on my final backtesting "method"

 

[tech/a]

Chorlton

I'll answer on this one.

Firstly and very importantly a longterm weekly system needs as much data as possible to test on.
A 2 yr period is far to short.
The system is likely to hold trades for over a year so be very careful as your result may show a less than average closed profit result with heaps of profit in OPEN trades.
So be sure to close all open trades at the end of the test period.

If your designing this syetem expecting it to perform in less than ideal conditions is un realistic.

I can tell you now that it will perform mediocre in a ranging market.
Terribly in a bear market and brilliantly in a bull market.

So take the longest period possible.
Then expect the results in various markets to be as above.

 

[Chorlton]

Chorlton

I'll answer on this one.

Firstly and very importantly a longterm weekly system needs as much data as possible to test on.
A 2 yr period is far to short.
The system is likely to hold trades for over a year so be very careful as your result may show a less than average closed profit result with heaps of profit in OPEN trades.
So be sure to close all open trades at the end of the test period.

If your designing this syetem expecting it to perform in less than ideal conditions is un realistic.

I can tell you now that it will perform mediocre in a ranging market.
Terribly in a bear market and brilliantly in a bull market.

So take the longest period possible.
Then expect the results in various markets to be as above.

Hi Tech,

You were one of the posters which I was hoping would post given your past experience & previous comments so Many Thanks.....

Even though my system is designed to trade LONG, I was keen to see how it would perform in other market conditions. I don't expect to make much of a profit during these periods (if any at all!!!!) BUT I do want to ensure that losses are kept to a minimum so hence my testing over these periods.

I fully accept your point about needing as much data as possible. My intention was to run seperate tests over 2 year periods, AND then to run a test over the WHOLE testing period which is 6 years.

As a result, are you suggesting that I simply ignore the smaller period tests and simply concentrate on the whole 6 year period??

So be sure to close all open trades at the end of the test period.

Ah... Good Point !! I need to take a look and see if AB does this.....


Kind Regards,

Chorlton

 

[tech/a]

A 2 yr period is far to short.

Enjoy the journey.

 

[nizar]

Hi Chorlton,

I agree with tech/a.

Use the full 6-year period first as long term systems aren't designed to show results in the short term.

Then if the initial testing is good, you can then manipulate the dates to overcome start and end date biases.

Remember to use SetTriggerAtDate in TradeSim to close out all open trades at any date you want.

I have wrote some stuff on testing on this post of my blog

 

[Chorlton]

Hi Chorlton,

I agree with tech/a.

Use the full 6-year period first as long term systems aren't designed to show results in the short term.

Then if the initial testing is good, you can then manipulate the dates to overcome start and end date biases.

Remember to use SetTriggerAtDate in TradeSim to close out all open trades at any date you want.

I have wrote some stuff on testing on this post of my blog

Thanks for your comments Nizar and I shall take another look at your blog

I've just managed to get the AB Trade Data into TradeSim (Thanks to some code posted) but the results vary dramatically from those in AB and the Distribution Charts in TradeSim definately don't look right....

In answer to Tech's last comment, I'm definately not "enjoying the journey" at the moment. The trading stuff is really interesting but the IT side is getting me frustrated

you can then manipulate the dates to overcome start and end date biases.

Any chance you can explain what you mean here? If it's covered on your blog, then there's no need as I'll definately be taking a look

Remember to use SetTriggerAtDate in TradeSim to close out all open trades at any date you want.

I'll take a look at this function in Tradesim but would I be right in thinking I just add the code to my system code in AB prior to exporting?

Thanks in advance,

PS. How's the intraday ideas coming along???

 

[howardbandy]

Hi Chorlton --

The techniques for testing and validating a system that uses weekly bars is no different than for one that uses daily bars, monthly bars, or hourly bars. Divide the data into in-sample and out-of-sample periods and walk forward. Evaluate the out-of-sample results to determine whether the system is sufficiently stable and profitable to use.

The difficulty with weekly bars is that there will be many fewer data points.

And, as always, beware of evaluating long-only systems in rising markets. Pay particular attention to the results for the out-of-sample periods that are declining in price. These will tell you how your system will react when the next market drop occurs.

Thanks,
Howard

 

[Sir Burr]

I've just managed to get the AB Trade Data into TradeSim (Thanks to some code posted) but the results vary dramatically from those in AB and the Distribution Charts in TradeSim definately don't look right....

Chorton,

Tradesim and Amibroker may do some things differently. One thing is Amibroker uses an open trades equity curve and Tradesim closed.

Other things maybe differences in trade timing caused by Amibroker coding for which Tradesim has no option. I export both the Amibroker (Backtest results) and Tradesim (Trade Log) to a csv file and compare each trade to see what's is going on if the results are different between the two.

I'd really try and get this worked out as Tradesim craps on Amibroker for backtesting (MC) speed and built-in reports. Play with Amibroker for any optimising.

SB

 

[stevo]

Chorton
Lots of good comments have already been made, but I have a couple of points to add anyway.

Just because a system uses weekly data doesn't necessarily mean that the system is long term. Have a look at average trade lengths for all trades, as well as the trade lengths for winners and losers before considering the length of the in sample and out of sample testing strategy.

I have seen quite good weekly systems with quite short hold periods (<10 weeks) and a couple of years data would be be ok for these sort of short term weekly systems. But if the average trade length is 52 weeks then 2 years is not a lot.

Also are you only testing 100 stocks (FTSE100)? Are you testing the current stocks in the index back in 1999 / 2000? If so I would consider broadening out the number of stocks you are testing. If I optimise on the ASX100 I will test on the ASX500 or All Ords Index (or even broader) to see what happens.

If you want to have a bit of fun pick the 100 worst stocks over a 2 year period and, once you have developed your system, see how it goes on the worst 100 stocks. I wouldn't expect a long only system to perform well on the worst performing stocks, but its worthwhile looking at how much a system loses.

Basically try to make the system fail and see how it performs. Often our expectations are outside the realms of possibility. Backtesting gives us a sense for what is possible.

If you are using AB to Tradesim double check the trades by stepping through a number of them using the Bar Replay feature. Also check that the trade delays line up correctly - I usually use a one bar delay - and that the dates make sense between AB & TS.

Then double check all the code that you are using to make sure that you are not doing something silly, I can be easily confused!!! Watch out for things like using PositionScore without a delay so that the positionscore is based on the end of the week but the buy is made on the open. Little things like this can have a big impact and give the wrong picture.

stevo

 

[nizar]

Hello All,


I'm currently trying to develop a Weekly LONG System for trading the UK Market (FTSE100), and would now like to begin initially backtesting some of my ideas.

I've just noticed this point as it was highlighted by Stevo.

Can I ask why you want to limit your universe to the FTSE100?

What this is effectively doing is telling your system that it can ONLY buy stocks that are ALREADY in the top 100.

In my opinion, a long term trend following system should also attempt to catch the big trends that will be a top100 stock in the future.

And also, a system which uses the top100 stocks as its universe cannot be backtested accurately in my opinion due to survivorship bias. This is harder to overcome and becomes more significant the further back you go.

I suggest to test over the whole market with a liquidity filter. This is what some traders suggested to me when I was trying to look for historical index constituents (no data provider has this).

The answer to your first question as to what start and end data biases means is explained in my blog.

And as for the SetTriggerAtDate function, look on TradeSim manual pg.108.

Im not familiar with AB but for me I added this condition to my exit trigger.

ExitTrigger:=YourExitTrigger OR ExtFml( "TradeSim.SetTriggerAtDate",dd,mm,yyyy);

 

[Chorlton]

Gents,

Just wanted to quickly say Thanks for all your comments and observations.

I would like to reply in more detail to some of the points raised but I'm currently busy with non-trading activities

However, as soon as I get a spare minute, I share post again...

Chorlton