Help me design a backtesting system

[Sunder]

Hi guys,

I thought I'd try learning how to effectively back test. After 9 months on and off of weekends writing my own software (okay, so total investment of time was probably close to 100-120 hours)... I found Amibroker, which did about 95% of what I wanted... Talk about frustrating.

I have been playing around with it all day as I have been sick with a flu (not sure if it's normal flu or swine flu, but my fiance thinks it's man-flu) and I have been very surprised by some of the results.

None of my results have consistently returned above a "buy and hold" strategy so far, but the biggest surprise is that some trading strategies that have returned 65% right on some shares, will return 35% right on others. (e.g. one iteration of a DMS returned about 11% annualised returns on BHP, but lost 5% on AMC over the same period. I'm guessing because BHP moves in long slow trends, but AMC was a lot more volatile)

And some strategies thatt are returning 35% right, are making more profit than other strategies that are 65% right, simply because the losses are 1% and the gains are 12%.

I'm right now "brute forcing" this. Trying out as many systems as I can over as many shares as I can. But I know this is not an thorough way of doing things.

Can anyone give some tips on how to design an effective and systematic back testing system?

Appreciate any feedback on either backtesting or trading systems.

 

[Mr J]

Are the samples statistically significant?

 

[Sunder]

I have data going back to January 2000.

That should be statistically signficant enough.

 

[tech/a]

Buy Howard Bandy's books.

 

[beamstas]

As said above
I'd go with Quantitative Trading Systems. It's a great book and well worth the money.

What do you mean by "65% right?"

Another thing, 9 years might seem statistically correct, but when designing a system you should make it for say 3 of those years, and then once it's done, test it for the 3 years previous and the 3 years after your sample period.

I'm feeling it's a bit of a waste of time trying to write all this down though.. As bandy has everything you'll need to teach you amibroker and then BLOW your mind with even more knowledge than you'll ever need (well for me anyway )

Brad

 

[MichaelD]

Can anyone give some tips on how to design an effective and systematic back testing system?

Yes. Throw out everything you ever "knew" to be right about trading systems. As you are discovering, pretty much everything you thought was right was in fact wrong. (viz - lower win rate systems perform better than higher win rate systems). You shouldn't underestimate the value of what this is teaching you about profitable trading.

Try starting with random entry and working on the exits and money management. Once you're happy with the exit and money management performance you can then finesse the entry - you'll be suprised by how little difference the entry makes versus the effect of your exit/money management.

 

[It's Snake Pliskin]

Buy Howard Bandy's books.

Welcome back tech/a!

I see Mr bandy will be in Melbourne in October.

 

[Sunder]

As said above
I'd go with Quantitative Trading Systems. It's a great book and well worth the money.

What do you mean by "65% right?"

I should have said 65% profitable trades.

I'll look for the book next time I drop into a book store.

Yes. Throw out everything you ever "knew" to be right about trading systems. As you are discovering, pretty much everything you thought was right was in fact wrong. (viz - lower win rate systems perform better than higher win rate systems). You shouldn't underestimate the value of what this is teaching you about profitable trading.

Try starting with random entry and working on the exits and money management. Once you're happy with the exit and money management performance you can then finesse the entry - you'll be suprised by how little difference the entry makes versus the effect of your exit/money management.

I've been finding this an "almost" universal rule, and much of my effort has been trying to either reduce false entry signals, or else tighten up the stop loss without creating false exit signals. Am I wasting my time with these efforts?

 

[tech/a]

I should have said 65% profitable trades.

I'll look for the book next time I drop into a book store.

Not in Bookstores.

http://www.quantitativetradingsystems.com/

I've been finding this an "almost" universal rule, and much of my effort has been trying to either reduce false entry signals, or else tighten up the stop loss without creating false exit signals. Am I wasting my time with these efforts?

Yes

 

[MichaelD]

much of my effort has been trying to either reduce false entry signals

Not a waste, but not much reward for effort in this area.

or else tighten up the stop loss without creating false exit signals

Why not try the opposite?

 

[Sunder]

I think one of the biggest difficulties I'm having is that to eliminate false starts and false exits, I'm entering too late into the trend, and exiting too late as well, which means it only works on low volatility stocks and indices.

I read elsewhere that a good stock trading system should work on all stocks, and should be fairly resilient to change. My "best" outcome so far is a 15% PA gain from Jan 2000 to present, but it doesn't work on about 2/3rds of the ASX200, and adjusting my timing from 50 day to 40 or 60 day, result in outcomes closer to the 12% mark, which means if the market changes on me, it could stop working.

I think I'm going to do a lot more reading of these threads, and come back and ask more specific and intelligent questions.

 

[skyQuake]

I think one of the biggest difficulties I'm having is that to eliminate false starts and false exits, I'm entering too late into the trend, and exiting too late as well, which means it only works on low volatility stocks and indices.

I read elsewhere that a good stock trading system should work on all stocks, and should be fairly resilient to change. My "best" outcome so far is a 15% PA gain from Jan 2000 to present, but it doesn't work on about 2/3rds of the ASX200, and adjusting my timing from 50 day to 40 or 60 day, result in outcomes closer to the 12% mark, which means if the market changes on me, it could stop working.

I think I'm going to do a lot more reading of these threads, and come back and ask more specific and intelligent questions.

Disagree with the working on ALL stocks part. E.g CBA and BBI are two VERY different stocks, traded by a very different group of people and would thus have little in common. Volatility and 'speculativeness' of a stock will greatly undermine your efforts.

 

[Sunder]

Disagree with the working on ALL stocks part. E.g CBA and BBI are two VERY different stocks, traded by a very different group of people and would thus have little in common. Volatility and 'speculativeness' of a stock will greatly undermine your efforts.

So what would be a good way to test whether a stock is suitable for a system? And is that stock likely to change while the system is in use?

 

[MichaelD]

So what would be a good way to test whether a stock is suitable for a system? And is that stock likely to change while the system is in use?

Don't let the bad advice of others cloud your learning experience.

A robust system will work across a range of instruments, as all it is required to do is to cut your losses short and let your profits run.

There are 101 ways to achieve this. Unfortunately, there are 1,000,001 ways to do the opposite, which isn't robust.

 

[awg]

So what would be a good way to test whether a stock is suitable for a system? And is that stock likely to change while the system is in use?

Would it be any use to populate your backtests with stocks from each sector?

(thats me asking a question btw)

The Financial and Materials sector behave rather differently, for instance.

You could try and optimise yr criteria for different sectors?

have you searched some of the older threads on system design etc?