Inspired For Backtesting

[MichaelD]

How long did it take you to develop systems which perform better than 30%?

There's more to trading profitably as a business than just the basic entry/exit system. There's also the money managment side of the business, and the use of other people's money (later) to boost returns. It's also fallacious to look at system return as a % as the return will vary depending on market conditions. It's more important to consider system drawdown when designing a system.

It took me 6 months of fairly solid system development before I settled on my current robust long term trend following system.

I'm now 3 months into developing a short term breakout system and am just starting to see robust results from this effort.

My edge is pretty poor at around 11 or 12%. What made a difference? Wider market filters or exits, etc?

Depends on your time frame but if you target long term trend following as the best place to start developing a system, then you should start with Random Entry and a wide trailing stop (eg wide ATR stop or long term moving average stop) and go from there.

Once you've established an exit with random entry that outperforms buy and hold then you can refine the system from there. Don't be too quick to whack all sorts of entry filters into a system - they'll often do more harm than good.

 

[weird]

Michael, do you see any issues with a "Random Entry", and comparing backtesting results ?

 

[nizar]

The fact that random entry works speaks volumes about what is required to develop a profitable trading system.

Money management ie. position sizing, risk management ie. proficient use of stops, and exits, are the most important components of a trading system. Entry probably the least important, yet everyone focuses on the perfect entry.

Notice how the most used trading cliche: cut your losses fast and let your winners run, is all to do with the exit.

Bin57 - Im still developing a trading system. Started really about a month ago. Last months results were 38% winners over 29 trades. R/R=2.43. Still alot of work to do. My main weakness at this stage (one of several) is not really knowing when to move up my stop and to what level. This part is too discretionary for my liking, but maybe it has to be. Because of this flaw, At times im allowing for far too much profit giveback. Like i said, still alot of work to be done.

As for books, tech has mentioned a couple:
*The trading game: Ryan Jones (1999)
*Stock patterns for daytrading: Barry Rudd (1999)
*Trading systems and methods: Perry Kaufman (1998)

 

[MichaelD]

Michael, do you see any issues with a "Random Entry", and comparing backtesting results ?

No, there are no issues with a random entry. It should be considered the "gold standard" when developing a system. MOST systems will not perform as well as a system based on random entry, a powerful indicator of where system development effort should be concentrated. If you can improve upon random entry, you are doing well, but you need to have these results as a comparison baseline to know if you are doing good or harm with your extra filters.

A couple of points worth mentioning, however, are;

1. With random entry you'll get more entry signals than you have capital to trade, so it is essential to apply Monte Carlo analysis to get a distribution of results rather than a single figure for a system.

2. When comparing system results, you should not focus on % profit, but % drawdown as the important comparator. i.e. if two systems have much the same profit % but one has significantly less drawdown %, then the lower drawdown system is superior. One very useful technique is to normalize all system results to the same drawdown % and then compare profit % to get an idea of which is the best system.

 

[weird]

I would think with a random entry, it could be difficult to determine a sample size required for significance in statistical results.

I would be interested if anyone wishes to calculate the possible entry combinations when using a random entry , picking any stock from the ASX300, for a testing period of 5 years, using daily bars?

 

[stevo]

Bin
Good questions. I can give some ideas but really it is a matter of testing anything that appears to make sense, and some things that don't.

Some ideas;
1. Have you looked at different time frames?
2. Try using an index to determine times when it could be good to stand aside.
3. As an exercise look at a monthly time frame. Try something like enter when the stochastic goes above 50 and exit when it drops below 50. Add some things and see what happens. Try it with RSI, Bollinger bands, moving averages. Do the same in weekly and daily!
4. Look at ATR trailing stop on a weekly time frame.
5. Consider what sort of filter could weed out the slow movers. If a stock has little channce of moving at least 20 to 30% in 3 to 6 months then why trade it?
6. Consider position sizing. I am going to get much better test on a decent system if I use 10% of capital position sizing (ie pyramiding profits) versus a flat rate position sizing strategy - like $100,000 a trade. Consider fixed risk position sizing (1% or 2%) - this is what I use. A 10% return in one position sizing strategy might be a 20% plus return using another strategy.
7. To reduce trades look at increasing time frame.
8. I am not trying to beat buy and hold. I am trying to get a suitable return with reduced risk. Buy and hold is, for me, a very risky strategy. If I buy and hold the wrong stocks I am in serious trouble since I have no exit strategy.

I think entries are very important. Sure random entry can work with a suitable exit and position sizing, but a good entry will give the portfolio a bit more zing. But it is good to have a benchmark to beat. I did do some work with random exits and found that a good entry with a random exit beat a random entry system with a moving average exit.

http://drawdown.blogspot.com/2005/08/random-entries-and-exits.html
http://drawdown.blogspot.com/2005/08/random-entry.html

 

[nizar]

Bin
Good questions. I can give some ideas but really it is a matter of testing anything that appears to make sense, and some things that don't.

Some ideas;
1. Have you looked at different time frames?
2. Try using an index to determine times when it could be good to stand aside.
3. As an exercise look at a monthly time frame. Try something like enter when the stochastic goes above 50 and exit when it drops below 50. Add some things and see what happens. Try it with RSI, Bollinger bands, moving averages. Do the same in weekly and daily!
4. Look at ATR trailing stop on a weekly time frame.
5. Consider what sort of filter could weed out the slow movers. If a stock has little channce of moving at least 20 to 30% in 3 to 6 months then why trade it?
6. Consider position sizing. I am going to get much better test on a decent system if I use 10% of capital position sizing (ie pyramiding profits) versus a flat rate position sizing strategy - like $100,000 a trade. Consider fixed risk position sizing (1% or 2%) - this is what I use. A 10% return in one position sizing strategy might be a 20% plus return using another strategy.
7. To reduce trades look at increasing time frame.
8. I am not trying to beat buy and hold. I am trying to get a suitable return with reduced risk. Buy and hold is, for me, a very risky strategy. If I buy and hold the wrong stocks I am in serious trouble since I have no exit strategy.

I think entries are very important. Sure random entry can work with a suitable exit and position sizing, but a good entry will give the portfolio a bit more zing. But it is good to have a benchmark to beat. I did do some work with random exits and found that a good entry with a random exit beat a random entry system with a moving average exit.

http://drawdown.blogspot.com/2005/08/random-entries-and-exits.html
http://drawdown.blogspot.com/2005/08/random-entry.html

Great advice here from someone who really dominates the markets. Thanks Stevo.

 

[MichaelD]

I would think with a random entry, it could be difficult to determine a sample size required for significance in statistical results.

Excellent question.

My limited knowledge of stats leads me to believe that if you look at the averages of a Monto Carlo analysis and consider 3 standard deviations away from the averages that you'll have something like a 99% probability of results falling within these parameters.

 

[Bin57again]

Stevo
Yes, I have a good handle on those pointers but I like your attitude on point 8.
Michael - agree that drawdown is important. From this, can I ask you, Stevo, Nizar another question on drawdown please? Let's say I have a trend following system with annual return of 20%. Starting capital is $100,000 and max. drawdown for the year is $15,000. Standard error is $2000. Now, the important part - I'm comfortable with a max.drawdown of $20,000. Is this enough of a buffer? Should I really only trade a system that has a max.drawdown of 50-75% of my tolerance? Could anyone recommend some monte carlo software? I'm not aware Amibroker has such a feature (is there anything which can be used in conjunction)?

 

[retroaugogo]

Hi Bin,

Here's some software that could interest you.

Rather than think of a system and then test to see if it works Stratasearch finds the best system for the data it is given.
Also has monte carlo simulations.

Haven't used it live yet but it's given me some nice numbers.

They have a 30 day trial.

If you end up using the program let me know.
Maybe we can swap notes.

 

[stevo]

Stevo
I'm comfortable with a max.drawdown of $20,000. Is this enough of a buffer? Should I really only trade a system that has a max.drawdown of 50-75% of my tolerance? Could anyone recommend some monte carlo software? I'm not aware Amibroker has such a feature (is there anything which can be used in conjunction)?

Only you can answer what your tolerance is and how much buffer you want just in case. Obviously we want to maximize profit and minimise DD - but we can only do what a system / market can supply.

You can do Monte Carlo in AB; http://drawdown.blogspot.com/2005/08/monte-carlo-with-ab.html

It's not as fast or flexible as TradeSim but with some basic Excel skills it can be done.

 

[MichaelD]

Let's say I have a trend following system with annual return of 20%. Starting capital is $100,000 and max. drawdown for the year is $15,000. Standard error is $2000. Now, the important part - I'm comfortable with a max.drawdown of $20,000. Is this enough of a buffer? Could anyone recommend some monte carlo software?

As Stevo has said, the max drawdown you are comfortable with is something only you know, however a suggestion - devise a plan with 1/2 the drawdown you anticipate is your maximum tolerated. It's easy to say you can tolerate a drawdown of $20,000, but when it actually starts to get close to that it's pretty hard to keep trading, and that's exactly the time when you should keep to the plan.

I personally use MetaStock and TradeSim for backtesting.

 

[Bin57again]

You can do Monte Carlo in AB; http://drawdown.blogspot.com/2005/08/monte-carlo-with-ab.html

It's not as fast or flexible as TradeSim but with some basic Excel skills it can be done.

Stevo
Did you create the charts in Excel as well?

Hi Bin,

Here's some software that could interest you.

Rather than think of a system and then test to see if it works Stratasearch finds the best system for the data it is given.
Also has monte carlo simulations.

Haven't used it live yet but it's given me some nice numbers.

They have a 30 day trial.

If you end up using the program let me know.
Maybe we can swap notes.

Retro
Thanks for that. I'm always interested in these kind of products - even if they turn out to be more glitz than profit value. I'm short of time right now but I'll try the trial period in the New Year and tell you what I think.
By the way, I'm sure most of the guys on here would warn you of parting with your cash on these kind of products (not that you're in any way naive) but maybe ask around - does anyone on this thread know/use this product?

 

[stevo]

Bin
The frequency charts were created in Excel using the Frequency statement. I used MIN and MAX to set up the bins. The slow part about using Amibroker and Excel is generating the portfolios in AB to transfer to excel.

 

[CanOz]

Chops
I don't know of any decent free software for portfolio backtesting - don't waste your time. EOD Amibroker is $149 US, and as tech mentions is very good.

If you are serious about backtesting I would go for AmiBroker. I also use Amibroker combined with TradeSim. Tradesim is excellent. You can do Monte Carlo analysis with Amibroker and Excel as well even though it's a little slow. If you don't already know how to code in Amibroker then you will need to spend some time understanding the software and language by running through the tutorials etc

Some suggestions and ideas to get you started;
1. For starters select a universe of stocks to test on, but don't start with too few - at least 400 to 500 stocks. I use the All Ords stocks for basic testing. You should expand/ alter the universe if you think you have a good system.

2. Don't get bogged down in one time frame. Do some simple tests on daily, weekly and monthly time frames to see which one is easiest to work with. Stick to trading long strategies for starters - you will get confused if you try testing shorting strategies initially.

3. Test the sorts of things that are pushed in trading books all the time. Try out MACD, RSI, Stochastics, Bollinger Bands Donchian channels etc using the methods outlined in trading books. Try the different time frames and see what you think of the results.

4. Try OB /OS style indictators but also reverse the strategy - buy when overbought, sell when oversold. As an example take bollinger bands in a weekly timeframe using the traditional OB/ OS approach then reverse it. Add a trailing stop.

5. Settle on a timeframe (daily / weekly / monthly) that you are most comfortable with for your first system.

6. Take some of the ideas that you have obtained from testing the standard indicators and start experimenting with trailing stops. Try moving averages, donchian channels, ATR trailing stops and anything else you want to throw into the pot.

7. Stick with simple position sizing strategies for starters - 10% of equity or $10,000 per position on $100,000 capital. You can expand on these strategies later.

8. Ideally keep track of the results obtained in a simple spreadsheet or even hardcopy printouts - giving enough detail so you can repeat the tests in the future.

9. Don't worry about walk-forward testing until you have something that you think has a chance.

10. The same goes for optimisation. Just stick to the usual periods touted for indicators, or if you want to change them use something like known number sequences for convenience - 1, 3, 5, 8, 13, 21, 34, 55 etc. Use any sequence you like - they are not magical.

11. Always delay your trades. For EOD trade the next day, not on the close of the current day, EOW trade in the next week not the close of the week, etc.

This will probably take around 6 months maybe longer depending on your skills with Amibroker. I really don't have any comprehensive references on backtesting other than maybe the TradeSim manual. Leon Wilson's book - Breakthrough Trading could be useful. Obviously the online help for Amibroker is essential and there is a lot of stuff on the Amibroker Yahoo site.

Also - get decent data! Free data is useless for serious portfolio backtesting.
regards

Ahhh, found what i was looking for, after i found my old post!

Thanks T/A and Stevo!

Cheers,


Steve

 

[nizar]

Thanks T/A and Stevo!

I second that.
Some of the posts in this thread are GOLD.
Really PRICELESS

 

[CanOz]

I second that.
Some of the posts in this thread are GOLD.
Really PRICELESS

Nizar, I'm clearly getting the impression that that there is allot of trial and error in developing the entry signals, just a combination of things and when it backtests well in terms of accuracy, then apply an even better system of money management to it. The exits could be, as MD says discretionary EOD.

I think its time i get myself Amibroker.

Cheers,

 

[nizar]

Nizar, I'm clearly getting the impression that that there is allot of trial and error in developing the entry signals, just a combination of things and when it backtests well in terms of accuracy, then apply an even better system of money management to it. The exits could be, as MD says discretionary EOD.

I think its time i get myself Amibroker.

Cheers,

Can.

I think the best thing to do initially is to test a system using random entry. Iv seen random entry do 30% a year over a long period of time. Then you can test for exits and money management. THen you can test through random exits, to find a good entry.