Inspired For Backtesting

I have decided to heed the advice of good old Rummy,

"Plan backwards as well as forward. Set objectives and trace back to see how to achieve them. You may find that no path can get you there. Plan forward to see where your steps will take you, which may not be clear or intuitive."

However, I'm hoping you can make things seem a little clearer for me than poor Donald did, and if you can offer me tips/ free software for backtesting, that would be great.

Cheers in advance,
Chops.

Best value for money is Amibroker.($400 approx)
I use Metastock coupled with Tradesim ($2,500 approx).
There is also Wealthlab ($900 approx).

Then look forward to around a year to master and a year to design test and implement a good trading methodology.While the path is long the learning curve will be vertical.

Chops
I don't know of any decent free software for portfolio backtesting - don't waste your time. EOD Amibroker is $149 US, and as tech mentions is very good.

If you are serious about backtesting I would go for AmiBroker. I also use Amibroker combined with TradeSim. Tradesim is excellent. You can do Monte Carlo analysis with Amibroker and Excel as well even though it's a little slow. If you don't already know how to code in Amibroker then you will need to spend some time understanding the software and language by running through the tutorials etc

Some suggestions and ideas to get you started;
1. For starters select a universe of stocks to test on, but don't start with too few - at least 400 to 500 stocks. I use the All Ords stocks for basic testing. You should expand/ alter the universe if you think you have a good system.

2. Don't get bogged down in one time frame. Do some simple tests on daily, weekly and monthly time frames to see which one is easiest to work with. Stick to trading long strategies for starters - you will get confused if you try testing shorting strategies initially.

3. Test the sorts of things that are pushed in trading books all the time. Try out MACD, RSI, Stochastics, Bollinger Bands Donchian channels etc using the methods outlined in trading books. Try the different time frames and see what you think of the results.

4. Try OB /OS style indictators but also reverse the strategy - buy when overbought, sell when oversold. As an example take bollinger bands in a weekly timeframe using the traditional OB/ OS approach then reverse it. Add a trailing stop.

5. Settle on a timeframe (daily / weekly / monthly) that you are most comfortable with for your first system.

6. Take some of the ideas that you have obtained from testing the standard indicators and start experimenting with trailing stops. Try moving averages, donchian channels, ATR trailing stops and anything else you want to throw into the pot.

7. Stick with simple position sizing strategies for starters - 10% of equity or $10,000 per position on $100,000 capital. You can expand on these strategies later.

8. Ideally keep track of the results obtained in a simple spreadsheet or even hardcopy printouts - giving enough detail so you can repeat the tests in the future.

9. Don't worry about walk-forward testing until you have something that you think has a chance.

10. The same goes for optimisation. Just stick to the usual periods touted for indicators, or if you want to change them use something like known number sequences for convenience - 1, 3, 5, 8, 13, 21, 34, 55 etc. Use any sequence you like - they are not magical.

11. Always delay your trades. For EOD trade the next day, not on the close of the current day, EOW trade in the next week not the close of the week, etc.

This will probably take around 6 months maybe longer depending on your skills with Amibroker. I really don't have any comprehensive references on backtesting other than maybe the TradeSim manual. Leon Wilson's book - Breakthrough Trading could be useful. Obviously the online help for Amibroker is essential and there is a lot of stuff on the Amibroker Yahoo site.

Also - get decent data! Free data is useless for serious portfolio backtesting.
regards

stevo said:

Chops
I don't know of any decent free software for portfolio backtesting - don't waste your time. EOD Amibroker is $149 US, and as tech mentions is very good.

If you are serious about backtesting I would go for AmiBroker. I also use Amibroker combined with TradeSim. Tradesim is excellent. You can do Monte Carlo analysis with Amibroker and Excel as well even though it's a little slow. If you don't already know how to code in Amibroker then you will need to spend some time understanding the software and language by running through the tutorials etc

Some suggestions and ideas to get you started;
1. For starters select a universe of stocks to test on, but don't start with too few - at least 400 to 500 stocks. I use the All Ords stocks for basic testing. You should expand/ alter the universe if you think you have a good system.

2. Don't get bogged down in one time frame. Do some simple tests on daily, weekly and monthly time frames to see which one is easiest to work with. Stick to trading long strategies for starters - you will get confused if you try testing shorting strategies initially.

3. Test the sorts of things that are pushed in trading books all the time. Try out MACD, RSI, Stochastics, Bollinger Bands Donchian channels etc using the methods outlined in trading books. Try the different time frames and see what you think of the results.

4. Try OB /OS style indictators but also reverse the strategy - buy when overbought, sell when oversold. As an example take bollinger bands in a weekly timeframe using the traditional OB/ OS approach then reverse it. Add a trailing stop.

5. Settle on a timeframe (daily / weekly / monthly) that you are most comfortable with for your first system.

6. Take some of the ideas that you have obtained from testing the standard indicators and start experimenting with trailing stops. Try moving averages, donchian channels, ATR trailing stops and anything else you want to throw into the pot.

7. Stick with simple position sizing strategies for starters - 10% of equity or $10,000 per position on $100,000 capital. You can expand on these strategies later.

8. Ideally keep track of the results obtained in a simple spreadsheet or even hardcopy printouts - giving enough detail so you can repeat the tests in the future.

9. Don't worry about walk-forward testing until you have something that you think has a chance.

10. The same goes for optimisation. Just stick to the usual periods touted for indicators, or if you want to change them use something like known number sequences for convenience - 1, 3, 5, 8, 13, 21, 34, 55 etc. Use any sequence you like - they are not magical.

11. Always delay your trades. For EOD trade the next day, not on the close of the current day, EOW trade in the next week not the close of the week, etc.

This will probably take around 6 months maybe longer depending on your skills with Amibroker. I really don't have any comprehensive references on backtesting other than maybe the TradeSim manual. Leon Wilson's book - Breakthrough Trading could be useful. Obviously the online help for Amibroker is essential and there is a lot of stuff on the Amibroker Yahoo site.

Also - get decent data! Free data is useless for serious portfolio backtesting.
regards

Click to expand...

Stevo an excellent post with premise.

Regards Bob.

chops_a_must said:

I have decided to heed the advice of good old Rummy,

"Plan backwards as well as forward. Set objectives and trace back to see how to achieve them. You may find that no path can get you there. Plan forward to see where your steps will take you, which may not be clear or intuitive."

However, I'm hoping you can make things seem a little clearer for me than poor Donald did, and if you can offer me tips/ free software for backtesting, that would be great.

Cheers in advance,
Chops.

Click to expand...

Just to add to some of Stevo's excellent post from a slightly different angle.

(1) What is the purpose of developing a System of trading?

Its to have a set of parameters which when applied to a universe of stock or even singular instruments over a period of time returns a positive expectancy.
IE you return more than you lose---consistently.

(2) I'm confused Does the method have to work on ALL of the trades I take?

No infact you wont know at the time that you take a trade wether that trade will end profitably.The purpose of trading a system which is known to have a positive expectancy is that over a great number of trades the winners- the losers will return a profit.

(3) Do I have to take EVERY single entry triggered by my system?

No ----Montecarlo analysis will show you that a profitable method will work on ANY of the trades triggered. All systems I know of will trigger 10-50 times MORE trades than you can possibly take. Montecarlo testing is simply testing 1000s of combinations of triggered trades and returning the net results of the groups of stocks traded over X time.

Its like me designing a system then calling 20000 people giving them each my capital base (Say $100,000 each) the rules of the system and asking them all to trade it over the next X years---record the results then come back to me with them.

You would find (with a great system---one I would trade) that all would come back with varying positive results.There will be a low extreme and a high extreme and 1000s in the middle (or Mean).

With a bad system (One I wouldnt trade) you would find perhaps many winning participants and some losers.---back to the drawing board.

This is the problem with developing a system that you can only test in singularity.You just dont know wether you would have the same profits or even a loss had you started the system on a different date OR with a different set of trades--OR both. Montecarlo simulation removes this doubt.

In summary a great system will be profitable over time regardless of the stock traded OR when you trade it.

(4) But testing a system over PAST data---how can I have confidence that in FUTURE it will perform the same as it has in the past?

Bottom line is you dont. Walk forward testing will help,but is very slow and tedious (try doing that with 20000 portfolios!!).
The way I have answered this question SO FAR is that all systems will have a "Blueprint" or a set of NUMBERS that are characteristic of that method.
They have max and minimum numbers.If your trading remains within the Blueprint then you WILL recieve the returns that the System has returned in its blueprint.

If it trades outside of the blueprint then you would stop trading the system as market conditions would be such that they are outside ANY TESTED conditions and as such are UNKNOWN.

(5) Should my system work in ALL markets---Bull or Bear

If its designed for both then YES. If its designed for bull markets then NO.
All my trend following systems perform poorly in Bearish conditions and very well in bullish---thats what they are designed for. I am currently looking a "Switches" for turning on and off systems in times of obvious market conditions which are not "ideal"---results are mixed at this point but encouraging enough to pursue.


In the next post I'll just add to some of Steves comments and discuss some things I have found.

stevo said:

Chops
I don't know of any decent free software for portfolio backtesting - don't waste your time. EOD Amibroker is $149 US, and as tech mentions is very good.

If you are serious about backtesting I would go for AmiBroker. I also use Amibroker combined with TradeSim. Tradesim is excellent. You can do Monte Carlo analysis with Amibroker and Excel as well even though it's a little slow. If you don't already know how to code in Amibroker then you will need to spend some time understanding the software and language by running through the tutorials etc

Click to expand...

Best I have seen so far however still lacking in many areas which one day maybe overcome. IE If I switch a method off today and re enter exactly the same portfolio in 3 mths taking taxation in consideration---then how will it perform?--there are many more but this is just one example.

Some suggestions and ideas to get you started;

1. For starters select a universe of stocks to test on, but don't start with too few - at least 400 to 500 stocks. I use the All Ords stocks for basic testing. You should expand/ alter the universe if you think you have a good system.

Click to expand...

We differ a little here. I feel its much easier to design a system on a SINGULAR stock or entity---IE and index or future.
However the system would be specifically designed around IT or the Universe you select.T/T is based upon the BT margin list and performs worse on the whole of the ASX,it performs best on the Hong Kong exchange!!?

2. Don't get bogged down in one time frame. Do some simple tests on daily, weekly and monthly time frames to see which one is easiest to work with. Stick to trading long strategies for starters - you will get confused if you try testing shorting strategies initially.

Click to expand...

Strongly agree.

3. Test the sorts of things that are pushed in trading books all the time. Try out MACD, RSI, Stochastics, Bollinger Bands Donchian channels etc using the methods outlined in trading books. Try the different time frames and see what you think of the results.

Click to expand...

This is the best place for Oscillators in my veiw,however I personally am not a great fan of them I prefer to use price action as much as possible.

4. Try OB /OS style indictators but also reverse the strategy - buy when overbought, sell when oversold. As an example take bollinger bands in a weekly timeframe using the traditional OB/ OS approach then reverse it. Add a trailing stop.

Click to expand...

I have found that simply reversing a trading method from long to short when designed for long side trading to be less than successful. Stock tends to act very differently when falling,than climbing. My veiw is that short side should be handled ALONE. But as I dont trade short,Ive not looked deeply into this way of trading. The old 80/20 rule--- the market is 80% bullish so thats where I'll trade---a personal thing only.

5. Settle on a timeframe (daily / weekly / monthly) that you are most comfortable with for your first system.

Click to expand...

While I trade systems in Daily timeframe Steves system of Weekly is more profitable and easier in my veiw to design.I think weekly is the highest timeframe that I would use.

6. Take some of the ideas that you have obtained from testing the standard indicators and start experimenting with trailing stops. Try moving averages, donchian channels, ATR trailing stops and anything else you want to throw into the pot.

Click to expand...

Ive not yet been able to code throwing a trailing stop in if a move of X is achieved over Y period IE a very quick positive rise of 50% over a few days to a week---thats the only time Id use a trailing stop because if you use it any sooner it becomes the EXIT!,

7. Stick with simple position sizing strategies for starters - 10% of equity or $10,000 per position on $100,000 capital. You can expand on these strategies later.

Click to expand...

Agree however slight cahnges can make dramatic changes to result.
IE I have found 8 positions to be open at the one time in a portfolio to be the optimum---or 12.5% of capital. I have found stops over 10% to weaken the equity curve---IE it isnt as smooth. The optimum stop regardles of how its placed appears to be 8-12% from the purchase price.
So If you designed a mechanical stop which was consistantly inside or ourside these parameters than I would suggest it could be having a negative impact on result.

8. Ideally keep track of the results obtained in a simple spreadsheet or even hardcopy printouts - giving enough detail so you can repeat the tests in the future.

Click to expand...

Great advise---Im slack I dont do it as well as I should.--time factor and obvious profit is my weak excuse!

9. Don't worry about walk-forward testing until you have something that you think has a chance.

Click to expand...

I agree here but is covered above and in particular under the info on Montecarlo testing---without it I strongly believe you could well have a system that breaksdown 5% of the time and NOT know it no matter how often you walk forward test.

10. The same goes for optimisation. Just stick to the usual periods touted for indicators, or if you want to change them use something like known number sequences for convenience - 1, 3, 5, 8, 13, 21, 34, 55 etc. Use any sequence you like - they are not magical.

Click to expand...

My veiw is that optimisation has its place with singular entities and perhaps very very small universes which are in someway correlated. To optimise un correlated universes will be a pointless exercise.To prove this to yourself run the optimisation 6 mths ago and now---they will be entirely DIFFERENT.
Not what you want.

11. Always delay your trades. For EOD trade the next day, not on the close of the current day, EOW trade in the next week not the close of the week, etc.

Click to expand...

This is the BIGGEST mistake newbie systems testers make.Makes enourmious differences to results.often turning profitable systems into disasters.

This will probably take around 6 months maybe longer depending on your skills with Amibroker. I really don't have any comprehensive references on backtesting other than maybe the TradeSim manual. Leon Wilson's book - Breakthrough Trading could be useful. Obviously the online help for Amibroker is essential and there is a lot of stuff on the Amibroker Yahoo site.

Click to expand...

Longer I think---but maybe Im slow on the up take after all Im only a Duck!

Ill post the name of my systems testing BIBLE when I get home,a grasp of Mathematics is a great help. You dont have to be a genius---Im a duck who builds things---cant be that hard!

Also - get decent data! Free data is useless for serious portfolio backtesting.
regards

Click to expand...

Crapp data crapp results. Pay for good software and good data---oneday you expect to make a motzza---if your undercapitalised and cant afford the above---what are you doing trading????????

chops_a_must said:

I have decided to heed the advice of good old Rummy,

"Plan backwards as well as forward. Set objectives and trace back to see how to achieve them. You may find that no path can get you there. Plan forward to see where your steps will take you, which may not be clear or intuitive."

However, I'm hoping you can make things seem a little clearer for me than poor Donald did, and if you can offer me tips/ free software for backtesting, that would be great.

Cheers in advance,
Chops.

Click to expand...

Me too Chops, i posted some stuff on the other thread on backtesting started by snake, maybe the moderators would move my comments to this thread as its all related. Nizar is quite interested as well.

Cheers,

tech and stevo thanks for taking the time.
Excellent discussion here.

CanOz said:

Me too Chops, i posted some stuff on the other thread on backtesting started by snake, maybe the moderators would move my comments to this thread as its all related. Nizar is quite interested as well.

Cheers,

Click to expand...

Yes im very interested.

tech and stevo - TradeSim there are standard ($200), professional ($480) and enterprise editions ($1,500) available. WHich one do you guys use?

so tradesim enhances the backtesting capabilities of metastock/amibroker?
Looks good.

Also - What live data feed do you use?

We both have Enterprise edition.

Live feed is Marketcast for me. I cannot use Marketcast with any of the systems software.
To do that I would have to have Metastock pro.

More backtest info here.

https://www.aussiestockforums.com/forums/showthread.php?t=5157

It's Snake Pliskin said:

Click to expand...

If it trades outside of the blueprint then you would stop trading the system as market conditions would be such that they are outside ANY TESTED conditions and as such are UNKNOWN.

Click to expand...

Well, the philosophy behind my backtesting is about limiting this because:

"Reports that say that something hasn't happened are always interesting to me because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know."

It's all about reducing the known unknowns, and the unknown unknowns.

So then really until the new unknown becomes the now known it is best to suspend the known when it is in unknown territory.

Thats something which I have yet to know as the known is and has stayed within the known.

tech/a said:

So then really until the new unknown becomes the now known it is best to suspend the known when it is in unknown territory.

Thats something which I have yet to know as the known is and has stayed within the known.

Click to expand...

Yes! Now I know what you mean

tech/a said:

So then really until the new unknown becomes the now known it is best to suspend the known when it is in unknown territory.

Thats something which I have yet to know as the known is and has stayed within the known.

Click to expand...

Bahaha!

Now you're getting it!

Tech/Stevo
Good advice. Thanks.
I'm struggling with system development at the moment. My biggest problem is getting comfortable with the system I'm trading. I have 2 specific problems. Can you guys give me a hand please?:

1. Entry signals
Because MA crossovers or oscillator systems or other well known mechanical set-ups throw up so many signals, when I review the trade reports, I think to myself "how do I eliminate those losing trades?". I remind myself to focus on expectancy not on being right but the pull of improving entries is always with me. You know what I'm going to say next - off I go to the Holy Land to find the Holy Grail. I never find a way to improve the entries. Do you guys just accept that on some days you'll look at a signal and think "gees, I'm not sure about this but anyway, my rules say enter so I'll enter?"

2. Exits/stops, etc
Mark Douglas, Van Tharp et al all say that systems are judged by their exits, etc. Seems good advice. How can you argue when such authorities say "know your risk, know when to take profits, do it consistently, it's just a numbers game?" You'd be mad not to take their advice. But then, there you are with your trend following system with 50 trades in a year, with a boat load of commissions, heavy tax bill and only parts of a move, when Jimmy FinReviewMan next door who knows F All except what Kohler says has his ANZ, BHP, etc for a year and nets 30%+. Sure, OneTel and TLS, but a well diversified portfolio of quality ASX stocks looks great.
I want to be a trader so bad but I'm not getting something. What am I missing?

Best regards
Bin

Bin57again said:

1. Entry signals
Some days you'll look at a signal and think "gees, I'm not sure about this but anyway, my rules say enter so I'll enter?"

2. Exits/stops, etc
There you are with your trend following system with 50 trades in a year, with a boat load of commissions, heavy tax bill and only parts of a move, when Jimmy FinReviewMan next door who knows F All except what Kohler says has his ANZ, BHP, etc for a year and nets 30%+.

Click to expand...

1. Rules are rules for a good reason. It's very difficult to improve on a system by using discretionary trade filters. Most of my biggest/best trades were the crappiest looking charts on entry. Conversely, the "can't lose" charts usually do lose. One added entry filter which seems to help many long systems (but test your system with and without it) is CLOSE>OPEN.

2. Because a. that's not what happens every year, and b. buy/hold forgoes considerable opportunity cost. However, if your system is not beating buy/hold it is not yet ready to be traded.

(ps systems based on MA crossovers or oscillators generally perform very poorly due to whipsawing.)

Bin57again said:

Tech/Stevo
Good advice. Thanks.
I'm struggling with system development at the moment. My biggest problem is getting comfortable with the system I'm trading. I have 2 specific problems. Can you guys give me a hand please?:

1. Entry signals
Because MA crossovers or oscillator systems or other well known mechanical set-ups throw up so many signals, when I review the trade reports, I think to myself "how do I eliminate those losing trades?". I remind myself to focus on expectancy not on being right but the pull of improving entries is always with me. You know what I'm going to say next - off I go to the Holy Land to find the Holy Grail. I never find a way to improve the entries. Do you guys just accept that on some days you'll look at a signal and think "gees, I'm not sure about this but anyway, my rules say enter so I'll enter?"

2. Exits/stops, etc
Mark Douglas, Van Tharp et al all say that systems are judged by their exits, etc. Seems good advice. How can you argue when such authorities say "know your risk, know when to take profits, do it consistently, it's just a numbers game?" You'd be mad not to take their advice. But then, there you are with your trend following system with 50 trades in a year, with a boat load of commissions, heavy tax bill and only parts of a move, when Jimmy FinReviewMan next door who knows F All except what Kohler says has his ANZ, BHP, etc for a year and nets 30%+. Sure, OneTel and TLS, but a well diversified portfolio of quality ASX stocks looks great.
I want to be a trader so bad but I'm not getting something. What am I missing?

Best regards
Bin

Click to expand...

Just adding to what Michael has already covered. If, after commissions, taxes and other costs, you cannot beat buy and hold, which in some cases will be 30%+, then your system needs to be modifying.

Another interesting point id like to make is that when you have a system with a positive expectancy over a large number of trades, by backtesting, trading, or even papertrading, you have to make sure you take enough trades for that expectancy to hold true.

Just like how the chances of getting heads or tails is 50/50. Over 10 flips of the coin you may well get 7 or 8 heads. But over 1000 flips, likely to give around 50% of each.

I read an interesting analogy in Van Tharps book. About a game where you have a 60% of doubling your money (or 40% chance of losing it. Double or nothing). 40 participants were given $1000 each, and told to play the game as many times as they liked over a given time frame. At the end of it they all lost money. Why? Position size was too big. Typically, they would bet $100. Loss. And then 2 or 3 more losses.. Damn down to 600 or 700 now, due for a win, so LETS WIN BIG. Then they put 300 or 400. Another loss and they are pretty much gone. Very slim chances of winning now. $10 position sizes and they would easily make money. (Of course these guys didnt pay brokerage). And it seems to be the beginner trader (or should i say gambler/punter) mentality that your chances of winning increases after each successive loss. Ie. 3 losses in a row, good chance the next one is winner!

Money management is so so key to successful trading.

Nizar
How long did it take you to develop systems which perform better than 30%? My edge is pretty poor at around 11 or 12%. What made a difference? Wider market filters or exits, etc? Could you recommend any books on systems development? I read a couple by Lars Kestner and all the usual ones by Tharp et al.
Thanks, Bin