Amibroker System Backtesting

[Trav.]

For those interested I ran an optimisation on my PositionSize for one for my strategies I am playing with and bellow are the results.

Before Fixed PositionSize is 10k

After is a mixture of 5k to 15k and definitely not what I would have expected

VolatileBear, 15000
QuietBear, 5000
QuietBull, 15000
VolatileBull, 10000​

I will check over a few different strategies as I suspect a different style of strategy will produce a different mix of PositionSize

Anyway Optimisation code used

PositionSize = IIf( VolatileBear, Optimize("VolatileBear", 10000, 5000, 15000, 5000), IIf( QuietBear, Optimize("QuietBear", 10000, 5000, 15000, 5000),
IIf( QuietBull, Optimize("QuietBull", 10000, 5000, 15000, 5000), IIf( VolatileBull, Optimize("VolatileBull", 10000, 5000, 15000, 5000), 0))));​

Before / After -



Obvious change to profit but some subtle changes overall, which I will need to review.

I am not an expert with all these metrics but tend to be drawn to a couple and even these surprise me (MDD) but will be interested to see how a change in stop parameters impacts this area as well.

Gee the fun never stop, enjoy

 

[Warr87]

are you always in the market? Wouldn't it be easier to protect capital and move to cash during, say, volatile bear market?

 

[Trav.]

are you always in the market? Wouldn't it be easier to protect capital and move to cash during, say, volatile bear market?

@Warr87 that is the point of the exercise mate to determine what is the best strategy for handling such conditions, as I don't have the answers and my trading is short term I should be able to stay in the market at a reduced position size ( or so I would have thought ) as I can get in / out pretty quick as required.

To answer your question my normal strategy has the standard index (sector) filter and switches off the entry signal so no new positions are taken.

But as you have seen there is a lag for a trend following strategy to get back in the market ( and to exit the market ) so I am always keen to try something new and more than likely this trial will end up in the scrap heap with a number of other little projects I have done, but all have served a purpose to teach me something about trading or improving my coding skills.

I am also looking at developing / trialling a mean reversion system which all the above learning's can be used and put into an automated system ( one day hopefully )

As there are so many styles of trading / investing and risk profiles we will have different systems / preferences but I hope in sharing my learning's we can all help out each other in some way.

Cheers

 

[Warr87]

lagging will always be an issue for a trend system.

One more idea may be to also make sure that you don't enter a current counter-trend/pullback — that is, the previous bar (or maybe 2) have a positive ROC, or maybe today's MA is higher than yesterdays MA. The CAM system confirms an uptrend with MACD rising (today's MACD > Ref(MACD, -1)).

Either way, the results you have right now are still pretty impressive.

 

[Trav.]

The CAM system confirms an uptrend with MACD rising (today's MACD > Ref(MACD, -1)).

and don't forget the of ADX Rising as well, but you are right there are a number of ways to skin a cat ( or the old rabbit ) but thanks for your thoughts.

The footy is just about to start so will ponder the above after the Hawks notch up another victory over the Cats....

Oh I just noticed my ' skin a cat reference ' and how appropriate ..

 

[Warr87]

My MA/ADX system I am trialling was drawn from that. It backtests very well (lets see how the forward test goes). I am a fan of ADX, MA, and ROC.

As you say, many ways to skin a cat haha.

 

[Willzy]

For those interested I ran an optimisation on my PositionSize for one for my strategies I am playing with and bellow are the results.

Before Fixed PositionSize is 10k

After is a mixture of 5k to 15k and definitely not what I would have expected

VolatileBear, 15000
QuietBear, 5000
QuietBull, 15000
VolatileBull, 10000​

I will check over a few different strategies as I suspect a different style of strategy will produce a different mix of PositionSize

Anyway Optimisation code used

PositionSize = IIf( VolatileBear, Optimize("VolatileBear", 10000, 5000, 15000, 5000), IIf( QuietBear, Optimize("QuietBear", 10000, 5000, 15000, 5000),
IIf( QuietBull, Optimize("QuietBull", 10000, 5000, 15000, 5000), IIf( VolatileBull, Optimize("VolatileBull", 10000, 5000, 15000, 5000), 0))));​

Before / After - View attachment 104684

View attachment 104687

Obvious change to profit but some subtle changes overall, which I will need to review.

I am not an expert with all these metrics but tend to be drawn to a couple and even these surprise me (MDD) but will be interested to see how a change in stop parameters impacts this area as well.

Gee the fun never stop, enjoy

I Like your experiment @Trav. What do your results look like when you simply test each regime on its own to the exclusion of the others? Your results suggest that the QuietBear is the worst performer whilst the QuietBull and VolatileBear are equally best. How does the VolatileBear look when traded in isolation?

Also, very happy to discuss mean reversion strategies with you if your interested. I've dedicated a considerable portion of my research to them, though I'm not rich yet.... Lol

 

[Warr87]

Also, very happy to discuss mean reversion strategies with you if your interested. I've dedicated a considerable portion of my research to them, though I'm not rich yet.... Lol

I keep trying, and then give up, haha.

 

[Trav.]

@Willzy good idea to strip it apart and see what the main driver and the weakest link.

I am not the best backtester in the world so feel free to correct me if I have this wrong.

From @Willzy suggestion I did 2 types of tests over the previous sample period and stock universe and then recorded each regimes results for the test.

***** Note results are different from the previous as I found that I had the initial equity amount incorrect and allow position size shrinkage ticked for some reason ? but end of the day all results are tested on a equal basis and gives a good indication of what impact the regime filter has.******

1. Incorporated each regime into the buy line and put the PositionSize back to $10k so the playing field was level

PostionScore Weighting - VolatileBear, 0, QuietBear, 1000, QuietBull, 2000 ,VolatileBull, 0​

2. Incorporated each regime into the buy line and allowed for PositionSize to be varied

PostionScore Weighting - VolatileBear, 0, QuietBear, 1000, QuietBull, 2000 ,VolatileBull, 0
PositionSize as per combined system test VolatileBear, 15000, QuietBear, 5000, QuietBull, 10000 ,VolatileBull, 20000​

Results are in the attached spreadsheet

Summary
- Individually the Volatile Bull Regime is a clear winner on it's own and lends to @Warr87 comment about protecting capital in a bear market.
- But for a complete system with regime filter controlling position size and position score it has the edge.

Note: I need to be more cleverer in looking at backtests and will need to brush up on the CBT to extract better metrics for each regime and what it is doing in the complete system test

And finally this still leaves a few question for me as the weighting is not what I expected and will test against another style of system today to compare.

Cheers

 

[Warr87]

I've mentioned this in a few posts before, as I am a fan of Clenow, but have you thought about instituting position sizing based on the basis points and ATR ? Many people who have tried to code his momentum strategy in AB have done so with the rotational trading and his risk management is based on:

Account value * 0.001 (Risk, i.e. 10 basis points here) /ATR (20 period)

You could try it this way and change the risk based on the regime filter. That is, risk 30 basis points in a volatile bull and only 5 basis points in a volatile bear, for example.

In Clenow's momentum system he uses this formula and will keep filling positions as long as he has the cash (what gets filled is based on his ranking score). You have may to adjust this to a certain number of maximum positions but if you use this method be careful that during a bear market you don't have 50 positions (if you use a low basis points multiplier then you will natural have more positions).

This may not be suited to this system but I thought the defined risk and position size idea may help.

 

[Trav.]

@Warr87 thanks mate, I will have a think about it

 

[Warr87]

Honestly, it may not be suitable at all. The idea with that position sizing is that you end up putting more money into those stocks that are doing well. It is also rebalanced monthly, too, which means scaling in and out of positions based on momentum and volatility.

Hopefully it is helpful to some degree (or gives someone reading this thread an idea).

 

[Trav.]

And finally this still leaves a few question for me as the weighting is not what I expected and will test against another style of system today to compare.

Wow what an experience that was.

I Picked another system of the shelf and dusted it of for the purpose of testing Market Regime Filter and I was surprised at the results.

Back test parameters as per before ASX200, 1/1/2018 - 11/6/2020

1. Incorporated each regime into the buy line and put the PositionSize back to $10k so the playing field was level
PostionScore Weighting - VolatileBear, 1000, QuietBear, 1000, QuietBull, 0,VolatileBull, 1000

2. Incorporated each regime into the buy line and allowed for PositionSize to be varied
PostionScore Weighting - VolatileBear, 1000, QuietBear, 1000, QuietBull, 0 ,VolatileBull, 1000
PositionSize as per combined system test VolatileBear, 15000, QuietBear, 5000, QuietBull, 20000 ,VolatileBull, 20000

Column B - Default systems of the shelf (default)
Column D - Optimized with parameters above in 2 (Complete)
Columns F-I - Fixed positions
Columns K-N - Variable as defined by Filter




More detailed results in the attached if you are interested, but some intriguing results above and does make you challenge your initial thoughts on were the money will come from.

What I find interesting in both systems tested that the complete system does very well and this will need to be investigated further so onto the next challenge and I will need some help !

Note: I need to be more cleverer in looking at backtests and will need to brush up on the CBT to extract better metrics for each regime and what it is doing in the complete system test

 

[Willzy]

@Willzy good idea to strip it apart and see what the main driver and the weakest link.

I am not the best backtester in the world so feel free to correct me if I have this wrong.

From @Willzy suggestion I did 2 types of tests over the previous sample period and stock universe and then recorded each regimes results for the test.

***** Note results are different from the previous as I found that I had the initial equity amount incorrect and allow position size shrinkage ticked for some reason ? but end of the day all results are tested on a equal basis and gives a good indication of what impact the regime filter has.******

1. Incorporated each regime into the buy line and put the PositionSize back to $10k so the playing field was level

PostionScore Weighting - VolatileBear, 0, QuietBear, 1000, QuietBull, 2000 ,VolatileBull, 0​

2. Incorporated each regime into the buy line and allowed for PositionSize to be varied

PostionScore Weighting - VolatileBear, 0, QuietBear, 1000, QuietBull, 2000 ,VolatileBull, 0
PositionSize as per combined system test VolatileBear, 15000, QuietBear, 5000, QuietBull, 10000 ,VolatileBull, 20000​

Results are in the attached spreadsheet

Summary
- Individually the Volatile Bull Regime is a clear winner on it's own and lends to @Warr87 comment about protecting capital in a bear market.
- But for a complete system with regime filter controlling position size and position score it has the edge.

Note: I need to be more cleverer in looking at backtests and will need to brush up on the CBT to extract better metrics for each regime and what it is doing in the complete system test

And finally this still leaves a few question for me as the weighting is not what I expected and will test against another style of system today to compare.

Cheers

Had a quick squiz at your spreadsheet and the results are interesting! In order to confirm an edge I'd focus on the 10K per trade section where you've tested each regime in isolation. Looking at the number of trades taken will be key here also.

In order to ensure we're comparing apples with apples I'd suggest doing exactly as you have done with 10K per trade but also allowing an almost unlimited number of positions.

So set initialEquity = 1000000, maxNumPos = 100, posSize = 10000
Then when comparing outcomes I'd look specifically at numTrades and AvgTrade ($ or %).

Cheers!

 

[Willzy]

Wow what an experience that was.

I Picked another system of the shelf and dusted it of for the purpose of testing Market Regime Filter and I was surprised at the results.

Back test parameters as per before ASX200, 1/1/2018 - 11/6/2020

1. Incorporated each regime into the buy line and put the PositionSize back to $10k so the playing field was level
PostionScore Weighting - VolatileBear, 1000, QuietBear, 1000, QuietBull, 0,VolatileBull, 1000

2. Incorporated each regime into the buy line and allowed for PositionSize to be varied
PostionScore Weighting - VolatileBear, 1000, QuietBear, 1000, QuietBull, 0 ,VolatileBull, 1000
PositionSize as per combined system test VolatileBear, 15000, QuietBear, 5000, QuietBull, 20000 ,VolatileBull, 20000

Column B - Default systems of the shelf (default)
Column D - Optimized with parameters above in 2 (Complete)
Columns F-I - Fixed positions
Columns K-N - Variable as defined by Filter


View attachment 104723

More detailed results in the attached if you are interested, but some intriguing results above and does make you challenge your initial thoughts on were the money will come from.

What I find interesting in both systems tested that the complete system does very well and this will need to be investigated further so onto the next challenge and I will need some help !

2018 - 2020 could be a bit narrow, do you get similar results say 2000 - 2018?

 

[Trav.]

@Willzy all good points mate.

For the intent of the thread I will leave this testing at what has been done as it is, and will look at creating another thread for the back testing portion ( maybe transfer some of these posts ) as I think that developing a robust methodology is very important as we all get carried away with some big numbers which at the end of the day are pretty unrealistic.

So I am thinking about setting up (with everyone's input) some framework that will help us all to standardise a process ( well 90% of it) and with my new found skills in CBT hopefully we can add some code to help users build some meaningful metrics.

Cheers

 

[Trav.]

****** Re-posting this here as it is applicable to Back Testing *******

OK after a little detour and a slight change in strategy oh and about 6 hours of reading and googling I think I have come up with a solution

As per above I managed to assign the appropriate type of entry to each trade

Then I found some code in the AmiBroker manual that was designed for counting different stop methods and did a 'macgyver' as per @Warr87 suggestion and managed to sum all types of entries as per below

Columns are All Trades and Long Trades

Code attached free of charge !! (i'm getting addicted to these little emoji's)

I will post over in the new backtesting thread with same info and then we can explore some other information that I uncovered during this process. Don't worry it's not ground breaking but like most things with AmiBroker all the information is there but you have to find it then have a use. For me I read multiple articles but if it doesn't apply to me that day I will just move on. A bit like today when @Skate re-posted some great information about Norgate data that I would have read but dismissed at the time, as I have very limited capacity up stairs

Anyway I digress, I hope the above is useful to you.

EDIT:

You will need the following in the main code

RegimeFilterCBT = IIf(VolatileBear, 1, IIf(QuietBear, 2, IIf (QuietBull, 3, IIf (VolatileBull, 4, Null)))); // Assigns a number to entry regime
StaticVarSet( Name() + "RegimeFilter", RegimeFilterCBT ); // Sets the value of static variable so it can be retrieved in CBT

****** Re-posting this here as it is applicable to Back Testing *******

 

[Trav.]

we can explore some other information that I uncovered during this process

While this thread has been put together a little haphazardly I will continue the theme for now.

And before we get some structure here I thought that I would post some code that was in the AmiBroker help files online (link below) but as usual they don't make it easy for beginners like me. So I have put together the following to help us in our backtesting by identifying the number and type of stop that the system exited the trade on.

http://www.amibroker.com/kb/category/analysis/backtest-analysis/custom-backtest/

The following values are used for indication of the particular exit reason:

1. normal exit
2. maximum loss stop
3. profit target stop
4. trailing stop
5. n-bar stop
6. ruin stop (losing 99.96% of entry value)
​

So the above different types of stops have been added in the attached file and produced the following when I inserted some standard Stops from the snippet library.

​

"Thank your mother for the rabbits!" Rex Hunt

 

[Trav.]

Backtesting Trade Ideas

I think that we each need to have a standardised approach to test all our trade ideas so we can compare each trade idea against each other in a consistent manner then we can make an informed decision on whether or not to continue with development with the idea.

I have fallen into the trap of testing an idea and it looked great only to find out I had some setting ticked ( or unticked ) which then caused errors in the backtest report. One thing you find out quickly is that AmiBroker does not make any mistakes it is the operator ( me ) who provides poor information which causes errors in accuracy.

I have tried a few options over the last year or so and to be honest I still don’t have the perfect way to test hence the idea for this thread so a consistent approach can be developed with input from interested forum members.

Analysis Settings Options

1. The first way I started testing was by adjusting the analysis settings manually via the control panel and then saving into a AmiBroker Analysis Settings (*.ABS) files, which worked great until I had a few different files and made mistakes by not loading a file or having the incorrect file open

​

o AmiBroker guide Back-testing your trading ideas - https://www.amibroker.com/guide/h_backtest.html​

2. Now I initialise trading parameters in the AFL code via the SetOption function which takes out the human error ( that’s me again ) by having the incorrect options selected for a particular setting.

o I like to have the following options pre-configured

SetOption ( "CommissionMode", 2 ); // $ per trade
SetOption ( "CommissionAmount", 30 ); // $ Amount per trade
SetOption ( "InitialEquity", 100000 );
SetOption ( "AccountMargin", 100); // 100 means no margin
PositionSize = 10000;
SetPositionSize( PositionSize, spsValue ); // Dollar value of size
MaxPos = 20;
SetOption( "MaxOpenPositions", MaxPos );
SetTradeDelays( 1, 1, 1, 1 ); // Trade following day
SetOption ( "AllowSameBarExit", 0); // Off only allows next bar exit
SetOption ( "SeparateLongShortRank", 1); // Separate list for short system created​

o But obviously there is a heap more settings that can be adjusted and I think that I will look at having a file with all available options in it so it overrides any accidentally set in the Analysis window which may cause incorrect backtesting results​

3. Custom Backtest Procedures

o As mentioned here in the AmiBroker Tips and Tricks thread https://www.aussiestockforums.com/posts/1075825/ you can load in you CBT file automatically and bring in some custom metrics which we have already started to learn how to develop in the previous posts.​

Backtesting Strategies

1. What stock universe do you want to work with?

o All Ordinaries, ASX200, ASX 300 etc​

2. Market Time Frame to be used

o If you use a narrow time frame you might be only testing a bull market
o What is In Sample period - Generally, where you are getting good results or optimised over a set time frame and you will be in a bull market for a long system
o What is Out of Sample period - Separate to the In Sample period where you might encounter different market conditions and your system is not optimised to this period and you could expect poorer results​

3. Optimisation

o I think this is a great tool to see the effect of a rule or parameter you have added to your system.
o How much is too much. I am always cautious here but for me if you are using indicators with a period like MA(C,8.5) then I think you are over optimising and wasting your time.​

4. Walk Forward

o This is an area that I need to learn more about and AmiBroker is great in tackling the issue with In / Out of Sample data by automatically advancing dates as per below setup​

​

Backtesting Metrics

1. What metric is important to you and your system ( or combination )

o Trade / System Drawdown
o Annual percentage gain
o Holding period
o Trading frequency
o Exposure
o Expectancy
o Equity smoothness
o Percent winners
o Win to loss ratio​

Summary

So as you can see I have only touched on the topic of Backtesting your trade ideas and this is an area that we all need to understand and have a structured approach with so hopefully everyone will add a comment or tip so we can learn and become better system testers which in turn will hopefully make us better trades.

Cheers

Trav

 

[Trav.]

I have found a paper from Howard Bandy (2012) who goes into good detail about walk forward testing and many other things

Developing Robust Trading Systems, with Implications for Position Sizing and System Health