Amibroker System Backtesting

[qldfrog]

the only point I can add and maybe of value is:
do you truly believe the market react in any way nowadays as it did 10 or 20y ago, before heavy internet trading qants etc;
Out of this then: be careful of pre 2010/2015 backtests/ optimisation;
Just my thoughts :thanks to the latest crash we have a variety of recent periods to use and check system behaviours with a market more reasonably close to what we can expect in the near future

 

[rnr]

Backtesting Trade Ideas

I think that we each need to have a standardised approach to test all our trade ideas so we can compare each trade idea against each other in a consistent manner then we can make an informed decision on whether or not to continue with development with the idea.

I have fallen into the trap of testing an idea and it looked great only to find out I had some setting ticked ( or unticked ) which then caused errors in the backtest report. One thing you find out quickly is that AmiBroker does not make any mistakes it is the operator ( me ) who provides poor information which causes errors in accuracy.

I have tried a few options over the last year or so and to be honest I still don’t have the perfect way to test hence the idea for this thread so a consistent approach can be developed with input from interested forum members.

Analysis Settings Options

1. The first way I started testing was by adjusting the analysis settings manually via the control panel and then saving into a AmiBroker Analysis Settings (*.ABS) files, which worked great until I had a few different files and made mistakes by not loading a file or having the incorrect file open

View attachment 104748
o AmiBroker guide Back-testing your trading ideas - https://www.amibroker.com/guide/h_backtest.html​

2. Now I initialise trading parameters in the AFL code via the SetOption function which takes out the human error ( that’s me again ) by having the incorrect options selected for a particular setting.

o I like to have the following options pre-configured

SetOption ( "CommissionMode", 2 ); // $ per trade
SetOption ( "CommissionAmount", 30 ); // $ Amount per trade
SetOption ( "InitialEquity", 100000 );
SetOption ( "AccountMargin", 100); // 100 means no margin
PositionSize = 10000;
SetPositionSize( PositionSize, spsValue ); // Dollar value of size
MaxPos = 20;
SetOption( "MaxOpenPositions", MaxPos );
SetTradeDelays( 1, 1, 1, 1 ); // Trade following day
SetOption ( "AllowSameBarExit", 0); // Off only allows next bar exit
SetOption ( "SeparateLongShortRank", 1); // Separate list for short system created​

o But obviously there is a heap more settings that can be adjusted and I think that I will look at having a file with all available options in it so it overrides any accidentally set in the Analysis window which may cause incorrect backtesting results​

3. Custom Backtest Procedures

o As mentioned here in the AmiBroker Tips and Tricks thread https://www.aussiestockforums.com/posts/1075825/ you can load in you CBT file automatically and bring in some custom metrics which we have already started to learn how to develop in the previous posts.​

Backtesting Strategies

1. What stock universe do you want to work with?

o All Ordinaries, ASX200, ASX 300 etc​

2. Market Time Frame to be used

o If you use a narrow time frame you might be only testing a bull market
o What is In Sample period - Generally, where you are getting good results or optimised over a set time frame and you will be in a bull market for a long system
o What is Out of Sample period - Separate to the In Sample period where you might encounter different market conditions and your system is not optimised to this period and you could expect poorer results​

3. Optimisation

o I think this is a great tool to see the effect of a rule or parameter you have added to your system.
o How much is too much. I am always cautious here but for me if you are using indicators with a period like MA(C,8.5) then I think you are over optimising and wasting your time.​

4. Walk Forward

o This is an area that I need to learn more about and AmiBroker is great in tackling the issue with In / Out of Sample data by automatically advancing dates as per below setup
View attachment 104747​

Backtesting Metrics

1. What metric is important to you and your system ( or combination )

o Trade / System Drawdown
o Annual percentage gain
o Holding period
o Trading frequency
o Exposure
o Expectancy
o Equity smoothness
o Percent winners
o Win to loss ratio​

Summary

So as you can see I have only touched on the topic of Backtesting your trade ideas and this is an area that we all need to understand and have a structured approach with so hopefully everyone will add a comment or tip so we can learn and become better system testers which in turn will hopefully make us better trades.

Cheers

Trav

A question to those forum members interested in system design.

I am just chasing your thoughts concerning the legitimacy of coding a stop, for back-testing purposes, which intentionally references a future value?
Whilst this question could relate to any software (as in not only Amibroker) that may be applicable to the task I have included it here as Trav has referenced "back-testing trade ideas".

Cheers,
Rob

 

[Trav.]

. Now I initialise trading parameters in the AFL code via the SetOption function which takes out the human error ( that’s me again ) by having the incorrect options selected for a particular setting.

o I like to have the following options pre-configured
SetOption ( "CommissionMode", 2 ); // $ per trade
SetOption ( "CommissionAmount", 30 ); // $ Amount per trade
SetOption ( "InitialEquity", 100000 );
SetOption ( "AccountMargin", 100); // 100 means no margin
PositionSize = 10000;
SetPositionSize( PositionSize, spsValue ); // Dollar value of size
MaxPos = 20;
SetOption( "MaxOpenPositions", MaxPos );
SetTradeDelays( 1, 1, 1, 1 ); // Trade following day
SetOption ( "AllowSameBarExit", 0); // Off only allows next bar exit
SetOption ( "SeparateLongShortRank", 1); // Separate list for short system created

So I have settled on the following setup which will hopefully reduce the human error in having incorrect settings applied or at at least inconsistent setting.

In the made code I have inserted the following line ( you will have to replace the **** with your folder structure)

#include_once "C:\*****\*****\*****\Formulas\Include\BackTest_Settings.afl"
​

I also created a new AFL file and stored it in the following folder with the file name as shown

\Formulas\Include\BackTest_Settings.afl​

That AFL file contains the above quoted settings and whatever you want to remain consistent across all your systems when you include the backtest settings file which does open up a few interesting ideas.

 

[Trav.]

2018 - 2020 could be a bit narrow, do you get similar results say 2000 - 2018?

@Willzy I hadn't forgotten your suggestion but I am having trouble with the $XVI data as my database for the $XVI starts at 2008.

I have the 20 year package and all the other data comes in fine, so I am just investigating this issues.

Cheers

 

[Roller_1]

Hey Trav, Did you have some info somewhere on the different regimes you're testing? Think i've missed something...

 

[Trav.]

@Roller_1 yeah sorry mate, I started this discussion in the Amibroker Tips and Trick thread which I managed to derail a bit and got this new thread created so I can understand the confusion.

https://www.aussiestockforums.com/posts/1076776/

The link should take you to the start of the discussion about regime filters which should hopefully link everything together for you.

Cheers

 

[Trav.]

Link below is for the Amibroker guide that briefly explains how to read the backtest report.

https://www.amibroker.com/guide/h_report.html

I also like this point as previously I was unsure about the colouring as I normally just had green

Some of the metrics in the backtest report are color-coded. Blue means "neutral", Green means "good", Red means "bad". Metrics that are not colorized are always black.
This color coding is of course arbitrary and should be used as guideance only. Treat 'red' as a warning flag and advice to check the value in detail.

As of now the following metrics are colorized:
Net Profit, Net Profit % - bad < 0, good > 0
Annual Profit %, bad < 0, neutral betwen 0 and 10, good > 10
RAR % bad < 0, good > (10 / Exposure)
Avg. Profit/Loss all trades (Expectancy $) - bad < 0, good > 0
Avg Profit/Loss % all trades (Expectancy %) - bad < 0, good > 0
Max. system % drawdown - bad: dd worse than -30%, neutral: dd between -30 and -10%, good - -10% to 0%
CAR/MaxDD, RAR/MaxDD - bad < 1, neutral between 1 and 2, good > 2
Recovery factor - bad < 1, neutral between 1 and 2, good > 2
Payoff ratio - bad < 1, neutral between 1 and 2, good > 2​

 

[Trav.]

For those testing systems on a weekly periodicity and trade delays set to 1

As per below post from the AmiBroker forum for a trade on Monday (next daily bar) do you run backtest in Daily periodicity and use TimeFrameCompress ??

This is something that I didn't know about in a weekly system. I have used TrimeFrameCompress for bringing in Foreign data for a filter but I would have thought that AmiBroker backtest would have taken in account that most people would have traded on the first opportunity being the Monday (next daily bar). I will need to test and see what this does to the performance ( good or bad ?)

https://forum.amibroker.com/t/periodicity-weekly-cbi/2783

 

[Trav.]

Just a quick test run below.

Same code used in the body of each system tested but I have included the following lines in the test system that we are trying to be a weekly system run on the analysis settings of Daily Periodicity

TimeFrameSet(inWeekly);
//*******CODE
TimeFrameRestore ();
TimeFrameExpand( C, inWeekly );

Results with trade delays set to 1

Normal code run in backtest with the periodicity set WEEKLY..........Modified code with the TimeFrameSet code added and run on DAILY periodicity



So we can see a couple of lines of code can dramatically change your backtest results, in this cause negatively.

Does anyone have experience in this?

Did I run the test correctly or use the code incorrectly?

Happy to hear your thoughts or even see what differences you have come across.

Cheers

 

[qldfrog]

For those testing systems on a weekly periodicity and trade delays set to 1

As per below post from the AmiBroker forum for a trade on Monday (next daily bar) do you run backtest in Daily periodicity and use TimeFrameCompress ??

This is something that I didn't know about in a weekly system. I have used TrimeFrameCompress for bringing in Foreign data for a filter but I would have thought that AmiBroker backtest would have taken in account that most people would have traded on the first opportunity being the Monday (next daily bar). I will need to test and see what this does to the performance ( good or bad ?)

https://forum.amibroker.com/t/periodicity-weekly-cbi/2783

View attachment 104829

not sure I interpret/misinterpret properly: weekly system, trade delay = 1, price =open
a buy in BT will still buy at the open price on the Monday following Friday EOW..
That was my assumption, is it otherwise?
that assumption was really matching real action when I was following my systems week by week against backtest

 

[Roller_1]

look at the number of trades Trav looks like the code is doing something completely different in both tests

 

[Chronos-Plutus]

I have found a paper from Howard Bandy (2012) who goes into good detail about walk forward testing and many other things

Developing Robust Trading Systems, with Implications for Position Sizing and System Health

Brokers know your positions. I don't like it.

 

[MovingAverage]

For those interested I ran an optimisation on my PositionSize for one for my strategies I am playing with and bellow are the results.

Before Fixed PositionSize is 10k

After is a mixture of 5k to 15k and definitely not what I would have expected

VolatileBear, 15000
QuietBear, 5000
QuietBull, 15000
VolatileBull, 10000​

I will check over a few different strategies as I suspect a different style of strategy will produce a different mix of PositionSize

Anyway Optimisation code used

PositionSize = IIf( VolatileBear, Optimize("VolatileBear", 10000, 5000, 15000, 5000), IIf( QuietBear, Optimize("QuietBear", 10000, 5000, 15000, 5000),
IIf( QuietBull, Optimize("QuietBull", 10000, 5000, 15000, 5000), IIf( VolatileBull, Optimize("VolatileBull", 10000, 5000, 15000, 5000), 0))));​

Before / After - View attachment 104684

View attachment 104687

Obvious change to profit but some subtle changes overall, which I will need to review.

I am not an expert with all these metrics but tend to be drawn to a couple and even these surprise me (MDD) but will be interested to see how a change in stop parameters impacts this area as well.

Gee the fun never stop, enjoy

Very interesting @Trav. I'm curious, have you looked at the monte results for some of those system metrics? I love to know what the spread or std dev is on trade and system drawdown across the position sizes.

 

[Trav.]

@MovingAverage no mate I have not looked at the Monte Carlo results as that is another thing that I need to get my head around.

When I get time next week I will re-run some backtests and get some data for you so we can look at the differences (good or bad)

 

[Trav.]

When running backtest the ApplyStop function can trigger a number of stops as per post #18 above

I found this bit of code on the AmiBroker forum that plots the exit type and value on the chart which is ideal for reviewing your backtest results to ensure that the system is doing what you expect it to do. Again the AFL code for the applystop doesn't need to be checked but the way we use it can be the problem ( for example - exit intraday or next bar)

Equity( 1 );

bi = Barindex();
fvb = FirstVisiblevalue( bi );
lvb = LastVisiblevalue( bi );
dist = 1.5*ATR(10);
sellorcovertrigger = "Regular,Stop,Profit,Trail,N-bars,Ruin"; 

for( i = fvb; i <= lvb; i++ )
{
    if( Buy[i] )
        PlotText( "Buy\n" + BuyPrice[i], i, BuyPrice[i] - dist[i], colorGreen, -1, 0 );

    if( Sell[i] )
        PlotText( StrExtract(sellorcovertrigger, Sell[i]-1) + "\n" + SellPrice[i], i, SellPrice[i] + dist[i], colorRed, -1, 0 );

    if( Short[i] )
        PlotText( "Short\n" + ShortPrice[i], i, ShortPrice[i] + dist[i], colorRed, -1, 0 );

    if( Cover[i] )
        PlotText( StrExtract(sellorcovertrigger, Cover[i]-1) + "\n" + CoverPrice[i], i, CoverPrice[i] + dist[i], colorGreen, -1, 0 );
}

 

[Trav.]

I thought that I should paste up an example of the above code (obviously the arrows are not in the above code)

 

[Antiwarclan]

Hi All,

I am new and have recently started using Amibroker. I have some questions about system backtest and reading the backtest report. My apologies, if this thread is not the right place for the following questions.

Q1 - I am wondering which periods do you guys use in your backtest? Currently I am building my own trading system and I have been backtesting my system twice, firstly from 1/1/2008 to 31/12/2011, and secondly 1/1/2018 to 24/06/2020.

Q2 - If possible can you guys share about which items do you focus on when reading your backtest report (items such as Annual returns, Avg W/L, MDD and etc).

Cheers,
Antiwarclan

 

[Trav.]

Welcome to the forum @Antiwarclan

I am sure everyone has there favourite way to test as there are so many options I don't think that there is a right or wrong way.

Are we talking Weekly or Daily ?

If Daily I personally focus on the last 2 years of data for the initial review of the system the run the Walk Forward option with default settings to see how it performs in the previous years with out of sample date.

For weekly you will probably want to look at an a bigger initial time frame before the walk forward run.

Post # 20 here has a pdf that may help you interpret the backtest report and what metrics to focus on, as it is not just about the profit that the system has.

How many losses in a row can you handle?
Maximum trade draw down or system draw down levels are really dependent on you.
Avg. Bars Held is something that I look at being short term trader but if weekly does this matter?

So unfortunately the above is not much help but if you have something specific to ask about back testing then fire away as hopefully someone might be able to help.

Cheers

Trav

 

[MovingAverage]

Hi All,

I am new and have recently started using Amibroker. I have some questions about system backtest and reading the backtest report. My apologies, if this thread is not the right place for the following questions.

Q1 - I am wondering which periods do you guys use in your backtest? Currently I am building my own trading system and I have been backtesting my system twice, firstly from 1/1/2008 to 31/12/2011, and secondly 1/1/2018 to 24/06/2020.

Q2 - If possible can you guys share about which items do you focus on when reading your backtest report (items such as Annual returns, Avg W/L, MDD and etc).

Cheers,
Antiwarclan

Q1: Personally, I don't focus first and foremost on a specific date range. I'm more concerned about backtesting with a statistically relevant number of trades. For example, testing over a five year period for a system that takes 5000 trades during that time is probably ok. But if you test a different system over the same period of time and it only takes 150 trades then that 5 year period is not sufficient. I also consider testing over different market conditions: trending up, sideways and tending down.

Q2: Personally, the parameters I focus most on are: system draw down; trade draw down; and consecutive wins and losers. These are the ones that mess with my head the most when I have real money in the market. And of course CAGR is the obvious one.

Good luck

 

[Antiwarclan]

@Trav. Thank you for the reply. Yes, I have downloaded the pdf document and reading it now.

@MovingAverage Thank you for your reply too. you were referring to "backtesting with a statistically relevant number of trades", i suppose you mean that it depends on the type of system and the rules. In that case, if the number of trades less than the relevant number or expected number then the sample period may not be sufficient. Am i interpretating it correctly?