Amibroker System Backtesting

[MovingAverage]

@Trav. Thank you for the reply. Yes, I have downloaded the pdf document and reading it now.

@MovingAverage Thank you for your reply too. you were referring to "backtesting with a statistically relevant number of trades", i suppose you mean that it depends on the type of system and the rules. In that case, if the number of trades less than the relevant number or expected number then the sample period may not be sufficient. Am i interpretating it correctly?

Kind off.

What I was trying to say is that you have to get a good number of trades in your backtest for your backtest results to be meaningful. If your backtest only executes a small number of trades regardless of the date range then your backtest results will not provide you with a proper insight into your system behavior. Generally, the more trades you have in your backtest the more confidence you can have in the backtest results. which is why i was suggesting that you shouldn't focus so much on whether you should run your backtest between certain dates but whether your selected date range is allowing the system to generate a good number of orders. Make sure that your backtest period generates enough trades and if not extend out your date range.

Hope I'm not confusing you more

 

[MovingAverage]

If you can get your hands on this book I can’t recommend it enough. It provides a relatively simple explanation of what you should consider to run meaningful backtests.

 

[Antiwarclan]

@MovingAverage Thanks heaps. Found the book on amazon and downloaded in my kindle. This will be my reading for the next few weeks.

 

[MovingAverage]

@MovingAverage Thanks heaps. Found the book on amazon and downloaded in my kindle. This will be my reading for the next few weeks.

That’s good. Also pay close attention to the section on out of sample testing

 

[Trav.]

Something that I have been meaning to share was a tip about the Walk-Forward optimize process.

I didn't really understand this feature fully ( probably still don't ) except that it did a In Sample and Out Of Sample run to verify the system in a variety of market conditions....

the thing that I wanted to share ( this may have been obvious to you - but when learning to use AmiBroker it takes me a while to understand all the features mentioned in the first pass of reading something)......sorry back to reason for the post

How to tie the Walk-Forward process in with the Custom Back Test file.

Example below..

@Willzy provide some CBT code here https://www.aussiestockforums.com/posts/1076414/ in which he created a new metric for testing system performance against.



So when you go into the Analysis Settings you can select the optimization target which defaults to CAR/MDD as per below, but when you are being a little trickier and want to use some u beaut custom metric as @Willzy has put together then you can use it instead of the default.



Sounds pretty simple but when you use the drop down box your CBT target doesn't appear.....you have to type it in as per your program description..

so as per below we type in OBFN and when you run the Walk-Forward optimization it will use this target now.






Well that's it from me, maybe you will run a few tests against the standard CAR/MDD and the pearsonsR2 metric as per above ?? yes the fun never ends.

Enjoy

Trav

 

[MovingAverage]

Something that I have been meaning to share was a tip about the Walk-Forward optimize process.

I didn't really understand this feature fully ( probably still don't ) except that it did a In Sample and Out Of Sample run to verify the system in a variety of market conditions....

the thing that I wanted to share ( this may have been obvious to you - but when learning to use AmiBroker it takes me a while to understand all the features mentioned in the first pass of reading something)......sorry back to reason for the post

How to tie the Walk-Forward process in with the Custom Back Test file.

Example below..

@Willzy provide some CBT code here https://www.aussiestockforums.com/posts/1076414/ in which he created a new metric for testing system performance against.

View attachment 106001

So when you go into the Analysis Settings you can select the optimization target which defaults to CAR/MDD as per below, but when you are being a little trickier and want to use some u beaut custom metric as @Willzy has put together then you can use it instead of the default.

View attachment 106000

Sounds pretty simple but when you use the drop down box your CBT target doesn't appear.....you have to type it in as per your program description..

so as per below we type in OBFN and when you run the Walk-Forward optimization it will use this target now.

View attachment 105999


View attachment 106002

Well that's it from me, maybe you will run a few tests against the standard CAR/MDD and the pearsonsR2 metric as per above ?? yes the fun never ends.

Enjoy

Trav

Another great tip @trav -- well done.

Walk forward testing is such an important process to understand how your system will perform in the future. I'm sure a lot of folks here understand why it is important but there some that would definitely benefit from incorporating it into their system testing and validation.

 

[Trav.]

Outlier Trades

How do you deal with them??

For me I am looking at eliminating a couple of the best performing trades to make the Back Test a little more realistic. As per the example below I have a couple of trades that appear to be really good and skew the back test results.



So I want to eliminate these out of my back test by placing them in their own watchlist which I have named 'Exclude Symbols' then in the Filter Settings select the Exclude tab and find your watchlist as per below.



I leave the worst performing trades in play, as I don't want to get to cocky with a system being tested.

Do you do the same or something different??

 

[BoNeZ]

How do you deal with them??

I don't have a better solution but I do run my back tests over different time frames and different universes meaning I will run one against all stocks, another using the all ordinaries and one using the ASX 300 etc and I find removing the smaller stocks reduces many outliers.

With stocks like ANO and OMH your position size is a significant percentage of the daily volume so maybe limit your trade to a maximum of something like 1% of the daily volume. You would still execute the trade but with a reduced profit.

It depends on the your strategy and how much risk you want to take. Personally I wouldn't trade ANO or OMH because getting out could be difficult if things go wrong. I trade in larger stocks with higher turnover.

 

[Roller_1]

Outlier Trades

Do you do the same or something different??

I also do the same Trav, exclude the huge winners but leave the losers. The same way as you too

 

[Trav.]

@BoNeZ and @Roller_1 thanks for the responses.

I do have a liquidity filter as well which I have been playing with sizing to limit what you have mentioned @BoNeZ

Forever tinkering and learning

Cheers

 

[Trav.]

As mentioned over in the Dump It Here thread I thought about the use of momentum and other options to ROC on the index filter (or sectors)

Colin Twigg's has some good information here. https://www.incrediblecharts.com/indicators/momentum_indicators.php

So after reading this I decided to run a few tests similar to the below and found some improvement in my back test results.

The below code enables to you look at setting how many periods you want to sample over and what level you want to enter the market with (typical Oversold = 20 and Overbought = 80) so a level that confirms an uptrend.

LookBackPeriod = Optimize("Look Back Period", 10, 5, 20, 1);
StochLevel     = Optimize("Stoch Level", 50, 20, 80, 1);  

SetForeign("$XAO.au");

StochIndex  = StochD(LookBackPeriod);
IndexFilter   = StochIndex >= StochLevel;

RestorePriceArrays();

BuyCond1 = IndexFilter;

This can be done for any exit that you might have in your code. ie If you have a 2 stage trail stop then the second stage can be triggered by similar code above looking for oversold trend.

Give it a go and obviously try different indicators and levels and hopefully you see some positive improvements

 

[TraderJimmy]

Just a quick test run below.

Same code used in the body of each system tested but I have included the following lines in the test system that we are trying to be a weekly system run on the analysis settings of Daily Periodicity

TimeFrameSet(inWeekly);
//*******CODE
TimeFrameRestore ();
TimeFrameExpand( C, inWeekly );

Results with trade delays set to 1

Hi @Trav. I tried this code, and it got me trawling through the trade list to see what days the trades were executed on. Here is a couple of screenshotss.

Sample of trades using daily periodicity & timeframe code above. Note that the dates are all through the week, tuesdays, wednesdays etc


Previous backtest using weekly periodicity. Note that the dates all appear to be Fridays.


My assumption is that we want the buy trades to all be on a Monday - whereas exits can be on various days because our stops can be triggered midweek.

Thoughts?

 

[Trav.]

My assumption is that we want the buy trades to all be on a Monday - whereas exits can be on various days because our stops can be triggered midweek.

@TraderJimmy when I was playing around with this concept I probably ( did ) miss the mark with the way AmiBroker uses the data on a weekly period and this TimeSet function.

So with the normal method of running your weekly scan after close on Friday and then actioning the results on Monday...agreed.

Now to address the possibility of running a scan each night for the weekly system I have to go back to the drawing board and look at our options as I need to conduct some tests to get this right which I tried earlier today but failed, so Plan A was no good and onto Plan B tomorrow.

Also this note in the help file sort of tells us that switching Periodicity is not the way to go, which leaves setting the time frame in the AFL code to daily where we want it. ie Stops in daily mode, which will require a dummy system coded up for testing.



Thanks for bringing this topic up again, and hopefully we can get a solution this time.

Cheers

Trav

 

[TraderJimmy]

Thanks @Trav. I'll be interested to see anything you come up with. It's a bit beyond my pay grade .

I'm still in the "I don't know what I don't know" phase of learning AFL.

 

[TraderJimmy]

@Trav. not sure if this is a) helpful, or b) a priority for you - but I had a play around with the time periods here. For others, Trav and I were talking about how to tweak the Amibroker code on the Weekly Trend Trader strategy.

The problem I/we am trying to solve is:
- the ability to run a backtest for a weekly strategy on any day of the week,
- force buy/sell triggers to be on an "end of week" basis,
- force buys and sells to occure on the first day of the week @ open,
- allow trail stops to sell whenever they are triggered (eg midweek)

If tried the following:

• Set periodicity to "daily"
• Use TimeFrameCompress on buy/sell conditions
  • eg
    • wc = TimeFrameCompress( Close, inWeekly );
    • cond1 = wc > Ref( HHV( wc, 10 ), -1 );
  • I got this from pg 354 of Bandy's Amibroker book
• Set buy delay to zero
• Use the following
  • Buy = dayofweek() == 1; // buy on Monday

This got all the buy signals triggering on Monday , but the results were odd - and were all selling on the day after the buy (Tuesday).

So I'm not sure I'm any closer, just thought I'd share findings.

 

[Warr87]

I'm not really sure why you are forcing into weekly and then forcing the buy on monday? if you set the system to weekly then the end of the week is friday and the beginning of the week is monday. if you have a trade delay of 1 bar then it will buy at the open, which is monday. looks like you are overcomplicating things a little tbh.

keeping period in the settings to weekly, adding a trade delay of 1 bar, and adding BuyPrice = Open; and SellPrice = Open; , should suffice.


Also, if you run the weekly system before the end of the week it should use the current candle (though not completely finished). if you check the chart on wednesday, or thursday, for example it should show that date as the EOW.

 

[TraderJimmy]

Hi @Warr87 yes, we had those simple settings, and the returns looked good, but when looking into the trade results, they were buying throughout the week. This suggested that something was off, buy conditions, periodicity...something. Or it could be data? See post #52.

So we're trying to get the backtest to reflect the intention of the strategy.

 

[Trav.]

So I'm not sure I'm any closer, just thought I'd share findings.

We are slowly getting there mate. I have some code that incorporates your suggestion (via Mr Bandy ) and works on a basic level, but still needs some work to be practical.

- Simple MA system
- Buy on Close > Weekly MA and Day of Week Monday
- Sell on Close < Daily MA
- Periodicity set to Daily so we can compress data, which can be seen on the staircase orange line (WeeklyMA(20))

Explore - I have added the Day of Week column so you can verify that it only buys on Monday == 1

Feel free to pull it apart and incorporate into your system, and let me know what you think / need.

SetOption("InitialEquity", 50000);
SetOption("MaxOpenPositions", 10);

SetTradeDelays(0, 0, 0, 0);
PositionSize    = 10000;

DailyPeriod = 20; // number of averaging periods
DailyMA = MA(Close, DailyPeriod ); // simple moving average
SellDaily = Cross( DailyMA, Close );

WeeklyPeriod = 20; // number of averaging periods
WeeklyClose = TimeFrameCompress( Close, inWeekly );
WeeklyMA = MA( WeeklyClose, WeeklyPeriod ); // simple moving average
SellWeekly = Cross( WeeklyMA, Close );

BuyCond1 = dayofweek() == 1; // buy on Monday
BuyCond2 = Cross( Close, TimeFrameExpand(WeeklyMA, inWeekly) ); // buy when close crosses ABOVE WEEKLY moving average

Buy = BuyCond1 AND BuyCond2;
Sell = SellDaily; // sell when closes crosses BELOW DAILY moving average

buy = ExRem( buy, sell );
sell = ExRem( sell, buy );

// ****** CHARTING ****** //
Plot(Close, "Close", colorDefault, styleBar );
SetChartOptions(1, chartShowDates, chartGridMiddle, 0, 0, 0);
SetChartBkColor(colorBlack);
SetChartBkGradientFill((colorBlack),(colorBlack));

Plot( DailyMA,  "Daily MA(20)", colorAqua, styleLine);
Plot(TimeFrameExpand(WeeklyMA, inWeekly), "Weekly MA(20)", colorOrange, styleLine);

PlotShapes(IIf(Buy, shapeSquare, shapeNone),colorGreen, 0, L, Offset=-40);
PlotShapes(IIf(Buy, shapeSquare, shapeNone),colorLime, 0,L, Offset=-50);
PlotShapes(IIf(Buy, shapeupArrow, shapeNone),colorWhite, 0,L, Offset=-45);

PlotShapes(IIf(Sell, shapeSquare, shapeNone),colorRed, 0, L, Offset=-60);
PlotShapes(IIf(Sell, shapeSquare, shapeNone),colorOrange, 0,L, Offset=-70);
PlotShapes(IIf(Sell, shapeUpArrow, shapeNone),colorWhite, 0,L, Offset=-65);


// ****** EXPLORE ****** //
if( Status( "Action" ) == actionExplore );

Filter = Buy OR Sell ;

AddColumn(Buy, "Buy", 1, colorDefault, colorDefault, 50);
AddColumn(Sell, "Sell", 1, colorDefault, colorDefault, 50);
AddColumn(Close, "Close", 1.2, colorDefault, colorDefault, 50);
AddColumn(ROC( C, 1 ), "%", 1.2, colorDefault, colorDefault, 50);
AddColumn(DayOfWeek(), "Day 1 to 5", 1);

 

[TraderJimmy]

That's great, thanks @Trav.

I can use this to replace the core logic in my code and then build the complexity back out.

 

[CNHTractor]

I was wondering if someone could check my code on a looping stop. The looping stop is taken from code that @trav posted https://www.aussiestockforums.com/posts/1074838/

I am using the code that @Skate has recently posted “Skate's Daily VIX Strategy” https://www.aussiestockforums.com/posts/1089878/

The changes I have made is to comment out //ApplyStop( stopTypeTrailing , stopModePercent , ts , exitatstop = 2 ); // Apply Stop = [ts] Trailing Stop [exitatstop = 2] means check High-Low prices but exit NEXT BAR on regular trade price.

And post the following code

 //=================================================================================

//6. Add a two-stage trailing stop  ==> amend to looping stop

//=================================================================================

StopLevel = 1 -(ts/100);

trailARRAY = Null;

trailstop = 0;

TrailStopSell = 0; // added from @Trav code



for( i = 1; i < BarCount; i++ )

 {


   if( trailstop == 0 AND Buy[ i ] )

    {

       trailstop = High[ i ] * stoplevel [i];  // was "* stoplevel;"

    }

    else Buy[ i ] = 0; // remove excess buy signals


   if( trailstop > 0 AND Close[ i ] < trailstop )  //was "Low"

    {

       TrailStopSell[ i ] = 1; // was "Sell[ i ] = 1;"

       SellPrice[ i ] = trailstop;

       trailstop = 0;

    }


   if( trailstop > 0 AND TrailStopSell [i] == 0)   //was " if( trailstop > 0 )"

    {  

       trailstop = Max( High[ i ] * stoplevel [i], trailstop );

       trailARRAY[ i ] = trailstop;

    }


}


Sell = TrailStopSell  OR C < MA( C, 50 ) AND NoStrength AND NoUpTrend; // Sell when the close is less than the moving average of the last 50 Days with the closing price is less than the Simple Moving Average of the last 12 Days or Sell when uptrend ended or RSI cross

I am using this code to build an exploration that will show not only normal sell condition but a trail stop. As far as I know the apply stop does not show in the exploration

However, when I run a Backtest or exploration I get totally different results, whilst the Trail stop may be different and I am happy to investigate I am also get different Buy results. I am posting the exploration on SPT for 1 January 2020 to current. Using @Skates original code there are no Buys or Sells. However, using the modified code with a looping stop I am getting multiple Buys and sells.

Attached are the AFL files I have used Formula30.afl is Skates code and Formula32.afl is the modified code.

Thanks for any guidance.