# Ideal backtesting



## jjbinks (25 September 2016)

Hi guys,

I have recently got into trying to develop a mechanical system for trading in Amibroker. 

I haved coded the system and am optimising parameters but was wondering what the best way to do out of sample testing is. 

Curently I am developing my code on data from 2000-2010. Would it be adequate to then just test 2010-2016 results to see if the results are comparable. Or is there a better way to do this?

Thank you and much appreciated


----------



## captain black (25 September 2016)

jjbinks said:


> Hi guys,
> 
> I have recently got into trying to develop a mechanical system for trading in Amibroker.
> 
> ...




The walk forward timeframe you use for testing the validity or stability of a system is also one of the parameters that can be tested. Using the stepped walk forward in Amibroker you could do a 1 year in-sample / 1 year out-of-sample walk forward test from 2000 up to 2016. You'll be able to plot an IS and OOS comparison of the equity curves. Perform a series of IS and OOS walk forward tests over varying IS and OOS periods and see how much variation there is between each IS and OOS equity run. If the variation of the runs is high then there's a higher probability of the system being unstable.

The length(s) of your IS and OOS periods should be determined by the frequency of trades. For a short term system you would use different testing periods than you would for a longer term trend following system for example.


----------



## jjbinks (25 September 2016)

Is there a downside in simply setting aside data to manually test your strategy. 

e.g. i haven't used any data from 2010-2016 in testing my current strategy. So wants I finish tweaking it with data from 2000-2010 i will test it from the 2010-2016 to see if results are acceptable?


----------



## captain black (25 September 2016)

jjbinks said:


> Is there a downside in simply setting aside data to manually test your strategy.
> 
> e.g. i haven't used any data from 2010-2016 in testing my current strategy. So wants I finish tweaking it with data from 2000-2010 i will test it from the 2010-2016 to see if results are acceptable?




That's one way to approach the robustness of your system. Going forward with a strategy I like to know what the best parameters for measuring the health of the system are. One of the ways to measure going forward (ie. once your live trading it) is to measure your performance once again by how far the variation between previous IS and OOS tests have been compared to the variation of IS testing and your live results.

For example, if your walk forward analysis has shown that using a 1 year step forward test shows that to be the optimal walk forward period for testing then continuing to measure your systems performance against the previously tested variations will show whether the system is still tracking it's expected performance or whether it's "broken".

I have a short term momentum system I trade on the ASX that I use a 3 month reassessment period to test it's robustness against historical variances in IS and OOS samples. My longer term trend following systems (daily and weekly) are reassessed annually. 

The downside of doing just a single walk forward run ( 2010 - 2016) is that it gives you very little statistical information going forward to use to measure the health of your system.


----------



## jjbinks (25 September 2016)

thanks for your help so far.

I'm just having trouble making sense of the walk forwards results. 

I have set it to optimise over 4 years and then do OOS testing in the following year.

Do decide on whether the system is worth using based on how many OOS years are profitable?

In terms of parameters, if I do decide to go ahead with the system then would I work out the parameters from the last 4 years uptil now?


Thanks again


----------



## captain black (25 September 2016)

jjbinks said:


> thanks for your help so far.
> 
> I'm just having trouble making sense of the walk forwards results.
> 
> ...




The decisions I make about how robust the system is likely to be going forward are based on a number of different types of statistical analysis, but one of those is the variations between my IS and OOS sample results. I measure the spread of the results using variance and standard deviation and compare these values historically to how my system is performing going forward.


----------



## captain black (25 September 2016)

jjbinks said:


> In terms of parameters, if I do decide to go ahead with the system then would I work out the parameters from the last 4 years uptil now?




Apologies, missed this question.

Once you've established the optimal IS and OOS time period then the parameters to use going forward are from your most recent IS test. A walk forward test is simply a representation of what would happen in real trading.


----------



## captain black (25 September 2016)

I'll also add that for IS testing I prefer to use the CMAE optimization engine rather than an exhaustive search. It tends to find broad plateaus in its results rather than singular "best result" data points.

My early testing on systems when I first started using Amibroker was with Particle Swarm Optimisation using Fred Tonetti's IO addon. A PSO optimisation engine is also available to use natively in Amibroker now but I've found the CMAE to be my favourite weapon of choice


----------



## Wysiwyg (26 September 2016)

To me, any period not in the system build is OOS. I test if the strategy is over optimised by doing manual runs (no parameter optimising as in the built in walk forward)) over any period outside the test period and look at what the Index was doing in the periods of drawdown. The drag on long only stock systems by a range bound or downtrending Index is frustrating. Obviously stocks rise in these sub optimal periods but your code still has to pick them.
However I am cynical of the backtest scene. A lot of fancy tricks for a rough idea of how things might go if the market conditions suit the system from the present moving forward.

This link explains what I mean. 

http://www.futuresmag.com/2015/09/01/dont-build-better-backtest


----------



## captain black (26 September 2016)

Wysiwyg said:


> This link explains what I mean.
> 
> http://www.futuresmag.com/2015/09/01/dont-build-better-backtest




Data mining and curve fitting are certainly 2 important factors to consider in the health of a system. That gets back to my point earlier to the OP about having a statistically significant number of data points in the IS and OOS testing. If the variance between the IS and OOS data is increasing over time then you need to look at whether the system is being curve-fitted or it's broken or a number of other factors. A single WF run wont provide that data. 

The next "WF" run is using real money, I'd rather have confidence in a system's health by using statistical techniques than have it shown to me using real money.


----------



## Boggo (26 September 2016)

Wysiwyg said:


> This link explains what I mean.
> 
> http://www.futuresmag.com/2015/09/01/dont-build-better-backtest




For system design and backtesting etc Davey's book is worth the price tag imo 

https://www.amazon.com.au/Building-Winning-Algorithmic-Trading-Systems-ebook/dp/B00JUUZU2E

https://booko.com.au/9781118778982/...rlo-Simulation-to-Live-Trading-Wiley-Trading-


----------



## captain black (26 September 2016)

Boggo said:


> For system design and backtesting etc Davey's book is worth the price tag imo
> 
> https://www.amazon.com.au/Building-Winning-Algorithmic-Trading-Systems-ebook/dp/B00JUUZU2E
> 
> https://booko.com.au/9781118778982/...rlo-Simulation-to-Live-Trading-Wiley-Trading-




Nice find Boggo, I've got most of Howard Bandy's books related to system design and statistical validation but it's always good to see it from another perspective.

I notice the charts you post in other threads are in Amibroker now rather than Metastock, hope the transition went smoothly 

I liked this quote from the Amazon link to the book:



> Market patterns change, and so do system results. Past performance isn't a guarantee of future success, so the key is to continually develop new systems and adjust established systems in response to evolving statistical tendencies.


----------



## Gringotts Bank (26 September 2016)

Wysiwyg said:


> However I am cynical of the backtest scene.




I trust the process, but with equities I found I massively underestimated the effect of slippage.

An equity backtest which returns 100%pa (brokerage included but not slippage) will return about 30%, in real life trading - that's my experience anyway.  This is with a liquidity filter included.


----------



## Boggo (26 September 2016)

captain black said:


> Nice find Boggo, I've got most of Howard Bandy's books related to system design and statistical validation but it's always good to see it from another perspective.
> 
> I notice the charts you post in other threads are in Amibroker now rather than Metastock, hope the transition went smoothly




I've got Bandy's books too but in some areas they get a bit too scientific for me but that is me rather than the books.
Bandy's 'Foundations of Trading' is a nice concise summary of his work though and in some ways is similiar to Davey's book.

Took me a while to let go of Metastock. I spent many late nights learning how to get the best from it and was reluctant to give it up. With some prompting from many people I went with Amibroker and am sorry now that I didn't do it years ago.
Just the backtesting and optimisation alone leaves Metastock for dead.

Cheers


----------



## captain black (26 September 2016)

Boggo said:


> Bandy's 'Foundations of Trading' is a nice concise summary of his work though and in some ways is similiar to Davey's book.




I'm still working my way through it but from what I've read so far I agree that it would be a great read for someone who's looking for a concise summary of statistical analysis/modeling of trading systems. I promised Howard a review of the book so I'd better finish it soon!



Boggo said:


> With some prompting from many people I went with Amibroker and am sorry now that I didn't do it years ago.
> Just the backtesting and optimisation alone leaves Metastock for dead.




Yeah, I started out with EzyCharts in the beginning and when I was looking for something better I'm forever thankful I went down the Amibroker path.


----------



## Roller_1 (26 September 2016)

Gringotts Bank said:


> I trust the process, but with equities I found I massively underestimated the effect of slippage.
> 
> An equity backtest which returns 100%pa (brokerage included but not slippage) will return about 30%, in real life trading - that's my experience anyway.  This is with a liquidity filter included.




What sort of filters were you using??


----------



## Gringotts Bank (26 September 2016)

Roller_1 said:


> What sort of filters were you using??




They vary a bit, but say I want to take a $5k position, I'd want daily turnover of at least $300k.  $10k->$500k and so on.  I know the ratio is too big, and that's the reason for the slippage.  As you'd be aware, a stock with a daily turnover of 500k can easily turnover 50k the very next day with a huge widening of the spread thrown in for good measure, so...  that's the risk you take trading outside the top 20.


----------



## Roller_1 (26 September 2016)

Gringotts Bank said:


> They vary a bit, but say I want to take a $5k position, I'd want daily turnover of at least $300k.  $10k->$500k and so on.  I know the ratio is too big, and that's the reason for the slippage.  As you'd be aware, a stock with a daily turnover of 500k can easily turnover 50k the very next day with a huge widening of the spread thrown in for good measure, so...  that's the risk you take trading outside the top 20.




You could limit slippage by using the opening auction for buy/sells or just limit orders. I have a few partial fills but nothing too drastic, using similar filters


----------



## jjbinks (28 September 2016)

thanks guys.
Definitely cleared up some confusion I have. 

Shall continue my search for a good strategy!


----------



## Wysiwyg (25 December 2016)

A simple way to check the usefulness of a an entry signal is to create a shape for the signal on the chart, set your chart back several periods and see what happens after each signal. Saves running unnecessary backtests by quickly showing the ensuing behaviour.


----------



## jjbinks (26 December 2016)

Once again thanks for the suggestions guys
Just an update since when I first posted.  ( I feel as though I have learnt something since then!  )

I have read a few different books since i first posted this (including Daveys book suggested by boggo).
I have bit more understanding of back testing and the type of testing depends on the data your have and your strategy. 
Wysiwyg i do something similar to what you suggest for screening entry signals and exit signals.
I also do it in a automated way by assessing the performance of a specific entry combined with exit at specific time periods or random exits. (similar for exits). I also am comparing my entry/exit performance to random strategy.
I've been struggling to find exit strategies which work better than simple stop loss strategy. 
However I have taken a break from testing ideas to work on my programming skills. In particular learning python for data science at the moment. (Not sure that I will use it for trading strategies in the forseeable future but I did want to update my python knowledge).


----------

