Backtesting Mistakes

[Tanaka]

For my own benefit and anyone backtesting I thought it would be helpful for people to share their mistakes while backtesting. A few things I can think of:

1. Not including slippage
2. Not including transaction costs
3. Including code or indicators that were reading into the future
4. Not taking into account interest costs
5. Or maybe like me you forgot to close your yearly positions at the end of the backtest not realising the system was heavily in debt. (i wasn't silly enough to live trade it)

I have been using TradeSim to backtest for a while now. It doesn’t matter how you backtest, what mistakes do you know of? Or what things should you look out for?

Cheers,
Tanaka

 

[tech/a]

The biggest is Taking trades on the day of signal in EOD systems
Which is not possible you have to set your buy and SELL to be delayed by one bar.
(You cant be sure your signal is triggered until the close of the bar)

Can make massive differences---enough to go from profit to loss!

 

[Tanaka]

The biggest is Taking trades on the day of signal in EOD systems
Which is not possible you have to set your buy and SELL to be delayed by one bar.
(You cant be sure your signal is triggered until the close of the bar)

Can make massive differences---enough to go from profit to loss!

Excellent point tech! it took me a couple of months until I realised I was doing that in my tests.

 

[sinner]

The biggest is Taking trades on the day of signal in EOD systems
Which is not possible you have to set your buy and SELL to be delayed by one bar.
(You cant be sure your signal is triggered until the close of the bar)

Can make massive differences---enough to go from profit to loss!

My backtesting tool (ProRealTime) has a parameter to take historical trades "NextBarOnOpen" based on entry and exit signals...weird that yours doesn't do that?

e.g.

IF condition1 THEN BUY 1 %CAPITAL AT MARKET NextBarOnOpen SET STOP Close-(Close*0.05) ENDIF
IF condition2 THEN SELL AT MARKET NextBarOnOpen ENDIF

From my own experience re backtesting in the 24h markets.

Use the highest resolution data that you can get. Some softwares will *assume* you get the fill you want at the lot size you want, and the stop you want. Using M1 or tick data will ensure your system is testing against the right order flow.

Secondly, as a geekish type, I advise all traders to aim for simple robust concepts to be backtested. The harder it is to code, the less likely your results will be reflecting reality. Each portion of the trade (e.g. scan X bars back, identify setup, identify trigger, MM+RM, place trade, enter trade, manage trade, exit trade) should be delineated, simple, clean code that can be tested individually to ensure it works as expected across a wide range of market activity. If you are relying on the machine to do math that you can't do on your own then you shouldn't generally be using that math. e.g. are we actually referring to a 19 and 39 period EMA? Or are we referring to a 10% trendline and 5% trendline?

I refer to this specifically

We prefer not to refer to EMAs by a number of days, since this is somewhat misleading. With a simple moving average (SMA), the old data does disappear from the calculations. So if you have a 50-day SMA, the price value from 51 days ago does not matter at all, while the one from 50 days ago matters just as much as the one from 1 day ago. With an exponential moving average, the effects of the old data never completely disappear, they just become progressively less relevant.

We prefer instead to identify EMAs by their tracking rates, and so we use the terminology originally assigned by the late P.N. Haurlan. In the 1960s, Haurlan was the first person to write about using exponential moving averages for analyzing stock prices. He was an actual rocket scientist who worked at JPL in the day time and then analyzed stocks at night. For more on Haurlan, see 2004 MTA Booklet.

http://www.mcoscillator.com/learnin...illator/Calculating_the_McClellan_Oscillator/

EMAs, stochastics, standard deviations, linear regressions, splines, etc. If you don't know how or why these work, why on or earth would you use them? I know tech likes the "40 EMA" and "180 EMA", but shouldn't these be set to 39 and 179?

For example, take a look at the well known Japanese "systems", like Kagi, Renko, Heiken Ashi, Ichimoku, Three-Line Break, etc. They all break the price data out in a way that you could calculate in your head if you needed to yet none of them suffer from lack of a 24 period stochastic filter.

Robust + Simple = relatively reliable test results.

 

[Tanaka]

My backtesting tool (ProRealTime) has a parameter to take historical trades "NextBarOnOpen" based on entry and exit signals...weird that yours doesn't do that?

I'm using Bullcharts with Tradesim. You just need to remember to tick the box that says delay entry by X amount of bars. Silly me thought of course it would be ticked if I'm using EOD.

 

[Synergy]

A few ideas from my experience...

1) Partial Fills - In the region of the market I trade, partial fills are a big deal when backtesting, and happen regularly. In the lower priced stocks - particularly stocks just above 10c or under 2c - the depth at each price is likely to be massive. The backtest will often buy or sell at a price where in reality, you'd be left waiting in the queue.

2) Zero volume days - From what i can work out some data lists zero volume days and some doesn't. This alone is an issue. Something to watch is that you're not buying or selling on a zero volume day. You may be able to do it trading live, but probably not at the price your data says you can (the previous close).

3) Deceiving equity curves - I think it's best when designing a system to look at equity curves without compounding. Sticking with a fixed $ position size (eg. $10k, not 10% of pool) to create the curve lets you view each trade relative to the others. You don't want a 40% drawdown early on looking like a 1mm blip at the start of the curve.

4) Relying on outliers - No use if a system is profitable from only a few trades. Take the top 5 or 10% of trades out and see if it's still profitable.

5) Using Averages - Averages can be very misleading. They sometimes don't tell the story they are being used to tell. Otliers in either volume or price can distort averages hugely, especially on small timeframes. You might want to buy if average volume is above 100k - a 10 day vol average might not show that there was no volume for 9 days then 1M on the 10th day.

 

[Tanaka]

4) Relying on outliers - No use if a system is profitable from only a few trades. Take the top 5 or 10% of trades out and see if it's still profitable.

True, I am doing a large backtest from 1985 until now. PDN accounted for most of the profits, due to aggressive pyramiding it returned over 8000% in about 10 years!
took all those trades out and the system is mildly successful.

 

[TulipFX]

Make sure you have good lot sizing. I like testing with lot sizing because it gives an accurate drawdown value which stays relative to the balance. A constant lot size can under report the drawdown having you believe the system is safer then it is.

Watch out for data interpolation by the software. Use only the highest quality data is you are trading small price movements on lower timeframes.

Do not curve fit your data. If you optimise your variables to exact numbers and run your system on unseen data, you will be surprised at how poor it performs.

Swaps on forex. Sure, holding that position for 3 months seems fine in testing to gain a small profit finally - but with swaps and having the position sit there for months the profit will be gone, and so too will be part of your margin.

 

[AlterEgo]

The biggest is Taking trades on the day of signal in EOD systems
Which is not possible you have to set your buy and SELL to be delayed by one bar.
(You cant be sure your signal is triggered until the close of the bar)

I have to disagree with you, at least to some extent, on this point Tech. It's quite simple to calculate, before the close, what closing price will be needed to generate a signal for a particular stock. You can then place an order in to the market during the closing auction to buy or sell at the close.

Also, if I were trading a breakout system, I don't need to wait for the close to know that a new high has been made today, so I can set a buy stop order to buy on the break. Likewise, if I set a stop-loss on a stock, I don't have to wait for the close to generate a sell signal for the following day's open, it sells immediately it's triggered intra-day. Perhaps you don't consider this to be an EOD system, but just thought I'd mention it.

 

[TulipFX]

The end of day and market liquidity inefficiencies in stocks must be quite an issue to overcome.

I am glad these problems are much less severe in forex.

 

[Tanaka]

I just backtested a CFD system, the results were impressive. Then my bubble burst when I realised my data range would be giving the incorrect result. For example, as I was interested in CFDs I choose to backtest ASX200 shares. My sample period was from 1985 until now. The stocks tested were from the ASX200 today, a lot of them weren’t even in the ASX200 that far back, actually I don’t even think they called it the ASX200 in 1985 did they? I was only a little boy back then

I guess it is difficult to backtest a CFD system with a lot of data. The system is still quite good if tested over the last 3 years but I'm not sure if that's enough data to go by.

Any thoughts?

 

[Gringotts Bank]

that's called survivorship bias tanaka, in case you want to research it.

 

[Tanaka]

that's called survivorship bias tanaka, in case you want to research it.

Thanks Gringotts! Remember skimming past it in the manual, will have to follow up on it.

 

[Wysiwyg]

2) Zero volume days - From what i can work out some data lists zero volume days and some doesn't. This alone is an issue. Something to watch is that you're not buying or selling on a zero volume day. You may be able to do it trading live, but probably not at the price your data says you can (the previous close).

Quite true as I have mentioned elsewhere. An Amibroker feature is limiting the percentage of volume available to buy. I use 10% as recommended in the docs.

4) Relying on outliers - No use if a system is profitable from only a few trades. Take the top 5 or 10% of trades out and see if it's still profitable.

I wouldn't consider being a "back testing" expert but my most profitable back tests were ones that had outliers. The reason I believe is because the capital is tied up in that one trade so trading frequency is reduced thus risk per trade is reduced. Considering (from my experience) win/loss ratios generally range from 40 to 60 plus or minus, it is the outliers that make a system. My short hold/profit strategies trade more frequently but the brokerage (adding slippage) kills any promise.

I have to disagree with you, at least to some extent, on this point Tech. It's quite simple to calculate, before the close, what closing price will be needed to generate a signal for a particular stock. You can then place an order in to the market during the closing auction to buy or sell at the close.

Yes manually this is possible by placing the order relative to stock available but very unreliable to be coded into a buy since the whole daily volume is the only data for an EOD system to work with. Come to think about it, buying next day isn't reliable either because whole day volume is used their too.

Question :- Does intraday data have "volume traded" for every time frame? I.e. 1 minute or 1 hour?

 

[tech/a]

Question :- Does intraday data have "volume traded" for every time frame? I.e. 1 minute or 1 hour?

Yes

On outliers. Particularly positive ones.
Stops and trailing stops will hopefully eliminate to a degree negative outliers.
But I want to see as many positive outliers captured in my trading system.
They really are where the money is.

But agree that a system has to operate and smooth equity curves "without" the top 5 outliers at either end with 100% profitability is the aim.
This is the engine of the system---what keeps it in play until a money making outlier/s arrive.

It's quite simple to calculate, before the close, what closing price will be needed to generate a signal for a particular stock.

Yes your right you can.
But if you sell intra day (and you can set your systems testing up to do it if you have good software) you run the risk of the close being above your triggered sell and as such your still in the trade. but I will agree that you can set it up either way.
The problem is those who dont set it up on an EOD systems test and enter and exit on the trigger bar.
It distorts the results badly. They get the perfect fill when in reality its not possible---unless manually.

Survivorship is a biggie and not understood by most.
I doubt many would have perfect data back 10 yrs---listing splits and delistings,name and code changes,mergers.--let alone 20 yrs +

Another is a single run systems test.
If you Dont have the software to do multiple run (Montecarlo simulations) you could test 5000 portfolios only to find 1 (the one you think is worth trading) profitable.

 

[Gunslinger]

The biggest is Taking trades on the day of signal in EOD systems...

When you actually use a physical stop loss though, should the sell delay be zero?
Maybe factor in some slippage though but most times it should be liquidated before close.

Thoughts?

Cheers,
JB

 

[asxiq]

my 2 cents
1) one trick that i learned is to run the EOD trading systems is to front-run the trading strategy just before ( say for example, one minute before the close , or 1 second before the close if the backtesting platform can do the calculations , and send the Market on close order if the trading system is signal is generated)

2) the same can trick be employed for intraday trading , for example the current high of AORD is around 4250 at say 10.30 and assuming one is employing a breakout system which might say buy above prev high ( for example prev high is 4275 ), one could tweak the OHLC data and send a usually a H+0.5% ( or L-0.5%) in to the trading system to see if it generates in signal in the armour. once if signal is generated if could employ a buystop order

 

[tech/a]

While that could work I found hat itall washed in the end.
Often the buy was better in the morning and the sale better in the following open
As well as others being worse off.

 

[Billyb]

If you paper trade your system for a while there's a good chance you'll pick up on some errors.

But basically what I've found is if you understand the programming language and how it works and you understand the limitations of programming your trading ideas then chances of errors are much lower because you will pick those mistakes up when you 'spell check' your code.

The biggest is Taking trades on the day of signal in EOD systems
Which is not possible you have to set your buy and SELL to be delayed by one bar.
(You cant be sure your signal is triggered until the close of the bar)

Can make massive differences---enough to go from profit to loss!

I'm always surprised at how much BETTER this simple mistake makes your system look. One day can make a huge difference. Very important point.