Curve fitting theory and data samples

[AMSH]

Hey All,

I finished my third live trading system about a year ago and have been trading it with some success for the past six months. However, when I first starting designing and testing it I was still learning the ropes and as such, I ignored in-sample and out-of-sample data periods. So in essence, the system was tested over the entire data set available. I’ve just read Howard Bandy’s Quantitative Trading Systems and as a result I’ve lost the nerve to trade my current system. To be honest, I was using discretion to pick between trades and sometimes ignoring trades if they looked too volatile anyway so perhaps my confidence in the code wasn’t 100% in the first place. My question however, is this: are there situations where it’s acceptable to not have in and out-of sample periods.

I can’t see how the system I’m trading can be curve-fit and I was hoping somebody could enlighten me. I say this because of the following:

• The entry code is relatively simple; there are no complex functions or indicators used and there are relatively few actual entry conditions
• The system produces a large number of trades (about 400 a year, 4,000 over the last ten)
• The holding period is about 2 weeks per trade
• It returns a (genuine, I think) 35% per year and is VERY realistic (overly harsh slippage on entry and exit, accurately modeled brokerage, very restrictive volume and liquidity filters, no forward looking etc)
• The equity curve is consistent in slope over the past decade, with no major draw downs (11% ish max DD) or volatility (this is partly due to the large number of trades; I can limit position sizes and risk and I think this has a major effect on the overall DD)

I don’t see how it can be dangerous to trade this system and I’d like other people’s opinions (especially yours Mr Bandy if possible). I understand I can wait six months and walk-forward test or perhaps alter the conditions a bit and see if there’s a major change in results, but I’m more interested in the theory of it. If something produces a consistent profit over a sufficiently large number of trades and your equity curve is smooth, I don’t see how it can be curve fit.

To me it seems like if the frequency of trades is high enough over an extended period of time, there wouldn’t be a single static piece of code that would fit that data set well enough to produce consistent performance.

Am I wrong?

By the way, I definitely recommend the book. Nice to see somebody apply some analytical rigor to an industry/genre that seems to be full of people spouting unfounded BS.

Any comments appreciated.

 

[skc]

Not a systems expert but a couple of thoughts come to mind.

How many variables to you have? If you have 1 or 2 then the possibility of curve fit is greatly reduced. If you have 5 to 8 then watch out.

Can you get some data on other markets to test? E.g. US/UK markets, futures etc?

Can you articulate in words, what the system is doing? i.e. what is the edge if you want to tell somebody what your system does? Or is it just a forumla that happened to work?

 

[tech/a]

I've never tested any of my systems with out of sample data from the same bourse for the same reason you did.
I also developed most of mine well before I read about out of sample.
Since then all have been tested walk forward in R/T and my first 8 yrs ago has 8 yrs of data to look at that it wasnt tested on.

My comments are
(1) Try other bourses with similar stock types.
(2) Wouldn't worry about it if your live trading is within the limits of your test results.
(3) Even testing out of sample data isn't fail safe.All you need to have happen is the data your trading going forward is different enough to the in and out of sample data to cause the system to trade outside of its blue print!

Have you Montecarlo tested the systems to get a better idea of the upper lower and median values of your systems---that way you can judge a little clearer how the system is performing live.

 

[Gringotts Bank]

Just alter your entry parameters slightly and see what happens to the equity curve. If it falls apart, it's not robust.

I believe OOS testing is vital if you're using any indicators as entry signals, particularly if they have been optimized. But I doubt you're doing that. If you're using more 'organic' entry signals, and your equity curve is very even and smooth, I agree OOS is not necessary.

You do realise that you need not wait another 6 months? Just use the data you already have. OOS is an optimization process, so you can just ask your software to test 6 months, then OOS test 6 months on whatever data you already have.

 

[Gringotts Bank]

Even easier, just take all your entry parameters and run an optimization on all of them together. You'll quickly see how parameter tweaking affects profitability, DD, SD etc.

 

[colion]

I've seen systems with and without OOS testing do both well and fail. Bottom line watch the equity curve and when it does not meet your requirements optimize your system. That sequence works quite well.

 

[AMSH]

Thanks for the replies guys.

Skc, the system is essentially a trend follower but looks for a specific type of trend behaviour. The code’s made up of 4 filters (volume and index based) and 4 similar variables (again, fairly straight-forward code).

I’m aware of ways to get around the problem, I’m just after people’s opinions on how important you think it is during system design. A lot of the more technical stuff I’ve read on systems design implies it’s an absolute necessity for all types of systems design, but this doesn’t fit with what I’d expect.

I can understand it if your trying to trade a 20% win rate, 14 avg R/R system with a trade log of 200 trades – obviously small changes in the code that result in the trades changing could have massive effects on the outcome. In addition, the more restrictive/exhaustive the code (and the lower the trade frequency), the easier it is to find a single bit of code that will fit the data set very well.

But again, I think the relevance of OOS testing would be directly related to the number of trades in the log, the type of system you’re trading, the win rate and RR, and the smoothness of your equity curve – it doesn’t seem as clear cut as “you must always use OOS testing regardless of the situation”.

Tech, the system’s been Monte Carlo tested. In live testing it’s been trading well within the bounds expected. The main problem I’m having is that it generally buys as a counter-intuitive (read: scary) point in a trend – and this means you need to have a lot of confidence in it. I’m going to do some more testing over the coming weeks on different markets and might play about with the parameters to see the effect.

Hey GB, I realize I don’t have to wait six months but because I’ve tweeked the thing based on current data, I’d need six(?) months of fresh data to consider it OOS testing.

Thanks for the input guys. I’ll let you know how I go.