Out of sample/Forward testing?

[anthon]

As I am new to all of this, I am having trouble finding a decent explanation of the best way to perform out of sample and forward testing?

Say I tested my system from '90 to '01, how would I then do effective out of sample and forward testing?

Would out of sample be small periods outside of my originally tested range? And is forward testing from '01 to say '08?

If so, how then do I look at the different sets of results to make a decision? Should a good system have results that are closely correlated? If not, what?

Thanks for your help.

 

[tech/a]

Yes.

You could use smaller data sets but of course enough to make the results meaningful.
You could also use other bourses if the system is a stock trading system.

You'd be looking for the system to perform within those parameters obtained in your in sample testing. Anything vastly different above or below the high and low results obtained in your insample testing would be viewed as suspicious.

 

[julius]

The way to think about out-of-sample testing is that it represents the same results you would have got if you had traded the system live over that period.

The out-of-sample period should be long enough that it you can draw meaningful information from the results, which will be a function of how frequently your system makes trades. If it's a longer term trend following system, you'll need many years, where as around 12 months is probably fine for short term systems.

 

[Stormin_Norman]

Would out of sample be small periods outside of my originally tested range? And is forward testing from '01 to say '08?

out of sample would be using historical data for any period outside your initial tested range.

forward testing is testing it on current and future data via paper trading.

(at least that's how the jargon works in foreign exchange circles).

 

[Chorlton]

As I am new to all of this, I am having trouble finding a decent explanation of the best way to perform out of sample and forward testing?

Say I tested my system from '90 to '01, how would I then do effective out of sample and forward testing?

Would out of sample be small periods outside of my originally tested range? And is forward testing from '01 to say '08?

If so, how then do I look at the different sets of results to make a decision? Should a good system have results that are closely correlated? If not, what?

Thanks for your help.

Choose a period of time that would offer statistically useful results to assess the strategy and divide your OS (Out-Sample) data into these time periods. This would be determined (amongst others things) by the avg holding time for a trade, # of trades encountered during this time, etc.
If for example, "1 year" was chosen you would then have 8 seperate OS blocks to test against. (01-08) If after a particular testing block the strategy needed to be "tweeked" you would still have further OS blocks to test against. Whatever way you decide to split up your OS data, be aware that if you do "tweak" the system then any OS already tested against will become IS (In-Sample).

Its also worth noting that the latest OS data will always be more useful for assessing the reliability of a system as its closer to the current time period. Therefore, IMO, it is ok to "tweak" a system as you move through the different OS data blocks (towards current day). Depending on how close to current day you have "tweaked" the system, and you have no more OS data available , you may want to use paper trading in real time to offer some further level of confidence as to how the system is performing.

As a side note, there is a lot written that a system should perform equally well on multiple mrkts. Although logically this does make sense, I wouldn't always assume that a strategy was suspect if it didn't as, IMO, different Mrkts can (and do) exhibit different characteristics.

 

[Wysiwyg]

I have been experimenting with the Amibroker Walk Forward Optimiser and there a few things that aren't clear (well more than a few .

One is the equity curve reveals the out-of-sample data mirrors the in-sample data curve. The difference being the $$ value is less with the out-of-sample. If this normal then it is safe to say the only information gained from the Walk Forward test is that out-of-sample test returns less. Let me know if anyone has a higher or not mirrored out-of-sample than in-sample equity curve.

Now with the optimisation runs I get a table of results from each IS, OS pass. Good information but I don't see any optimisation advice. For example I added the optimisation code and from this code I thought the Walk Forward Optimiser would select the best parameter value from the minimum to maximum range and provide the best values in the table of results. I don't understand what or how anything is optimised by the program.

 

[Wysiwyg]

I notice Howard mentions regularly that in-sample is not as good as out-of-sample data testing. I thought I would try this set of indicators, position size and money management rules that looked good in a bull market on the worst local market action in recent times. The All Ords. from May 2008 through to March 2009 was in capitulation phase and the steepest decline of the bear cycle. The out-of-sample equity curve managed to survive and is an encouraging step toward paper trading real time.


Period = Daily
Position = Long // 4 maximum open at any time with equal distribution
Initial equity = $10,000
Commission = $40 // much more than I will be paying
Trade Entry/Exit = day after signal at opening price

 

[howardbandy]

Greetings all --

I have made several posts related to trading system design, testing, and validation, including in-sample and out-of-sample, and walk forward. Search ASF using my name.

Thanks,
Howard Bandy