Is there a point to backtesting?

Wysiwyg said:

I am very much in the learning stage of this but the idea of taking a ten year block (or longer) and breaking that up into

1) full ten year backtest
2) two, five year backtests
3) three, three and a third year backtests
4) four, two and a half year backtests
and
5) five, two year backtests

All the results could be assessed as to performance over varied durations.

I`m still skeptical about testing what has already gone and could really only use results as a guide to follow as opposed to discretionary execution, based on the here and now.

Click to expand...

Just make sure that if the purpose of backtesting is to assess the performance of the System (and to give you confidence in it going forward) make sure that the data you use is Out-of-Sample.

Otherwise, the results are meaningless.

Chorlton said:

Just make sure that if the purpose of backtesting is to assess the performance of the System (and to give you confidence in it going forward) make sure that the data you use is Out-of-Sample.

Otherwise, the results are meaningless.

Click to expand...

Yes it would be worth doing the same test procedure on another 10 year block outside the first. Unlimited alternatives to these.

There are plenty of bourses around if designing a system for stock.
I have 7 data sets.
You can of course test those as well.
In sample/out of sample/Montecarlo.
Should work in all.

PolarBear said:

thanks for the replies.

Howard, i think you hit the nail on the head with what i was after. I've been trying to work out how far back to backtest and then test the algorithm on (as you say) out-of-sample data. BUT, the key point you made was to train the system on in-sample data, and then test how good it is on out-of-sample data that is after (MORE RECENT data) than the in-sample data. This does sound obvious now, but something i overlooked. thanks.


TH, yes i'm trying to make a short term system. My goal is something that trades maybe once or twice a day that can consistantly make around 4 points profit per week.... that's the dream anyway.

cheers,
Daniel

Click to expand...

Hi Daniel -- If you are looking to trade once or twice a day, you will get 300 or so trades per year. Depending on your system, and what patterns it is looking for, and how much back data it needs to get synchronized, you probably only need two or years or so of intra-day bars. You can determine how much data is too little and how much is too much by some single backtest runs. If your system is stable, the length of the in-sample period should remain fairly constant, but you can check several time periods a few months apart to see if it appears to be the same. Once you know how much in-sample data you need, run automated walk forward tests to see whether the system trades well out-of-sample. Be sure the objective function you use is one you understand and like.

Thanks,
Howard

Tradesurfer said:

Polar Bear,

Exact time vs conditions presented are two different things. To explain, say I tested a long/short weekly trend based system.(actually used one) The system showed a short entry December 2007 on the s&p 500 and then closed out the short a few months ago and went long.

Click to expand...

Hi Derek --

That was an excellent trade. One comment I would make related to systems like yours, backtesting, and validation is that it is almost impossible to get adequate out-of-sample data to statistically validate a trading system that holds positions for 18 months. So tread very cautiously.

Thanks,
Howard

Wysiwyg said:

I am very much in the learning stage of this but the idea of taking a ten year block (or longer) and breaking that up into

1) full ten year backtest
2) two, five year backtests
3) three, three and a third year backtests
4) four, two and a half year backtests
and
5) five, two year backtests

All the results could be assessed as to performance over varied durations.

I`m still skeptical about testing what has already gone and could really only use results as a guide to follow as opposed to discretionary execution, based on the here and now.

Click to expand...

Hi Wysiwyg --

All the backtests you are describing are in-sample. In-sample testing is important in order to determine the best logic and parameter values. But in-sample results have no value in estimating the future performance of the system. For that you need out-of-sample tests.

A better use of ten years of data is seven runs. Each uses three years of data to pick parameters, then tests the following one year. Step the whole thing forward one year and do it again. Seven tests, each giving one year out-of-sample results. Analyze the concatenated out-of-sample results to see if the model learned the data and the system is tradable.

Thanks for listening,
Howard

Howard......

please...

You mention doing 3 years and overlapping...... and I assume that this is the approach that you have developed in your experience.

My question is..... is the three years to allow for the one year overlap.....? and. if you had data for 20 years. would you also use the three year method..............?

I am very new to this and am attempting to grasp the 'in sample/walk forward etc' concepts.

Thanks...

daveM said:

Howard......

please...

You mention doing 3 years and overlapping...... and I assume that this is the approach that you have developed in your experience.

My question is..... is the three years to allow for the one year overlap.....? and. if you had data for 20 years. would you also use the three year method..............?

I am very new to this and am attempting to grasp the 'in sample/walk forward etc' concepts.

Thanks...

Click to expand...

Hi Dave --

I used three years as the length of the in-sample period and one year out-of-sample as examples. The amount of data and the length of time for your system is something you will have to determine yourself -- there is no single guideline.

When I say system, I am describing a model -- the mathematical algorithm or pattern recognizer -- in combination with the data -- the price and volume of the issue you want to get your buy and sell signals from.

The in-sample period and data is used so that the system can get itself synchronized -- the model "learns" to recognize the pattern you have coded into the trading system program. But that data consists of both signal -- the part that you want it to learn -- and noise -- everything in the data that you are not explicitly looking for. Your hope is that the model will recognize the signal and adjust its parameters while processing the in-sample data, and that the signal portion of the data continues to act the same way for a long enough period of time for you to make money trading in the out-of-sample period. If you do one in-sample / out-of-sample test, you get one data point -- where the data point is the transition from development to trading. When you run a walk forward test, you set up a sequence of IS / OOS runs, each one of which gives you an example of what happens when you move from development to trading -- each of those transitions is a data point. The more data points you have, the better able you are to estimate the robustness and profitability of your system.

So -- it is up to you to describe your idea to yourself, code it into your trading system development platform (AmiBroker, MetaStock, ...), work with it to determine how long the ideal in-sample period is. After you have figured that part out, run an automated walk forward test and evaluate the likelihood that your system will be profitable when you trade it based on the out-of-sample results. Walk forward testing is the equivalent of "blind" testing in pharmaceutical development. I know of no other way to gain the confidence needed to make real trades with real money.

Thanks for listening,
Howard

Thanks for that, Howard.

Past Performance does not guarantee future performance... guess there's only one way to find out.

I think we all have one of those.

tech/a said:

I think we all have one of those.

Click to expand...

Smokin`

.