Would you trade this system?

Hi Tech/a --

You wrote: "I would argue that out of sample testing of models offers no guarentee that the in sample test results will perform any better or worse than the results returned within the parameters of those results obtained through multiple or Montecarlo testing going forward in realtime."

I may not be interpreting your statement correctly, but I'll make two comments that might apply.

1. The walk forward process and Monte Carlo analysis are complimentary, not exclusive. It is very common for Monte Carlo techniques to be applied during a system development that is using the walk forward process.

2. As I wrote in a post just a few minutes ago, there are no guarantees. The best that we can hope for is to develop confidence that our system will be profitable.

The points I am making and the techniques that I am describing throughout my book, my speeches, my workshops, and my posts to this forum are intended to help system developers and traders produce systems that are likely to trade profitably.

I hear comments about the difficulty of doing what I suggest, and I hear suggestions to the effect that reliable systems can be developed without out-of-sample testing. But there will be an out-of-sample test. My point is that using the techniques I describe, I can make a series of confidence building out-of-sample tests without risking real money before the marketplace makes its out-of-sample test.

Thanks for listening,
Howard

howardbandy said:

The system is being given more and more data representing more and more market conditions and is less and less likely to be tuned to the current conditions. One of the benefits of using the walk forward process is being able to tune the system to the current conditions.

Click to expand...

Wouldn't more data and more market conditions help ensure that the system isn't optimised to only the recent market conditions and potentially build a more robust system? If the test data universe is in a strong uptrend then the system may not handle a correction all that well if it is only tuned to recent market data.

There is a big difference optimising on a single stock with a year's worth of data versus optimising on 500 stocks over a 3 year period - I know what testing approach I prefer - more data and more market conditions.

Is there some sort of balance required in terms of testing - optimised to current conditions, but not too optimised. It is probably not possible to design a system that will always work, but it is possible to turn a system off / on if market conditions change.

Having said all that I do like rolling walk forward testing. Are there any rules in terms of what sort of walk forward period to use - if a system has a 3 month average trade length I would think that the rolling test period would have to be longer than 3 months and probably closer to at least 1 year.

stevo said:

Wouldn't more data and more market conditions help ensure that the system isn't optimised to only the recent market conditions and potentially build a more robust system?

Click to expand...

Hi Stevo --

In my opinion, No.

The market being modeled is dynamic -- the model of that market is static. As long as the market remains in or near the state it was in in when the model was built, the model will produce profitable buy and sell signals. When the market changes, the signals will degrade.

It is very difficult (impossible?) to build a model that works over all market conditions. The purpose of having shorter rather than longer in-sample periods is so the model can be tuned more precisely to the state of the market. The purpose of the walk forward process is to keep the model in tune with the then-current state of the market.

Thanks,
Howard

Greetings all --

I have made quite a few posts to the thread on "System Robustness" as well as to this one. I am starting to repeat myself.

No offense intended, but please forgive me if I do not make new postings where one of my earlier postings has already expressed my view.

Thanks,
Howard

howardbandy said:

Greetings all --

I have made quite a few posts to the thread on "System Robustness" as well as to this one. I am starting to repeat myself.

No offense intended, but please forgive me if I do not make new postings where one of my earlier postings has already expressed my view.

Thanks,
Howard

Click to expand...

Howard,

I am impressed with your patience so far. I thought the same.
Thanks for your thoughts and opinions. Maybe some should read your book then comment later.
Regards
Snake

nizar said:

Hi Howard,

Amibroker has been around for about 5 years i think.
You mention having written a paper in 1969

What did you use before amibroker was around?

Click to expand...

Hi Nizar --

Yes, I have been studying and applying modeling and simulation techniques to financial time series for a long time.

In my opinion, there is nothing currently available to the masses of us (who are not employed by companies that have proprietary development platforms) developing trading systems that is better than AmiBroker. Everything I have used in the past, including ones that I wrote myself, is obsolete.

Thanks,
Howard

howardbandy said:

The points I am making and the techniques that I am describing throughout my book, my speeches, my workshops, and my posts to this forum are intended to help system developers and traders produce systems that are likely to trade profitably.

I hear comments about the difficulty of doing what I suggest, and I hear suggestions to the effect that reliable systems can be developed without out-of-sample testing. But there will be an out-of-sample test. My point is that using the techniques I describe, I can make a series of confidence building out-of-sample tests without risking real money before the marketplace makes its out-of-sample test.

Thanks for listening,
Howard

Click to expand...

Howard.

Thanks for sharing your knowledge and experience with us.
You have taught me a great deal and for that Im very grateful.
Everything that you have said has made sense to me (often after further questioning by myself) and has been taken aboard for my own testing.

howardbandy said:

Hi Tech/a --

I do not understand this one:

How do you evelute the period ahead that you should use in the evaluation?

Thanks,
Howard

Click to expand...

That which you chose for out of sample testing.
Ill do as Snake suggests though.

Howard.
What features in your opinion make Amibroker so good.
Have you heard of Tradesim---which can be added to Amibroker.
http://compuvision.com.au/

tech/a said:

That which you chose for out of sample testing.

Click to expand...

Hi Tech/a --

The out-of-sample period immediately follows the in-sample period. It is as long as you wish -- as long as the results continue to be acceptable. (Theoretically, there can be time gap between the in-sample period and the out-of-sample period, but I have never seen that benefit the system. The out-of-sample period can never be earlier in time -- longer ago -- than the in-sample period.)

If the system works (is profitable, scores high on the objective function, etc) because it identifies some inefficiency in the market, as long as the market remains stable relative to that component, the system will continue to work. When the system degrades, it is because the underlying market has changed, and you will need to re-optimize.

If the walk forward process has been followed, then you already know what to expect when you re-optimize. That is, you have several observations of the out-of-sample performance following a re-optimization. If those results are consistently good, that gives you confidence in the walk forward process itself. You will know whether or not you can re-optimize whenever you want to -- the answer is probably that you can -- in fact many very good systems re-optimize after every bar.

Gaining confidence in the walk forward process is a very important part of the designing, testing, and validating trading systems.

Thanks for listening,
Howard

Howard.

Thanks for your time and effort.

I have a number of questions but will start with the Practical Application of Optimisation.

While I and I'm sure most here understand the idea behind optimisation of variables and possibly conditions. I would like your comments on the following.

We can choose anytime to re optimise.
That can be daily,weekly,at the beginning of a trade,at the end of a trade,really anytime.
Question 1 When is the "Optimum time to re calibrate?"

Which brings me to practical application.

Ive tested a system every which way Forward/backward/montecarlo,in and out of sample and I have an optimised variable set on 3 conditions of 10,25,8

I'm ready and confident to trade.

Scenario 1
A week later all my conditions and variables are met on a trade.
So I just run the optimisation again.---now its 15,20 and 6.Nothing like a week ago.

Scenario 2
A week later All the conditions are met so I enter the trade.My exit criteria are 10,20 and 4.
5 days later all are hit,but when I optimise the variables again the exit criteria is 8,25,8 and that happened 2 days ago.Now what?
OR
The optimum variables for exit are now 12,25,6 which hasnt now been hit although the original ones have. Now what?

How can you then trust that the optimised variables which you have set against the objectives of your trading are infact optimum.
Infact Id argue that after a long enough period the entries and exits would appear random but between maximum and minimum values.

Why would the trading results be any better than a fixed selection of variables (perhaps your first optimisation results). the results provided the logic behind the system is one of positive expectancy are going to fall between the extremes of the values chosen for optimisation the maximum and the minimum. They will never fall at the maximum or minimum as like the market the dynamics will continually move the possible result.

Therfore I cant see how/why optimisation will improve result over the longterm.

Infact if you were to re calibrate the variables 12 mths later they wouldnt (normally) resemble the initial variables selected at the time of completion of systems design. How can you have confidence when infact the best time for entry or exit may be found to have passed or be changing to such an extent that it ends up impacting on expectancy.

For me its Practical applicationrather than the thoery of how and why it should be better or indeed a necessity.

Can you show a Practical application of your arguement?
Perhaps a chart showing in dynamic realtime the difference between optimised and static variables of a method and the improvement.
I dont know---what do you suggest.

Tech.
Some really good points in there.

Im looking forward to Howard's response.

I do understand the practical application of optimisation, but why we need to re-optimise after having traded the system is beyond me.

Greetings all --

About the values of the variable moving around a lot.

What is your objective function? Some do not work well. For example, optimizing for total profit often does not trade well out-of-sample.

If the optimal values are changing wildly over short time periods, it sounds like either the trading model or the market is unstable. If the market is changed to something that is fairly smooth -- say XLE, one the S&P sector ETFs -- does the model seem to behave better? If so, then it is the market that is unstable. If not, then it is the model that is unstable. We all have difficulty showing confidence in an unstable system. (System == model plus market.)

If you want to experiment further:

Pick a set of variables that are optimal for an in-sample time period. Keep the ticker and time period fixed.

Make several test runs with each run having slightly different values for the variables.

For example, a moving average crossover system has two parameters -- the moving average lengths. Say the optimal values for the arguments are 20 and 7. Try nine runs -- 18 and 5, 18 and 7, 18 and 9, 20 and 5, 20 and 7, 20 and 9, 22 and 5, 22 and 7, 22 and 9.

If the results are very similar, then the system is robust relative to perturbation of the arguments. If the results vary widely, then the optimizer found an undesirable "peak" surrounded by steep cliffs, rather than a desirable fairly flat "plateau."

Thanks,
Howard

Hi Nizar --

You wrote: "I do understand the practical application of optimisation, but why we need to re-optimise after having traded the system is beyond me."

You do not need to re-optimize as long as the system is performing to your satisfaction when actually traded.

The key points are:

The walk forward process is important in and of itself. By making several walk forward steps, you will get some experience with how the system behaves during and after optimization.

The walk forward process establishes a length of time that is known to be a practical period of time after which a re-optimization can be done. (Re-optimization can almost always be done more frequently, if desired.)

Any system that is being traded in real-time trades and is satisfactory can be left alone. But -- eventually it will degrade. Every system eventually fails. Once a system has failed it seldom (never?) recovers. It must be either abandoned or re-optimized.

--------------

I think I need to make one major point again.

If I have used all of the data I have available to develop a trading system, and I make any choices at all, I am performing an optimization. (An optimization is just another name for an organized search, usually over a larger parameter space than is or even can be done by setting the values one at a time.)

If I optimize over all but, say, the last year of data, then look at the results from the last year of data and subsequently change the model, I no longer have any out-of-sample data left -- all the data is now in-sample.

Following one or more in-sample tests, there will be an out-of-sample test. I have a choice -- to follow good system design, test, and validation procedures and make that (or those) out out-of-sample test myself, or to let the market make the out-of-sample test after I have placed actual trades.

I sure am getting tired of saying that over and over. Somebody, please, say they understand.

Thanks for listening,
Howard

Hi Tech/a --

You wrote: "Therfore I cant see how/why optimisation will improve result over the longterm."

My point is -- whenever any of us makes any choice from among alternatives, we are already optimizing. So we might as well get the most benefit from it.

Thanks,
Howard

howardbandy said:

Hi Nizar --

You wrote: "I do understand the practical application of optimisation, but why we need to re-optimise after having traded the system is beyond me."

You do not need to re-optimize as long as the system is performing to your satisfaction when actually traded.

The key points are:

The walk forward process is important in and of itself. By making several walk forward steps, you will get some experience with how the system behaves during and after optimization.

The walk forward process establishes a length of time that is known to be a practical period of time after which a re-optimization can be done. (Re-optimization can almost always be done more frequently, if desired.)

Any system that is being traded in real-time trades and is satisfactory can be left alone. But -- eventually it will degrade. Every system eventually fails. Once a system has failed it seldom (never?) recovers. It must be either abandoned or re-optimized.

--------------

I think I need to make one major point again.

If I have used all of the data I have available to develop a trading system, and I make any choices at all, I am performing an optimization. (An optimization is just another name for an organized search, usually over a larger parameter space than is or even can be done by setting the values one at a time.)

If I optimize over all but, say, the last year of data, then look at the results from the last year of data and subsequently change the model, I no longer have any out-of-sample data left -- all the data is now in-sample.

Following one or more in-sample tests, there will be an out-of-sample test. I have a choice -- to follow good system design, test, and validation procedures and make that (or those) out out-of-sample test myself, or to let the market make the out-of-sample test after I have placed actual trades.

I sure am getting tired of saying that over and over. Somebody, please, say they understand.

Thanks for listening,
Howard

Click to expand...

Yes I understand!

The difference and the importance of splitting up your data into in-sample and out-of-sample for walk forward analysis has been explained very succintly here and in your other posts.

Thanks for the explanation of re-optimisation.

Thanks for your time and your continuing contributions.
Much appreciated.

howardbandy said:

So we might as well get the most benefit from it.

Click to expand...

Thanks for your postings here!

About your book, you mentioned too many variables may result in curve fitting. How many rules/variables would you consider too many?

I'd take a wild guess at 4+ but have no idea as how to "quantify" it. :

Would using variables that have no relation help in preventing curve fitting? Rather than 2 entry and 2 exit variables, for example 1 variable each for position sizing, turnover, entry and exit.

Cheers SB

howardbandy said:

I sure am getting tired of saying that over and over. Somebody, please, say they understand.

Click to expand...

I get it! Forward testing occurs at the hard right edge the moment you start trading with real money unless you reserved a portion of data to be out-of-sample that you could simulate forward testing with, after testing and optimising on the in-sample data.

I'm in the process of building a test plan and will be reserving said out of sample data.

Howard, thanks for being patient enough to get this important message into our sometimes thick heads!

ASX.G

Hi Howard,

I understand and I suspect that most of us here at ASF also understand the importance of the walk forward methodology using out of sample data in verifying the robustness of the system going forward. The concept is a very sound one and we thank you for your patience.

The one area that I personally like to see being discussed more is this concept of system decay, and how over time, the system will lose its edge and will ultimately fail. Logic would indicate (to me at least) that as more and more people trade a particular type of system, say a 30 day breakout system, that the system would have a greater chance of succeeding as an ever increasing number of traders initiate the trade upon confirmation of the breakout, to a point where it almost becomes a self-fulfilling prophecy. How then does such a system lose its edge over time ?

Does a system tend to fail as more traders use it ?? If the answer is yes then can one conclude that the system would more effective if it is used by fewer traders ?

When a system starts to lose its edge (presumably from over-exposure from too many traders using it) the number of traders using it will also tend to decrease over time as they seek out other more profitable strategies. When the number of traders have dropped off sufficiently, will the edge that the system had initially return ?? If the answer is NO, then what residual external factor that had caused the system to initially fail has remained in play to continually impact on the system even though the number of traders using it has dropped off dramatically ?

I can’t get my head around this system decay concept and hope that you may be able to shed some light on the subject.

howardbandy said:

Hi Tech/a --

You wrote: "Therfore I cant see how/why optimisation will improve result over the longterm."

My point is -- whenever any of us makes any choice from among alternatives, we are already optimizing. So we might as well get the most benefit from it.

Thanks,
Howard

Click to expand...

Yes

If the results are very similar, then the system is robust relative to perturbation of the arguments. If the results vary widely, then the optimizer found an undesirable "peak" surrounded by steep cliffs, rather than a desirable fairly flat "plateau."

Click to expand...

An important point with all objectives.
Not trading the best optimised variables but seeing if the system is flatish regardless of best optimisation.
The distribution is within a minor deviation.

Yes I can see this rather than attempting to trade the best optimised parameters.

On Walk Forward testing.
With longer term methods which can hold stock for instance 12 mths or more how can we gain a credible walk forward test?

If the Reasons a system will be profitable (Positive expectancy by way of more winners than losers OR greater nett profit gained from all winners than nett losses from losing trades) then how often Howard do you see a system breakdown when testing out of sample or walking forward.

Howard I also asked the question in regard to your remarks that Trend trading is no longer profitable (or words to that effect).
My question is that without a trend either long or short how is it possible to create a profit. At some point price has to move in a trend in some timeframe or another long enough to register a profit. Other than exotics (Options)
I'm interested in how you can make a profit at all without any trend at all.

Sir Burr said:

Thanks for your postings here!

About your book, you mentioned too many variables may result in curve fitting. How many rules/variables would you consider too many?

I'd take a wild guess at 4+ but have no idea as how to "quantify" it. :

Would using variables that have no relation help in preventing curve fitting? Rather than 2 entry and 2 exit variables, for example 1 variable each for position sizing, turnover, entry and exit.

Cheers SB

Click to expand...

Hi Sir Burr --

Let me start with a few general observations.

One -- trading systems are profitable because they recognize some inefficiency in the market they are trading.

Two -- the act of making a profitable trade removes some of the inefficiency and makes further profitable trading more difficult.

Three -- designing trading systems is very much like designing control systems for industrial processes. Financial data is more complex than most industrial data. As the systems being managed increase in complexity, the programs that manage those systems must also be more complex.

Four -- simple systems are too easily replicated by many systems developers and potential traders.

Five -- all systems eventually fail. A system that has failed will seldom (never?) return to profitability.

Now, on to my point --

One of the balancing acts we try to manage when designing trading systems is putting in enough logic and variables to identify the complex features that will provide the profitable trades without adding so many variables that we allow the system to become curve-fit to the data. The solution -- you knew this was coming -- is the walk forward out-of-sample test. If the system has been curve-fit, out-of-sample results will be poor.

So, the answer is to increase the complexity of the system in whatever creative ways you can in order to recognize inefficiencies and give profitable buy and sell signals. But do not make the system so complex that it becomes curve-fit to the data.

Every rule contained in the trading system adds complexity, removes residual degrees of freedom from the process, and increases the likelihood of curve-fitting. How many rules and how many variables? There is no simple answer, but you can test your systems yourself through the validation phase of system development.

Thanks,
Howard