Would you trade this system?

[nizar]

Sorry Nizar,

I noticed you had transaction costs but no slippage. I think slippage would eat away a large portion of potential profit. Did you used fixed dollar units of fixed % risk in your simulations?

Yes, you are correct.
Looks like your system is much better than mine!
Well done.

Seems like I have a long way to go, but thats to be expected as Iv just started on this journey.

Its gonna be alot of work to get a 2001 ASX300 list. Thanks for the list.

 

[nizar]

Great thread to read guys.Well done for sharing so much of your research.
I have a question if you don't mind.
When you use a back testing method to review your systems, do you then forward test for a period of time and compare actual results vs historical theorectical?
The reason I ask this is some years back when I was playing with metastock I would find systems that appear to work on back test well however when the 'go live' test was undertaken the profitability dropped dramatically and mostly became unprofitable.
I think this is because back testing assumes a get out at price target. This is not always the case (especially with derivatives) because the volume does not allow the trade to be completely exited.
You guys have obviusly spent a great deal more time than I did so wondering if this figures at all in your equations?

thanks and congrats again on a great thread

debono

Well the answer to the part in bold is YES!
We have to do this in order to test the robustness of the system and to validate the system.

There is alot of discussion about walk-forward analysis in one of the other threads i started, titled "System Robustness".

 

[bingk6]

In my point of view, mechanical systems have a far sooner expiry date when traded on the futures market than on the stockmarket.

Why? Because as a % of total market participants, i would suspect that mechanical systems traders make up a far greater proportion in the futures markets than they do in the stockmarket. As a result of this, there is much more research and development going on into designing mechanical systems in the futures market compared to in the stockmarket.

Hi Nizar,

I would have thought that it would be the other way round. Futures are based on indices (whose value are constructed from the SPs of many individual stocks). Therefore the only way to fade the futures and have any sort of impact on its value would be to impact all its constitutent stocks, which makes it far less controllable than individual stocks.

Interesting topic all the same regarding system decay. I guess once performance does not measure up to expectations that it would be time to reoptimize and try again.

 

[howardbandy]

Hi Gorilla --

You asked about this statement in my response to a Monte Carlo question:
"Another is to first create a statistical distribution from the observed actual trades, then make repeated random selections from that distribution. "

This technique begins with a number of observations, with the intent of generating several other sets of observations that have the same statistical characteristics, but are not necessarily the same values, and not necessarily in the same time order.

The following example uses 15 because I can list 15 without filling up too much space. I would probably have many more than 15.

Say I have run a trading system simulation and my out-of-sample results show 15 closed trades. The percent gain for them, as they occur in time order, is:
4
-1
3
2
-1
-1
1
0
3
1
2
-3
0
-2
6

I want to simulate what the equity curve would look like if I had other sequences of trades that came from the same distribution, but not necessarily those exact same numbers and not necessarily in that order.

I begin by creating a Cumulative Distribution Function using those observations. (If that statement did not make sense, go to either my book, Quantitative Trading Systems, Chapter 22, or a text for statistics, econometrics, or simulation.) If desired, I smooth the curve.

To simulate a single equity curve, I make 15 random draws from this distribution (with replacement, for those who care), each draw representing a closed trade, and create the equity curve for those 15.

I do this many times -- say 100 times -- and evaluate the results. I can ask questions such as:
What is the probability that this system has a drawdown greater than 10% at any time?
What is the probability that there will be a string of 3 or more losing trades in a row?
What is the average terminal relative wealth?
What is the standard deviation terminal relative wealth?

Terminal relative wealth, trw, is the ratio of the final account balance to the initial account balance. If the account begins with a balance of $10,000 and ends with a balance of $25,000, then terminal relative wealth is 2.5.

If I have been compounding, that is investing 100% of the account in every trade, then there is a simple relationship between the average percentage gain per trade, p, and the number of trades, n.

trw = (1+p) ^ n

For example, a system that makes 2 trades in a year and gains 10% on each has trw of (1.10)^2 or 1.210 -- a gain of 21.0%.
A system that makes 10 trades a year and gains 2% on each has trw of (1.02)^10 or 1.219 -- a gain of 21.9%.

Thanks,
Howard

 

[howardbandy]

Great thread to read guys.Well done for sharing so much of your research.
I have a question if you don't mind.
When you use a back testing method to review your systems, do you then forward test for a period of time and compare actual results vs historical theorectical?
The reason I ask this is some years back when I was playing with metastock I would find systems that appear to work on back test well however when the 'go live' test was undertaken the profitability dropped dramatically and mostly became unprofitable.
I think this is because back testing assumes a get out at price target. This is not always the case (especially with derivatives) because the volume does not allow the trade to be completely exited.
You guys have obviusly spent a great deal more time than I did so wondering if this figures at all in your equations?

thanks and congrats again on a great thread

debono

Hi debono --

My short answer is "no."

First, there are no historical theoretical return for comparison.

More importantly, the in-sample results have no value in predicting the future profitability or performance of a trading system. Only out-of-sample results count.

There has been a lot of discussion of this in this thread and in the thread on Robustness.

Thanks,
Howard

 

[howardbandy]

Hi Nizar,


Interesting topic all the same regarding system decay. I guess once performance does not measure up to expectations that it would be time to reoptimize and try again.

Hi Bibgk6 --

Right!

Thanks,
Howard

 

[nizar]

Hi Nizar,

I would have thought that it would be the other way round. Futures are based on indices (whose value are constructed from the SPs of many individual stocks). Therefore the only way to fade the futures and have any sort of impact on its value would be to impact all its constitutent stocks, which makes it far less controllable than individual stocks.

Interesting topic all the same regarding system decay. I guess once performance does not measure up to expectations that it would be time to reoptimize and try again.

I wasnt referring to only indices, but commodities as well.

 

[tech/a]

Hi Bibgk6 --

Right!

Thanks,
Howard

Howard.
Are you suggesting optimisation for all trading systems?

Or only single entities--Futures/Commodities/Indexes?

 

[tech/a]

Tech, a question that I have been pondering:

Why is Monte Carlo Analysis better than Optimising/Walk Forward?

We've been lead down the Monte Carlo path with Tradesim but is it the better way to go?

What's the difference?

Cheers SB

S/B its just another analytic tool.
Infact Montecarlo analysis as it is used by the software most have is far from ideal.true Montecarlo looks at EVERY possible variable.
There is a great deal of difference and each can and should co exist.The opinions on Optimising/Forward testing and Walk Forward testing are in some part different to mine.

My own opinion and research differ somewhat to that expressed here.

While I have attempted to express and question the thinking of others I'm afraid it falls on deaf ears answers are obtuse or questions completely ignored.
Fortunately I'm working on systems design with my son who is a Doctor of Physics and has powerful systems design and modelling software and analytical capability.I have my own tutor.
As he says "I can make any mathamatical arguement tell you what ever you want".

 

[howardbandy]

Howard.
Are you suggesting optimisation for all trading systems?

Or only single entities--Futures/Commodities/Indexes?

Hi Tech/a --

Yes, optimize everything. Optimize just means make a thorough search -- nothing more than that. If I am going to choose from among two or three or twelve alternatives, why stop there? I'll look at thousands. You read my post using the "marbles" analogy, yes? Post Number 34 in this thread. I explain why in it.

Thanks,
Howard

 

[howardbandy]

S/B its just another analytic tool.
Infact Montecarlo analysis as it is used by the software most have is far from ideal.true Montecarlo looks at EVERY possible variable.
There is a great deal of difference and each can and should co exist.The opinions on Optimising/Forward testing and Walk Forward testing are in some part different to mine.

My own opinion and research differ somewhat to that expressed here.

While I have attempted to express and question the thinking of others I'm afraid it falls on deaf ears answers are obtuse or questions completely ignored.
Fortunately I'm working on systems design with my son who is a Doctor of Physics and has powerful systems design and modelling software and analytical capability.I have my own tutor.
As he says "I can make any mathamatical arguement tell you what ever you want".

Hi Tech/a --

If there is question that you have asked me that I have not answered, or an answer that I have given that is obtuse, please ask again.

Thanks,
Howard

 

[tech/a]

Howard.

In your experience would you suggest
Fixed origin evaluation
OR
Rolling origin evaluation and why?
How do you evelute the period ahead that you should use in the evaluation?
If optimising at what point do you re calibrate?
How many out of series test periods are in your veiw enough to give confidence and at what point do you gain that confidence?

How do you do your out of sample testing,what software do you use?
Or are you simply walking forward through a data set?
How do you extrapolate single special events that fall within the out of sample test period/s.

I would argue that out of sample testing of models offers no guarentee that the in sample test results will perform any better or worse than the results returned within the parameters of those results obtained through multiple or Montecarlo testing going forward in realtime.

For Starters.

 

[howardbandy]

For Starters.

Hi Tech/a --

I'll start at the top. I read that post and got confused -- the questions asked seemed to be rhetorical, leading up to your statement at the end of the quoted text. May I suggest one question per post? And if you specifically want my comments, please ask me by name.

About anchored or rolling periods --

In my opinion, rolling. If anchored is used, the in-sample period for the last walk forward step during development, and then for all the "bring up to date" steps during actual use of the system, is "all the data." The system is being given more and more data representing more and more market conditions and is less and less likely to be tuned to the current conditions. One of the benefits of using the walk forward process is being able to tune the system to the current conditions.

Thanks,
Howard

 

[howardbandy]

Hi Tech/a --

I do not understand this one:

How do you evelute the period ahead that you should use in the evaluation?

Thanks,
Howard

 

[howardbandy]

Hi Tech/a --

You wrote: "If optimising at what point do you re calibrate?"

Assuming that you mean re-optimize --

I'll define re-optimize to mean the following:
1. Move the in-sample period forward in time so that it includes the most recent data, or nearly so.
2. Run an optimization to determine the values for the arguments that give the highest objective function score for that in-sample period.
3. Begin using the new values of the arguments to test the out-of-sample data or to begin trading using the new values.

There are a couple of possibilities:

1. Always re-optimize after a set number of out-of-sample bars. This is what would have been done during the walk forward phase of development of the model, so it makes sense to continue doing that.

2. Re-optimize whenever the out-of-sample results, or trading results, indicate that the system is performing at a lower level than anticipated. To answer this question requires some statistical analysis. The implications of re-optimizing after a variable number of out-of-sample bars is that you might continue to trade a system beyond the normal walk-forward period. If the system is performing well, feel free to do that.

There is no single answer. It depends on the characteristics of the market being modeled and the model.

Thanks,
Howard

 

[howardbandy]

Hi Tech/a --

You wrote: "How many out of series test periods are in your veiw enough to give confidence and at what point do you gain that confidence?"

There is no one-size-fits-all answer to this question.

The goal of the system development process -- whether you are using methods similar to those I describe or some other method -- is to develop confidence that the system will trade profitable in the future. The emphasis is on "develop confidence." The characteristics of a trading system that give you confidence may not be the same as the characteristics that give me confidence.

I believe that the "goodness" of a trading system is defined by each individual (or trading company) well before system testing begins -- at the time when the objective function was being defined.

If the combined out-of-sample results of a walk forward process produce results that are satisfactory in terms of the definition of the objective function, then that should give the level of confidence needed.

Thanks,
Howard

 

[howardbandy]

Hi Tech/a --

You wrote: "How do you do your out of sample testing,what software do you use?"

I use AmiBroker for most of my development work. In chapter 20 of my book, I describe using AmiBroker to perform a walk forward analysis.

At present, it is necessary to set the dates for the in-sample and out-of-sample periods "manually," but I would not be surprised to see AmiBroker extended to have automatic walk forward capabilities in the near future.

Thanks,
Howard

 

[howardbandy]

Hi Tech/a --

You wrote: "Or are you simply walking forward through a data set?"

I do not understand the question.

Thanks,
Howard

 

[howardbandy]

Hi Tech/a --

You wrote: "How do you extrapolate single special events that fall within the out of sample test period/s."

I do not understand the question.

Thanks,
Howard

 

[nizar]

Hi Howard,

Amibroker has been around for about 5 years i think.
You mention having written a paper in 1969 and so I assume you mustve already been a top trader for a number of years before amibroker was released?

What did you use before amibroker was around?