Howard Bandy "Quantitative Trading Systems" & systems design Q&A

[tech/a]

Howard.

Just recieved your book a couple of days ago.
Firstly send my Thanks to your packaging staff.
Tremendous job.

From a brief look at your book, there is a great deal of effort in it.
I lookforward to having a read.
With the influence of amibroker I can see I'll have to yet again add to my software.

Thought I'd start this thread in the hope that questions on systems testing either related to your book or in general can find a home here,and that you would be kind enough from time to time to drop by and make comment.

Howard.
There is very little on who you are.
In which University were you a Dean?

Thanks for your input so far and hopefully into the future. I like the idea that the book is purely mathamatically based.

 

[nizar]

Well i second that.

Packaging was great.
I received mine about 1 week after payment was made.

Having read the first 50 pages or so and i can say its very well written.
Looking forward to reading the rest.

 

[howardbandy]

Hi Tech and Nizar --

Thanks for the kind words about the book.

About my academic background --

I have degrees in math, physics, engineering, and computer science. I did a master's thesis on artificial intelligence, a master's thesis on statistics, and doctoral work that combined modeling, simulation, and communications in computer networks. My Ph. D. is in Computer Science from Case Western Reserve University in Cleveland, Ohio.

I founded the Computer Science program at the University of Nebraska's Omaha campus, was Professor of Computer Science at the University of Wisconsin's Platteville campus, and was Professor of Mathematics and Computer Science and Dean at Pacific Lutheran University in Tacoma, Washington.

About the book --

As I say in the preface to Quantitative Trading Systems, I have often been frustrated by books and articles that purport to describe fully functional trading systems, only to be sadly disappointed. One of the goals of my book is to be as complete and precise as possible in describing the concepts. Everyone who reads my book should be able to replicate everything I have done.

And, to the extend possible when working with financial time series, the methods used are based on sound principles of modeling, simulation, and statistical analysis.

About AmiBroker --

Throughout many years I have worked with many trading systems and trading system development platforms, including some that were proprietary to trading companies and others I had programmed myself. When I came across AmiBroker, I knew I had found the best platform I would ever be able to afford. Tomasz Janeczko has done a masterful job of designing and programming AmiBroker. It is complete, fast, and very affordable. He is continuing to expand it, particularly in areas that I think are important -- tools for validating systems and tools for automated trading.

Thanks,
Howard

 

[Chorlton]

Hi Howard,

I've also just received your book, and would like to echo the comments of both Tech and Nizar.

The book was well packaged and I received it within a very short period of time of placing my order!!!!

Flicking through it briefly, it seems to be well written, without unwanted padding and consequently I look forward to dedicating some time to reading it without other distractions....

All the best,

Chorlton

 

[nizar]

Howard,

I must say, that newsletter scam analogy is your book was gold!

I think I might even use it one day. LOL

 

[Chorlton]

Hello Howard,

Just re-reading the chapter on Walk-Through Testing.

On page 272, you compare the straightness of the Out-of-Sample Equity Curve. Even though this is probably an "excel" question, is it possible to explain how to do this?

In additional what importance do you place on measuring the straightness on the Equity Curve in this manner compared to say simply looking at the overall MaxDD on the system?


Thanks in advance,

Chorlton

 

[howardbandy]

Hi Chorlton --

The latest AmiBroker (V 5.05 and later) has built-in walk forward testing, which saves me hours every day.

It is unusual for the out-of-sample equity curve to be as steep or as smooth as the in-sample equity curve. If you find one that is, and you have not cheated in the testing and validation phases, your trading system is a potential gold mine.

There are several metrics built-in to AmiBroker that measure both the steepness of the equity curve and the smoothness of the equity curve. Try KRatio, RRR, CAR/MDD, RAR/MDD, or Ulcer Performance Index. Any of these can be used as the metric by which the Best alternative is chosen during the walk forward process. Or -- you can design and program your own.

Thanks,
Howard

 

[grantk]

Hi Chorlton --

The latest AmiBroker (V 5.05 and later) has built-in walk forward testing, which saves me hours every day.

Hi Howard

I read this in the release notes for the latest AB version, however I don't understand it's use. Perhaps I have the walk-forward concept mixed up - to me this means testing your formula which has been optimized on in-sample data, on out-of-sample data. So in AB if I have created and optimized my formula on data from 2006-2007, walk forward testing would be to run the formula on data from 2007-2008 for example. This currently takes two seconds to implement (just changing the date).

What am I missing in walk forward testing that AB now provides?

Cheers.

 

[howardbandy]

Hi Grant --

I think your understanding is correct. The advantage of using the automated walk forward is that it makes the process completely objective and you get to see several in-sample to out-of-sample transitions with one click of the mouse.

1. Define an objective function that fits your trading style.
2. Design a trading system.
3. Fool around with the system and find out how long the in-sample period should be for it to learn the signal without becoming overfit to the noise.

When all of that is done, and your preliminary results indicate that the system looks promising, perform the walk forward tests.

Say the in-sample period the system likes is 2 years, and you want to test the system on a ticker that has data beginning 1/1/2000.

In the Settings > Walk Forward:

Set up the walk forward with the in-sample period beginning 1/1/2000 and with a length of 2 years. Set the Last date to 1/1/2008.

Set the out-of-sample period to anything reasonable -- say 1 year.
(The length of the out-of-sample period is used for two things--
1. It is the period of time between re-optimizations.
2. It is the period of time reported for the out-of-sample results.)

I recommend Not using Anchored, which has the in-sample initial period stay fixed and the in-sample length grows longer with each step, but rather keep the in-sample length constant.

The window at the bottom will show you what dates it plans to use. If you have selected Easy Mode, all the IS and OOS dates will be properly set. If you selected Advanced Mode, check this window to be sure the dates are consistent.

On the Automatic Analysis window, pull down the Optimize menu and click Walk Forward.

The walk forward process will make 7 steps -- each with a 2 year in-sample and 1 year out-of-sample. The first IS period is 1/1/2000 to 1/1/2002. The final IS period is 1/1/2006 to 1/1/2008. The first OOS period is 1/1/2002 to 1/1/2003. The final OOS period is 1/1/2008 to 4/7/2008 -- the final OOS period is whatever data is available for 2008.

AmiBroker will give a report and show the charts for the in-sample and out-of-sample results. If the out-of-sample results look good, you have confidence that your system is recognizing the patterns you hoped it would, that those patterns persist beyond the in-sample period, and that they are profitably tradable for one year beyond.

What you have gained by doing this is seeing 7 transitions from in-sample testing to out-of-sample trading, all without any subjective judgment. This should raise your level of confidence that the next time you reoptimize and begin trading with the latest values, you have a good idea of what to expect in the actual trades.

As always, there are no guarantees. The best we can hope for is a high level of confidence. And we get that confidence by observing the out-of-sample performance following each in-sample period.

Thanks,
Howard

 

[glenn_r]

Hi Howard,

Firstly I appreciate your effort in publishing QTS it is a great book and it has helped me a lot to get a handle on system testing using AB.

I have been looking at a few mean reversions systems to trade the ASX20, as an example I ran your "Stoch of RSI" system published in QTS through the walk forward optimiser function in AB on the ASX20 and it produced the following report.

I was wondering if you could offer your comments on the results, firstly to see if I have the process correct and what should I be looking for, as I'm currently looking for a good profit factor, low draw down and around a 50/50 win/loss ratio.

If you have any comments or pointers on using the walk forward optimiser please pass them on.

I'm also testing your K-Score reversion system (as published in Active Trader) and a value index reversion system, both are also looking promising on early results.

Cheers
Glenn

 

[Chorlton]

Hi Chorlton --

The latest AmiBroker (V 5.05 and later) has built-in walk forward testing, which saves me hours every day.

It is unusual for the out-of-sample equity curve to be as steep or as smooth as the in-sample equity curve. If you find one that is, and you have not cheated in the testing and validation phases, your trading system is a potential gold mine.

There are several metrics built-in to AmiBroker that measure both the steepness of the equity curve and the smoothness of the equity curve. Try KRatio, RRR, CAR/MDD, RAR/MDD, or Ulcer Performance Index. Any of these can be used as the metric by which the Best alternative is chosen during the walk forward process. Or -- you can design and program your own.

Thanks,
Howard

Thanks for the reply Howard. I'll take a closer look at the metrics your mentioned.

 

[josh_in_a_box]

Just recieved the book today, cant wait till I get into it, 2 more chapters on my current book to go!

 

[howardbandy]

Hi Glenn --

First a caution to everyone -- the systems in my book, Quantitative Trading Systems, and in my articles and workshops, including those in Active Trader Magazine, are meant to be educational. Use them to get ideas or as starting points. When you apply them to tickers of your own choice and evaluate them using objective functions of your own choosing, then you can develop them into full trading systems of your own.

In my experience, it is easier to validate systems that have shorter holding periods and more frequent trades than systems that have longer holding periods and infrequent trades. So many of my examples will trade actively.

A comment on the walk forward results posted. I note a 4 year in-sample period and 1 year out-of-sample. The latest two out-of-sample periods are not profitable, so I would not expect 2008 to be profitable. I note exposure is very high. Is this a single ticker traded long and short, or a portfolio that almost always has something to trade? If it is a single ticker, separate the long and short results and see how they look -- you might want too different systems for the two different directions. If it is a portfolio, try tightening the portfolio by allowing few simultaneous positions -- that will take trades only in those tickers with higher positionscore values. Or try different positionscore computations.

Thanks,
Howard

 

[glenn_r]

Hi Howard

Thanks for you comments, I only used your "Stoch of RSI" as an example and it was setup for long trades only, the ASX20 is a portfolio of the top 20 stocks on the ASX.

My testing so far on the other systems that I'm working on has guided me to the fact that you need separate system's to trade the long and short sides, plus there are so many variables that it can become hard to maintain my aim, which is to create a simple, robust and profitable system.

Glenn

 

[chops_a_must]

Finished the book over the weekend Howard. Fantastic really. A couple of typos, but nothing compared to other publications in the field I've read.

Thought the explanations petered out towards the end, but maybe that's because the issues no longer seemed as complex, and I didn't need to read as closely.

Would loved to have seen another chapter devoted completely towards position sizing, and a longer section explaining why stops are bad for systems. But, that's probably because I like stirring and stirrers.

I also can't see a superficial reason why you shouldn't test an entire data set if you were looking purely for statistics (even if that may be used to develop a trading system off). Given you may not develop an acceptable system, before you run out of "purely" out of sample data periods, I can't see what difference that makes. Though I do see the need to test in out of sample data.

Also, if you were looking at coding a previously discretionary strategy, and testing over a period you actually had traded in a similar method, personally I would incorporate that as in sample data (or at least try to).

I think I have some kind of grip on the code now as well. I'll just "hatchet" the code from your book, and from my indicators etc. to get things working. But at least now, thanks to your book, I have the confidence to do so.

Cheers.

 

[Kauri]

Would loved to have seen another chapter devoted completely towards position sizing, and a longer section explaining why stops are bad for systems. But, that's probably because I like stirring and stirrers.


Cheers.

Hi Chops,
I haven't read the book and may be missing something, but if you don't have an initial stop at least then what do you base your position sizing/risk on??
Cheers
...........Kauri

 

[tech/a]

While stop losses maybe detrimental to the very best possible return you could achieve from a system.

They serve very important roles.
They will limit your chances of ruin.
They impart a form of discipline.
For discretionary traders they tell you where your CURRENT analysis is wrong.
They guide you to position sizing.
They give you an opportunity to determine Expectancy.(Albeit in that single trade).
They often save you from being trapped in a trade ranging between your entry and below.
They give more opportunity for maximum use of your capital.
They increase trade frequency---needed for short term methods.

 

[nizar]

Hi Chops,
I haven't read the book and may be missing something, but if you don't have an initial stop at least then what do you base your position sizing/risk on??
Cheers
...........Kauri

There are other ways to determine position sizing.
eg. Fixed dollar amounts (10k parcels out of 100k capital).

In my testing, I went through several initial stops and none of them improved CAR/max.DD.

So in the end I made the initial stop = the trailing stop because I wanted to use a fixed percent risk model for position sizing.

 

[howardbandy]

While stop losses maybe detrimental to the very best possible return you could achieve from a system.

They serve very important roles.
They will limit your chances of ruin.
They impart a form of discipline.
For discretionary traders they tell you where your CURRENT analysis is wrong.
They guide you to position sizing.
They give you an opportunity to determine Expectancy.(Albeit in that single trade).
They often save you from being trapped in a trade ranging between your entry and below.
They give more opportunity for maximum use of your capital.
They increase trade frequency---needed for short term methods.

Hi Tech, and all --

I may not have been clear enough in my comments about stops. My concern is with systems that regularly exit by the price falling (for long positions) enough to hit a maximum loss stop without having first triggered some other exit.

My recommendation is that exits come primarily from signal logic. In the case of systems that hold only a short period of time, timed exits (two days and out) and profit targets (two percent and out) can also work well. In the case of systems that hold longer and are looking for a trend, then trailing stops (such as parabolic or chandelier) work well, while profit targets and timed exits work poorly. I do not consider these trailing stops to be stops, but rather trailing exits. The last resort is the maximum loss stop ($2000 loss and out) -- this is the one I am referring to. A system that regularly exits by hitting the $2000 loss stop is poorly designed.

I agree that knowing the maximum adverse excursion (MAE) is essential in analyzing risk, establishing position size, etc. The MAE calculations come from examining all trades, including those made through signals, timed exits, profit targets, trailing stops, and maximum loss stops -- not just the maximum loss stop.

Sorry for any confusion.

Thanks,
Howard

 

[howardbandy]

Finished the book over the weekend Howard. Fantastic really. A couple of typos, but nothing compared to other publications in the field I've read.

Would loved to have seen another chapter devoted completely towards position sizing, and a longer section explaining why stops are bad for systems. But, that's probably because I like stirring and stirrers.


Cheers.

Hi Chops, and all --

Thanks for the kind words.

There is an "errata" page on the book's web site that has the known list of typos. If you have found others, please let me know and I'll bring that page up to date.

I did have plans to include more on risk analysis, position sizing, portfolio management, walk forward analysis, Monte Carlo analysis, and related topics. I ran out of time and pages. And AmiBroker was and is evolving -- as we see with the recent release of Version 5 which includes automated walk forward.

There is still time to write about those topics -- either in a new book, or a revised edition of Quantitative Trading Systems after Introduction to AmiBroker is finished.

Thanks,
Howard