Amibroker FAQ

[Abyss]

The QM in me still wants the lazy F$@#&*s out of the portfolio and back on the street, going back to their old job.....looking for a new one!

Joking BTW....just for the record.

 

[Gringotts Bank]

Guys, what is considered the best metric for optimization robustness? Thanks.

 

[howardbandy]

Guys, what is considered the best metric for optimization robustness? Thanks.

Hi GB --

The most important criteria is that you have confidence that the order you prefer for the alternatives that are examined during the optimization is the same order that the objective function puts them in. That is, you should have confidence that the alternative with the highest objective value score is the one you would pick given knowledge of all of them. Or, at very least, that you are confident that the highest ranked result is adequate for your purposes.

Additionally, I recommend using objective functions that focus on the trades, not on the equity curve. The equity curve, and the associated drawdown, is very path-dependent / trade sequence-dependent. Since we cannot expect trades in the future to occur in the same sequence as those in the testing, we should not allow the optimization process to take trade sequence, and the associated drawdown metrics, into account when optimizing. To avoid introducing this bias, when optimizing, position size should be a fixed number of dollars (or a single futures contract). Allowing any position sizing, even compounding, introduces an unfavorable bias that will always result in underestimating risk and overestimating profit. Do the position sizing after the system has been validated -- as part of the trading management process -- using Bayesian techniques. See page 39 of my Mean Reversion book for a flowchart. You can download the chapter that chart appears in at no cost from the book's website:
http://www.meanreversiontradingsystems.com/book.html
Click the link to Chapter 2. You will be getting a pdf document.

Given all that, some objective functions select alternatives that are better at identifying the patterns you are seeking -- patterns that precede profitable trading opportunities -- and not so easily fooled by the noise in the data. Some experimentation is required to determine what will work best for the model, the data, and the person doing the development.

Good terms to include are: ratio of winning trades to losing trades, distribution of maximum adverse excursion, and geometric return per trade. The keys to maximizing the growth of a trading system are: frequent trading, short holding periods, a high percentage of trades that are winners, the worst losing trades are manageable.

As always, regard in-sample results with skepticism -- they are always good, particularly after an extensive optimization, and they usually have little or no relationship with out-of-sample performance.

Best regards,
Howard

 

[Gringotts Bank]

Hi GB --

The most important criteria is that you have confidence that the order you prefer for the alternatives that are examined during the optimization is the same order that the objective function puts them in. That is, you should have confidence that the alternative with the highest objective value score is the one you would pick given knowledge of all of them. Or, at very least, that you are confident that the highest ranked result is adequate for your purposes.

Best regards,
Howard

Thanks Howard for your reply. Most of it I understood, except for this first paragraph and I'm hoping you could expand on it with an example please.

Also, regarding WF analysis, what 'step periods' are considered acceptable? Longer step periods look like they are more likely to give better results going forward, and vice versa. Also, how many steps would you recommend as a minimum?

 

[howardbandy]

Hi GB --

Say there are 3 variables, each with 10 values, tested -- 1000 alternatives in total. If you printed out the stats of all 1000, laid them out on the floor and moved them around, you would subjectively rank them -- prefer some over others. The objective function should give scores that agree with your subjective ranking.

At a minimum, each test period -- in-sample and out-of-sample -- must be long enough to, on average, have several completed trades. In general the periods should be as short as possible to give frequent resychronizations to keep the model tuned to the data, but long enough to give stability and not be fitting too complex a model to too few data points. There is no general rule. You will need to do some testing to see what works best.

Best regards,
Howard

 

[Gringotts Bank]

Hi GB --

Say there are 3 variables, each with 10 values, tested -- 1000 alternatives in total. If you printed out the stats of all 1000, laid them out on the floor and moved them around, you would subjectively rank them -- prefer some over others. The objective function should give scores that agree with your subjective ranking.

At a minimum, each test period -- in-sample and out-of-sample -- must be long enough to, on average, have several completed trades. In general the periods should be as short as possible to give frequent resychronizations to keep the model tuned to the data, but long enough to give stability and not be fitting too complex a model to too few data points. There is no general rule. You will need to do some testing to see what works best.

Best regards,
Howard

Thanks again. My subjective rating would be the steepest equity curve with the smallest draw down. So I guess that just means as the objective measure I should use CAR/MDD or maybe Sharpe, yes?

 

[Abyss]

Future leak-

Just playing around with some code I found online and the system in question has a future leak identified in ABs code check function.
I narrowed it down to this line of code and I am curious if someone could tell me why this line would generate a future leak.

AddToComposite(Normaliser, "~atr_"+Name(), "C", 1+2+8);

Thanks in advance.

 

[Caveroute]

Future leak-

Just playing around with some code I found online and the system in question has a future leak identified in ABs code check function.
I narrowed it down to this line of code and I am curious if someone could tell me why this line would generate a future leak.

AddToComposite(Normaliser, "~atr_"+Name(), "C", 1+2+8);

Thanks in advance.

What's a 'future' leak and how does it manifest itself here ?

 

[joshsmi]

Future leak-

Just playing around with some code I found online and the system in question has a future leak identified in ABs code check function.
I narrowed it down to this line of code and I am curious if someone could tell me why this line would generate a future leak.

AddToComposite(Normaliser, "~atr_"+Name(), "C", 1+2+8);

Thanks in advance.

AddToComposite just creates a symbol called ~atr "+ name().
It is probably the normaliser variable. How does that one look like?

 

[joshsmi]

What's a 'future' leak and how does it manifest itself here ?

For example Ref( C, 1 ) looks in to the future while Ref( C, -1 ) looks backward. Or zig-zag may also look into future. It is rather for pattern and trend recognition but not for backtesting. So these are two examples.

 

[howardbandy]

Thanks again. My subjective rating would be the steepest equity curve with the smallest draw down. So I guess that just means as the objective measure I should use CAR/MDD or maybe Sharpe, yes?

Hi GB --

I recommend that two objective functions be used.

The first is during development -- looking for the best rules and parameters. It is important that there be no position sizing, not even compounding. Allowing any position sizing introduces an unfavorable bias that overestimates profit and underestimates risk. You may not have too much choice for your objective function. If possible, do not include a term that attempts to minimize drawdown. Drawdown is very path dependent. Minimizing drawdown rewards a specific path that is certain to be above the mean and introduces another unfavorable bias. The emphasis during development should be on accurate recognition of the trades that you are looking for -- so emphasize metrics related to the trades themselves. Minimizing root mean square of MAE (Maximum Adverse Excursion) is good, because it is the number and magnitude of losing trades that limit position size, which in turn limits profit potential. Also include terms that have the trading frequency and / or holding period you prefer. And percentage winning trades you want. Definitely insist on a positive geometric gain per trade.

The second is during trading. Now you Do want to minimize drawdown. But don't rely on a single value such as average drawdown or maximum drawdown -- look at the entire distribution of drawdowns -- the Cumulative Distribution Function (CDF). Pay attention to the tail of risk -- the part of the distribution that is above the 95th percentile -- that is the LTCM / black swan area. I recommend using the "best estimate" set of trades and the technique I describe in my Modeling book to estimate risk, determine the highest safe position size, estimate profit potential, and decide whether the system is worth trading. After every trade, update the best estimate set and rerun the risk analysis, position size, and trade / no trade decision. (This repeated updating of the set of trades and re-estimating of risk and position size is an empirical Bayesian process, if it matters what the formal name for it is.)

Both CAR/MDD and Sharpe reward low drawdown. They are both good for the trading portion. See what other objective functions are available to you during development. CAR/MDD may be the best of a poor selection, in which case, use it. But be prepared for the bias -- which will cause out-of-sample performance to be lower than in-sample performance.

Best regards,
Howard

 

[Gringotts Bank]

Hi GB --

I recommend that two objective functions be used.

The first is during development -- looking for the best rules and parameters. It is important that there be no position sizing, not even compounding. Allowing any position sizing introduces an unfavorable bias that overestimates profit and underestimates risk. You may not have too much choice for your objective function. If possible, do not include a term that attempts to minimize drawdown. Drawdown is very path dependent. Minimizing drawdown rewards a specific path that is certain to be above the mean and introduces another unfavorable bias. The emphasis during development should be on accurate recognition of the trades that you are looking for -- so emphasize metrics related to the trades themselves. Minimizing root mean square of MAE (Maximum Adverse Excursion) is good, because it is the number and magnitude of losing trades that limit position size, which in turn limits profit potential. Also include terms that have the trading frequency and / or holding period you prefer. And percentage winning trades you want. Definitely insist on a positive geometric gain per trade.

The second is during trading. Now you Do want to minimize drawdown. But don't rely on a single value such as average drawdown or maximum drawdown -- look at the entire distribution of drawdowns -- the Cumulative Distribution Function (CDF). Pay attention to the tail of risk -- the part of the distribution that is above the 95th percentile -- that is the LTCM / black swan area. I recommend using the "best estimate" set of trades and the technique I describe in my Modeling book to estimate risk, determine the highest safe position size, estimate profit potential, and decide whether the system is worth trading. After every trade, update the best estimate set and rerun the risk analysis, position size, and trade / no trade decision. (This repeated updating of the set of trades and re-estimating of risk and position size is an empirical Bayesian process, if it matters what the formal name for it is.)

Both CAR/MDD and Sharpe reward low drawdown. They are both good for the trading portion. See what other objective functions are available to you during development. CAR/MDD may be the best of a poor selection, in which case, use it. But be prepared for the bias -- which will cause out-of-sample performance to be lower than in-sample performance.

Best regards,
Howard

Hi Howard,

You've exposed quite a few gaps in my knowledge!

AB doesn't seem to have the root mean square of the MAE as an optimization parameter. Is this a custom calculation? Also, how do I best insist upon a positive geometric gain per trade?

Thanks again.

 

[howardbandy]

Hi Howard,

You've exposed quite a few gaps in my knowledge!

AB doesn't seem to have the root mean square of the MAE as an optimization parameter. Is this a custom calculation? Also, how do I best insist upon a positive geometric gain per trade?

Thanks again.

Hi GB --

Yes, metric based on MAE (Maximum Adverse Excursion) is a custom calculation. MAE is reported for each trade by AmiBroker. (If someone is using a different trading system development platform, MAE can be computed by simulating the functions of the platform by examining prices for each bar a trade is active.) Use the Custom Backtester to gather the MAE from each trade, compute the metric, and use it as an objective function. The formula I have posted on my blogsite for computing standard deviation in a single pass might be helpful.
http://www.blueowlpress.com/WordPress/code/standard-deviation-in-a-single-pass/

Geometric return is the nth root of the product of n individual returns. Do this while you are computing the MAE metric in Custom Backtester. Gather the percentage gained trade-by-trade and express it as a fraction. A gain of 2% will be 1.02, a loss of 3% will be 0.97. No compounding -- only percentage gained per trade based on ratio of sell price to buy price. Begin the running return with a value of 1.0, multiply by the gain per trade as each trade is processed. Count the number of trades, n. After the final trade has been processed, raise the aggregate running return to the power of 1/n. Put a term in the objective function that rewards positive values and penalizes negative values. I force the objective function to zero if the geometric return is negative, taking these alternatives completely out of consideration.

Best regards,
Howard

 

[Abyss]

AddToComposite just creates a symbol called ~atr "+ name().
It is probably the normaliser variable. How does that one look like?

Thanks for that, the normaliser variable is

Normaliser = ATR(20);

But when I double slash...... //AddToComposite(Normaliser, "~atr_"+Name(), "C", 1+2+8);

The system runs fine and the code check doesnt identify any future leaks.

So..a couple more questions please, if Amibroker doesnt identify any future leaks, does that mean there isnt any?
And if thats so and the system performs really well in walk-forward testing consistently over various watch lists and many different tests does that mean the system is robust and a good candidate for trading?

Thanks for your time.

 

[trash]

Thanks for that, the normaliser variable is

Normaliser = ATR(20);

But when I double slash...... //AddToComposite(Normaliser, "~atr_"+Name(), "C", 1+2+8);

The system runs fine and the code check doesnt identify any future leaks.

So..a couple more questions please, if Amibroker doesnt identify any future leaks, does that mean there isnt any?
And if thats so and the system performs really well in walk-forward testing consistently over various watch lists and many different tests does that mean the system is robust and a good candidate for trading?

Thanks for your time.

The reason is atcFlagEnableInBacktest = 8 in your ATC line.

Possible reason could be that it is backtesting on a variable that is not there yet. Normalizer variable has to be created first.
Do you use "~atr_"+Name() symbol in some buy rules? Maybe it is marked as problematic doing it that way that's why it is calling for future leak. Just send a question to AmiBroker support. They can tell you the reason for sure.

 

[Abyss]

Ok thanks Trash, I have a long way to go with .afl so wont even pretend to understand that yet and will take your word for it.

But back to the future leak, if Amibroker doesnt detect a future leak in code check, does that mean that there isnt one? And if so, and the system performs well consistently in walk forward testing then its robust and a candidate to trade?

Thanks.

 

[Abyss]

Hate to be a pain but also....
How many times can you run a walk forward test over the same time frame, does it matter as long as you save fresh data for final testing?

Cheers.

 

[howardbandy]

Hate to be a pain but also....
How many times can you run a walk forward test over the same time frame, does it matter as long as you save fresh data for final testing?

Cheers.

Hi Abyss --

The goal of system development is for the developer / trader to have confidence that the signals generated by the system provide reward adequate for the risk. The key word is confidence. The primary limitation is risk.

Walk forward testing is the gold standard for validation. Ideally, the system will either pass validation or fail based on a single out-of-sample test or walk forward run. I know, that is pretty unrealistic. As you point out, every modification made to the system based on out-of-sample results reduces the value of additional tests using that data. You are right to save some data (it must be more recent than that used for any of the previous tests) for a final validation, done as a single test. Failing that, delay implementation using real money while the system is paper traded. Monitor the paper trades until you have confidence in the system.

If you have not done so already, I recommend reviewing the risk estimation procedures I describe in "Modeling Trading System Performance." That procedure begins with a "best estimate" set of trades, uses those trades to estimate drawdown over some future time period, calculates the maximum safe position size, and estimates profit potential. As real or paper trades occur, add them to the best estimate set, rerun that analysis, computing a new safe position size. When the performance of the system is poor, position size will automatically be reduced.

Best regards,
Howard

 

[howardbandy]

Ok thanks Trash, I have a long way to go with .afl so wont even pretend to understand that yet and will take your word for it.

But back to the future leak, if Amibroker doesnt detect a future leak in code check, does that mean that there isnt one? And if so, and the system performs well consistently in walk forward testing then its robust and a candidate to trade?

Thanks.

Hi Abyss --

AmiBroker tries to detect future leaks, but it is not always able to.

It cannot look inside a dll, so will miss a future leak in that code.
It does not looks inside an #Include, so will miss a future leak in that code.
It will try to evaluate future references in Ref statements, but it cannot anticipate values of variables assigned to reference indexes that might be made during execution, so it will miss those as well.

There are two common symptoms of future leakage:
1. Results are too good.
2. Indicators change and / or Buy and Sell signals appear and disappear in bars other than the current bar. You can watch this happen. Plot a ZigZag (which uses future data) and use Bar Replay. The most recent peak or valley will change as additional data is received.

Best regards,
Howard

 

[Abyss]

Howard,

Thank you for the reply, very informative and helpful. I have purchased all of your books but stopped short when reading MTSP realizing I needed a testable system to get the most out it. I think I have something test worthy now so will move on to that book soon, currently reading Flaw of Averages as recommended by yourself elsewhere and enjoying that book too.
I have read Intro to Ami, QTS and MRTS multiple times and thoroughly enjoyed all of them, though have by no means finished with them, every time I go back and read them again I learn something new that I missed the last time.
Your time here is appreciated.

Thank you.