System parameters

[Billyb]

Question for anyone interested but especially those with experience in systems development and backtesting

Firstly, what do I mean here by a robust system? One that performs similarly over the long term. In other words a more 'future-proof' system, one that is not as likely to produce nasty surprises in the future.

In terms of robustness of a system, assuming all other parameters are the same - which is more robust - a system whose stops are based on ATR (eg 1-ATR) or a system whose stop is based on a percentage value?

The difference being that the former will produce a different value for each market, whereas a percentage value stays the same no matter what. I think introducing ATR into a system will greatly increase the probability of curve fitting but I am inexperienced and curious hence this thread.

A few more examples to ponder:

Which system is more robust (assume other parameters are the same)

-One whose entry is based on a MA crossover versus one whose entry is based on a highest-high value (eg Highest high of the last 50 days)?
-System with an index filter (to turn the system on/off) vs one without
-One whose entry is based on a 20/40 day MA crossover vs one whose entry is based on a 25/45 day crossover? The latter has more data points for the system to consider
-Crossover of closing price over MA vs a crossover of two different MA's
-A system with both a trailing stop and a volatility stop (i.e two different stop systems) or one with just one type of stop?

In some of the above examples there might not be one more robust than the other.

Feel free to add your own examples.

Perhaps 'KISS' better with systems design? I dunno.

 

[colion]

There is no simple answer to your question.

The first hurdle you have to clear is to decide whether you are in the camp of those who believe in optimizing setting periodically or just settle on a fixed value. I've heard both arguments and both have some merit. I settled on fixed values a while ago after convincing myself that on balance neither seemed to hold an advantage and fixed was easier.

Whether one goes down the optimization or fixed road there is a question as to which indicator(s) should be used. I don't think any indicator should be used by itself but rather in concert with others. Toward this end, one should be sure not to combine indicators that are of the same type (e.g., momentum) but rather represent different indicator categories.

Having chosen the indictors one has to decide on the parameter values. In general, it is fair to say that one size usually does not fit all when dealing wtih different time frames (or at least that is my experience). Similarly, one size usually does not fit all for different stocks, etc.

So where does one start? I'd suggest using appropriate software to backtest your ideas which should point you toward a reasonable combination of indicators and appropriate settings. In this way you will have a feel for the systems metrics (e.g., drawdown, Risk-Reward, etc.) and very importantly how it trades relative to your "comfort" zone. Metrics and comfort zone are trader/investor specific so again one size does not fit all. Each of us has to find the right match or so it seems to me.

 

[Joe Blow]

I have removed all the off topic posts in this thread so it can get back on topic.

Any discussion on the original topic of the thread is welcome.

 

[Wysiwyg]

Firstly, what do I mean here by a robust system? One that performs similarly over the long term. In other words a more 'future-proof' system, one that is not as likely to produce nasty surprises in the future.

From my system testing experience there is no programming that can avoid a large gap down in stock price or an entry at the start of a down trend. Since these situations are not known till after they happen then the only out as a trader is to have a hard wired cut off point.

In terms of robustness of a system, assuming all other parameters are the same - which is more robust - a system whose stops are based on ATR (eg 1-ATR) or a system whose stop is based on a percentage value?

Initial hard wired stops serve the purpose of getting out of a trade no matter what we feel or believe. Percent or range matters not if the price moves against you, only the monetary loss you have incorporated (2% of total equity for example) to these points. The wider the initial stop, the smaller the position size and usually the longer the trade time if the price languishes.

Which system is more robust (assume other parameters are the same)
-One whose entry is based on a MA crossover versus one whose entry is based on a highest-high value (eg Highest high of the last 50 days)?

Neither! Both could be winners in a bull market. And that is the secret. A bull market allows for greater opportunity to profit as a greater percentage of stocks are swept along.

-System with an index filter (to turn the system on/off) vs one without

Previous response.

-One whose entry is based on a 20/40 day MA crossover vs one whose entry is based on a 25/45 day crossover?

See bull market. Simple as that.

Perhaps 'KISS' better with systems design? I dunno.

Nothing complex about up, down or ranging price so why pretend so? For marketing purposes it is so.

Finally, it is when you sell that determines a consistently profitable trader. A swing, trend, fundamental or scalp trader will have different time frames to work with.

 

[tech/a]

WYSY
What is your definition of Robustness?
Billy
What is yours?

Personally I think both of you have the wrong end of the stick
The examples as presented have no relevance to robustness.
On my opinion

 

[Wysiwyg]

WYSY
What is your definition of Robustness?

Well it would be a system proven with less loss and/or more profit in the universe of issues being traded over extended and various market conditions.

 

[tech/a]

This isn't robustness it is a criteria selection of yours which determines for you a better or best system. It's what most think is robustness.

Let me try to explain it in a different way
Say you optimize a system and get terrific results using a 30 day Ema cross
You change the parameters to 35 day ema and it turns a loss
The system would not be seen as robust

 

[tech/a]

So you an have a very profitable system whose parameters if changed only slightly kills it's profitability--- it's profitable but not robust

 

[TabJockey]

A good sign of a robust system is if it performs well on significantly different markets. If you get good results on the AXJO and also on AUD:USD using the same system, it is likely that the system is very robust. It probably capitalises upon a deep structural prevalent market inefficiency, which is what you want.

 

[5oclock]

A good sign of a robust system is if it performs well on significantly different markets. If you get good results on the AXJO and also on AUD:USD using the same system, it is likely that the system is very robust. It probably capitalises upon a deep structural prevalent market inefficiency, which is what you want.

Would it be fair to say a system as above may not generate the same level of profits over a given time as a system that needs more tuning for each market. I realize this is a bit of a "how long is a piece a string".But a "robust" system may tend to generate smaller profits than a system that only works well in a more narrow range of market conditions??

 

[TabJockey]

Would it be fair to say a system as above may not generate the same level of profits over a given time as a system that needs more tuning for each market. I realize this is a bit of a "how long is a piece a string".But a "robust" system may tend to generate smaller profits than a system that only works well in a more narrow range of market conditions??

Yes the more robust systems that I have developed have always had a lower per market average expected return as opposed to more narrow systems.

I have this really simple system I call "Bear Eater". It only gives signals during prolonged and powerful bear markets. Averages about 15 signals a year, but did not emit a single signal in 2009. Its an example of a pretty narrow system but its good to add to a portfolio of systems that can combine for effective risk/return results.

Even though on the surface more robust systems do seem nominally less profitable, I think they are probably in general more profitable over all, for three reasons:

a) They will almost certainly last longer before needing to be reworked

b) More markets = more signals = greater profit and compounding potential.

c) More markets = (hopefully) more diversification = less severe draw downs.

 

[5oclock]

Yes the more robust systems that I have developed have always had a lower per market average expected return as opposed to more narrow systems.

I have this really simple system I call "Bear Eater". It only gives signals during prolonged and powerful bear markets. Averages about 15 signals a year, but did not emit a single signal in 2009. Its an example of a pretty narrow system but its good to add to a portfolio of systems that can combine for effective risk/return results.

Even though on the surface more robust systems do seem nominally less profitable, I think they are probably in general more profitable over all, for three reasons:

a) They will almost certainly last longer before needing to be reworked

b) More markets = more signals = greater profit and compounding potential.

c) More markets = (hopefully) more diversification = less severe draw downs.

The current market would sort out most systems!! Do you any system working at the moment?? I think VSA has a lot of promise, a lot to learn but it does help shine a bit of light on what is going on. Also I think a trading method vs a more rigid system suits myself better.How would you describe your "systems", very ridgid ,or more of a "method" allowing so leeway on setup and entry ,exit ??

 

[TabJockey]

Yeah VSA definitely has promise but its not for me. Hell voodoo science like Gann and goddamn moon cycles has promise if it somehow suits your trading personality. My systems are 100% mechanical and fully rigid. I have found that I am only profitable if I am fully discretionary or fully mechanic, if I mix them I lose. But thats just me.

I have put allot of time into stock systems and basically came out with the conclusion that, especially with small caps, you need to take into account many factors that I am not capable of programming into metastock or amibroker. I think it really hurts to eb under capitalised in equity also, you want to have at least 50k to play with. So now I concentrate on forex with great results so far, I have one system live at the moment thats sitting on 6.6% for the last 30 days which I am really happy with (albeit a period of high volatility). Funnily enough I dont think the system is all that robust and I am furiously researching other mechanical strategies that might work.

I really enjoy researching, writing and testing the systems. It can be monotonous but I find it incredibly fufilling and I would recommend it as a personal money management approach for any tech savy investor who enjoys messing with statistics.

 

[Wysiwyg]

Long only trading should have a start/stop switch. Most stocks get swept along with the prevailing trend. Index price crossing a long moving average is tricky in respect to how long the moving average is. Long moving average of an Index crosses are terribly lagging. Then you have to address ranging markets.
Obviously newspapers, television and internet convey the information of worldly economic situations which each individual evaluates relative to their risk profile and investment/trading strategy.

 

[Billyb]

This isn't robustness it is a criteria selection of yours which determines for you a better or best system. It's what most think is robustness.

Let me try to explain it in a different way
Say you optimize a system and get terrific results using a 30 day Ema cross
You change the parameters to 35 day ema and it turns a loss
The system would not be seen as robust

Yes that is a good definition. Also, if one changes the paramaters say using your example then other statistics also shouldn't change too much - eg CAR, MDD

however I also think a robust system shouldn't produce nasty surprises in the future. If it does it means the system may have been over optimized (i.e not robust) or it could also mean that the market has change sufficiently so that the system no longer works on it . But IMO a really robust system should be able to have some sort of method to adapt to sudden changes, or completely avoid it eg system on/off switch. Then when it switches off, the system stays in cash, or perhaps a different system kicks in?

A good sign of a robust system is if it performs well on significantly different markets. If you get good results on the AXJO and also on AUD:USD using the same system, it is likely that the system is very robust. It probably capitalises upon a deep structural prevalent market inefficiency, which is what you want.

Agree very much so with all your points TabJockey, I would never trade a stock trend trading system that worked well on the ASX but lost money on the US market - clearly there is something wrong if such a huge discrepancy occurs.

As you have pretty much said, I think that if a system is so robust that it works across many completely different behaving markets then chances are it is a very generalised system, - i.e more likely it will produce more modest but consistent returns (which IMO is better). This is because such a generalised system couldn't have exploited the specific 'weaknesses' in a specific market. But once you start doing that, it's probably curve fitted anyway, and large drawdowns could occur when conditions change (as they will). Compound interest is a powerful thing, if we can minimise the time spent in drawdowns, then compound interest will work much better and we will be much better of in 10-20 years time.

I am no expert in this, but the conclusion I'm slowly coming to is - Simple systems, consistent modest returns (over a variety of market conditions) with modest short term drawdowns, plus the power of compound interest = success. As long as we can stick to it!

 

[Billyb]

I have put allot of time into stock systems and basically came out with the conclusion that, especially with small caps, you need to take into account many factors that I am not capable of programming into metastock or amibroker.

What are some of these factors?

 

[sinner]

Just for the record, I agree with tech on this and disagree with others.

Forex trading systems generally do not work on stock indices. Stock index systems probably won't work trading small caps. None of those will be useful to trade soybeans or corn. Hell, to me it seems obvious that even the most rudimentary trend following system will probably work on EURUSD or GBPUSD where the liquidity is very high but will lose hand over fist on USDCAD.

Robustness has nothing to do with trading on multiple markets. I personally found good trading systems, and the best trading systems I coded were the ones which operated on a single instrument or group of related instruments (grouped by liquidity or market type).

I'm a compsci by profession and education. Robustness to me, is exactly as tech said. Run the same system with 5 or 10 different sets of parameters against the instrument. If it still is showing an edge then you have a robust setup.

If buying the 20MA cross works for DAX and Dow Jones components, but buying the 12.5/25/50MA cross only works on DAX components, then buying the 20MA cross is still a robust setup for DAX components.

 

[TabJockey]

What are some of these factors?

Usually news related events. If you are ontop of your macro news you get a feel for how its going to effect markets. And the way markets react gives even more insight into how they will react in the future to similar news. Sometimes the reaction to the news is immediate, and sometimes it takes days to sink into the mindset of market participants.

I remember Fukushima if you were on top of the knock on effects to uranium, you could have gotten out before the plunge.

Another thing that's hard to program into systems is slippage. You can make an allowance for it ofcoarce but I have found that those really big whopping wins in my backtest (50% in 2 days, ect) would not have had orders filled at anything close to the prices assumed.

 

[TabJockey]

Just for the record, I agree with tech on this and disagree with others.

I agree with everything in that post, I picked a bad example of ASX and AUD:USD(I think I meant NYSE?). A better example would be AUD:USD and EUR:USD. If you I get good results on one, and still ok results on the other, that is an indication of robustness to me.

 

[sinner]

I agree with everything in that post, I picked a bad example of ASX and AUD:USD(I think I meant NYSE?). A better example would be AUD:USD and EUR:USD. If you I get good results on one, and still ok results on the other, that is an indication of robustness to me.

I still have to disagree matey, but think we are close to consensus.

I have found, especially in forex, some systems which are extremely robust (i.e. they show an unleveraged positive expectancy using 5 or even 10 different sets of parameters) in one instrument show a horrible expectancy for the same sets of parameters in another instrument.

To me, that means these systems are robust within their universe (even if the universe is only 1 instrument).

For example,

One of my trend following systems works in EURUSD, showing a profitable positive expectancy with pretty much any set of parameters you plug into it.

However it loses horribly trading the same system on USDJPY.

To you, it seems that indicates a lack of robustness. I have to disagree.

I have another trend following system which provides a generally equal (over a large enough sample size) positive expectancy on all the forex majors. But the realised returns of "the generic forex trend following system" are far far far lower than the realised returns on "the EURUSD trend following system" or "the AUDUSD carry trade system" or "the GBPUSD momentum system".

This is not a matter of curve fitting, just realising and acting within the liquidity and market structure of each individual instrument. You simply can't expect the same behaviour or results from AUDUSD as trading the EURUSD which is orders of magnitude more liquid with completely different swap rates, volatility, interest time of day, etc.

So if profitability has anything to do with robustness, I think this is an important point to consider.