Attributes of a Solid System

[TabJockey]

When creating a fully mechanical system, what are the stand out stat's that make you say "yeah this is probably going to make money"?

I have been trying out different ideas in Amibroker, some of which have returned great profits over the last few years even unoptimised.

Amibroker does not give me the details I think necessary to evaluate a system, I have to copy and paste the trades into an excel template which works out what I want to know.

This is what I look for (and have not found yet):

an average of 8-9 profitable months per year over the last 5 years.

Assuming a constant position size, what is the average monthly win/loss? in dollars.

Small standard deviation of monthly win/loss, maybe about 1.5x the mean.

Example of good system performance:

Position Size: $5000 per trade
Average win/loss per month: $700
Standard Deviation of Monthly win/loss: $900


Before I was mainly looking at the metrics that amibroker gave you, such as expectancy per trade, annual return and max drawdown. What I found when I looked deeper into the backtest data for my best system was:

> Below average trades per month in 2005-2007, 1.2% Exp

> Very few trades during the GFC (which is good), ~-0.2% Exp.

> A whoppingly massive amount of trades during 2009, 4.8% Exp.

> Back to average performance in 2010-2011 1% Exp.

This basically tells me that the system, although profitable for a large portion of the last 5 years, is fantastic for 2009 market conditions (a bounce out of a crash) but pretty average for sideways markets (2010-2011) and even boom times (2005-2007).

So if I took this system live today, my historical average expectancy is 2% per trade, but can I realistically expect to encounter the same market conditions as I have back tested over in the last 5 years? No, 2009 is not likely to happen again soon is it? and by then you could expect the current market inefficiencies exploited by my system to have disappeared into the ether.

Sorry for the wall of text, my question to you is:

What metrics really stand out to you when back testing to indicate a system that would be worth taking live?

and

How do you determine the relevant time frames for effective back testing?


I am almost exclusively looking for swing trading systems (1-2 week trades) and not looking at day trading systems (too expensive) or LT trend following systems.

 

[colion]

FWIW, althought I don't use mechanical systems, I do look for certain characteristics with charting techniques the most important of which is a smoothly rising equity curve. Beyond that trading at a comfortable rate which also reflects stop loss criteria.

 

[markhocky]

Firstly, great idea for a thread TabJockey.

This is something that is always on my mind when reviewing results too.

I find myself looking more and more at the distribution of trades these days (i.e. histogram). This can give you a very quick idea if the overall results were due to a few outstanding (lucky) trades, or whether there is consistency.

The distribution can then be used for bootstrap tests ( http://www.automated-trading-system.com/bootstrap-test/) to check if they are "statistically" better than random.

The other thing I'm very keen on, which you've already highlighted, is looking at how the results change over time. I like to see that the results make sense. I mean, if you are testing a trend following system you want to see it make money in a trending market. I very much liked the breakdown of your trading returns, very insightful.

Lastly a comment regarding reviewing the equity curve. I totally agree it's good to see nice smooth equity curve, but, the "smoothness" of the equity curve will be largely due to the position sizing you use. If the equity curve is showing volatility which is lower than you are prepared to handle, maybe it's worth bumping up the position sizing a bit. Conversely, don't immediately throw out a system which generates big returns with high volatility. Maybe with smaller position sizing you can still achieve good returns, but with more acceptable swings.

Anyway, just my thoughts...

Cheers,
Mark.

 

[colion]

Lastly a comment regarding reviewing the equity curve. I totally agree it's good to see nice smooth equity curve, but, the "smoothness" of the equity curve will be largely due to the position sizing you use. If the equity curve is showing volatility which is lower than you are prepared to handle, maybe it's worth bumping up the position sizing a bit. Conversely, don't immediately throw out a system which generates big returns with high volatility. Maybe with smaller position sizing you can still achieve good returns, but with more acceptable swings.

For mechanical systems applied to a multiple symbol sample position size is one factor that could affect smoothness but any and all factors should be investigagted until the system trades in an acceptable manner to the user.

As I mentioned, I do not use a mechanical system but rather an indicator and charting-based analysis which is always applied to a single symbol (or if one likes a portfolio of fixed composition could be used). As a result, in my case, the equity curve is not a function of position size and its smoothness simply reflects the combined effect of the signals generated by the indicators and charting techniques used.

 

[TabJockey]

For mechanical systems applied to a multiple symbol sample position size is one factor that could affect smoothness but any and all factors should be investigagted until the system trades in an acceptable manner to the user.

As I mentioned, I do not use a mechanical system but rather an indicator and charting-based analysis which is always applied to a single symbol (or if one likes a portfolio of fixed composition could be used). As a result, in my case, the equity curve is not a function of position size and its smoothness simply reflects the combined effect of the signals generated by the indicators and charting techniques used.

Thats a great approach, I never considered using mechanical systems as an input into discretionary trading. If you had experience in both styles it could certainly pay dividends.

 

[markhocky]

For mechanical systems applied to a multiple symbol sample position size is one factor that could affect smoothness but any and all factors should be investigagted until the system trades in an acceptable manner to the user.

...

Thanks Colion. You're quite right that there are many factors that will ultimately effect the equity curve. Didn't mean to imply otherwise.

What I was trying to emphasize is that the size of your positions can have a big impact as well. Even if you are only trading one instrument, you still must have a position on (even if it's 100%). So, if you find that your system (mechanical or otherwise) is producing high returns but with too much volatility, don't discard it until you have considered refining how much you are committing to each trade.

 

[TabJockey]

After some investigation, the first thing I look at is profit factor, if its less than 1.6 then dont even bother! Then I look at the distribution of profit factor over the years, should always be above 1.6! I am looking for a low draw down method with high persistancy.

 

[tech/a]

High expectancy????

 

[TabJockey]

Clearly you would think that high expectancy is the first thing you would need, however I have a system here with 4000 trades over the last 5 years and an expectancy of 6.5% that is still junk.

 

[tech/a]

Clearly you would think that high expectancy is the first thing you would need, however I have a system here with 4000 trades over the last 5 years and an expectancy of 6.5% that is still junk.

How so?
Is it failing?
If it's returning 6.5 then?

 

[TabJockey]

How so?
Is it failing?
If it's returning 6.5 then?

I am talking about a backtested strategy that returned an expectancy of 6.5% over the 2005-2011 period. The average return per trade was very very high for 2009 and then negative for 2008 and 2011. Too much variation in the returns. 2009 is not likely to repeat again soon, so could I realistically expect that sort of expectancy in this system? Only if I traded it over about 20 years, and I doubt this method would stay viable for 10.

 

[tech/a]

I am talking about a backtested strategy that returned an expectancy of 6.5% over the 2005-2011 period. The average return per trade was very very high for 2009 and then negative for 2008 and 2011. Too much variation in the returns. 2009 is not likely to repeat again soon, so could I realistically expect that sort of expectancy in this system? Only if I traded it over about 20 years, and I doubt this method would stay viable for 10.

So it has an expectancy of 6.5%
NOT $6.50 for every dollar invested sorry huge difference
We appear to value expectancy quit differently!

Greetings from Spain!

6.5% is very poor
6.5:1 is very good hence my questions
Thanks for your reply

 

[TabJockey]

When you say $6.50 for every dollar invested, do you mean per trade or per year?

These systems that I am working on have average holding periods of 1-2 weeks, so a consistent 6.5% expectancy per trade would be immensely profitable, certainly not poor at all.

Im not sure what expectancy you use but I use ( % Wins * Average Win ) - (% Losses * Average Loss ).

Can you show me a backtest summary with $6.50 for every dollar invested? Just so I know exactly what your talking about.

 

[FxTrader]

When you say $6.50 for every dollar invested, do you mean per trade or per year?

These systems that I am working on have average holding periods of 1-2 weeks, so a consistent 6.5% expectancy per trade would be immensely profitable, certainly not poor at all.

Im not sure what expectancy you use but I use ( % Wins * Average Win ) - (% Losses * Average Loss ).

The expectancy of a trading system is what you can make on average per trade over many trades and is usually expressed as a multiple of risk.

For example, say you take 100 trades with your system and the maximum amount you will risk per trade is constant at $10 and the total profit from the trades is $200.

Your profit from this then is 20R (20 x $10 risk per trade). Divide 20R / 100 trades and the expectancy per trade is .2R. So then for every $1 risked you expect to make $.20 (an excellent system expectancy)

The calculation you're using here, ( % Wins * Average Win ) - (% Losses * Average Loss ) only describes the average profit or loss of your system over x number of trades as a profit/loss factor per trade without any refenence to risk. As a trader evaluating your system, I would need to know the average risk per trade before I could determine expectancy. The calculation should be...

Expectancy = (Probability of Win * Average Win) – (Probability of Loss * Average Loss)

Have a read of these links...
http://lionel.textmalaysia.com/the-holy-grail-of-trading-expectancy.html
http://www.tradermike.net/2004/05/trading_101_expectancy/

 

[tech/a]

TJ
All I have with me is my I Pad
No charts no data etc while travelling Spain and France
FX has helped out though.

 

[TabJockey]

Ok I know how to express expectancy in terms of an R value but I dont see how:

Expectancy = ( % Wins * Average Win ) - (% Losses * Average Loss )

Is different from your:
Expectancy = (Probability of Win * Average Win) – (Probability of Loss * Average Loss)

Also when talking R values, allot of my systems have a stop loss 3x the average loss. The system cuts losers short on technical rules, but also has a wide stop loss to catch nasty trades.

For example just picking one system out, the average loss is 8.07% and the average win is 39.15%.

36.16% of trades are winners, 63.84% of trades are losers.

The average profit per trade therefore is 9%. If you want to express this as an R value, would it not be 9/8.07 -1.12R ?

Or would you use maximum loss? 9/24 = 0.375R

Niether one is really "correct".

 

[FxTrader]

Ok I know how to express expectancy in terms of an R value but I dont see how:

Expectancy = ( % Wins * Average Win ) - (% Losses * Average Loss )

Is different from your:
Expectancy = (Probability of Win * Average Win) – (Probability of Loss * Average Loss)

% in this case is not equivalent to probability, probability considers position size (risk).

Consider this example from one of the links I provided...

As an example let’s say that a trader has a system that produces winning trades 30% of the time. That trader’s average winning trade nets 10% while losing trades lose 3%. So if he were trading $10,000 positions his expectancy would be:

(0.3 * $1,000) – (0.7 * $300) = $90

So even though that system produces losing trades 70% of the time the expectancy is still positive and thus the trader can make money over time. You can also see how you could have a system that produces winning trades the majority of the time but would have a negative expectancy if the average loss was larger than the average win:

(0.6 * $400) – (0.4 * $650) = -$20

Also when talking R values, allot of my systems have a stop loss 3x the average loss. The system cuts losers short on technical rules, but also has a wide stop loss to catch nasty trades.

For example just picking one system out, the average loss is 8.07% and the average win is 39.15%.

36.16% of trades are winners, 63.84% of trades are losers.

The average profit per trade therefore is 9%. If you want to express this as an R value, would it not be 9/8.07 -1.12R ?

Or would you use maximum loss? 9/24 = 0.375R

Niether one is really "correct".

Without knowing the risk/position size for each trade I can't calculate the R value for expectancy. Your stop loss is your risk for a trade unless you close the trade early, then the actual risk must be adjusted to the closing price of the trade.

If your system has a mechanical rule base for closing out losers prior to stops then you must use the closing prices as total risk in your expectancy calculation.

If you are rarely hitting stops though and closing early often then that would call into question how the system is determining stops (risk) to begin with. You would then need to analyse the impact of actually letting the trades continue to see if they hit their stop losses to determine the effectiveness of closing the losers early.

 

[TabJockey]

% in this case is not equivalent to probability, probability considers position size (risk).

Consider this example from one of the links I provided...

As an example let’s say that a trader has a system that produces winning trades 30% of the time. That trader’s average winning trade nets 10% while losing trades lose 3%. So if he were trading $10,000 positions his expectancy would be:

(0.3 * $1,000) – (0.7 * $300) = $90

So even though that system produces losing trades 70% of the time the expectancy is still positive and thus the trader can make money over time. You can also see how you could have a system that produces winning trades the majority of the time but would have a negative expectancy if the average loss was larger than the average win:

(0.6 * $400) – (0.4 * $650) = -$20




Without knowing the risk/position size for each trade I can't calculate the R value for expectancy. Your stop loss is your risk for a trade unless you close the trade early, then the actual risk must be adjusted to the closing price of the trade.

If your system has a mechanical rule base for closing out losers prior to stops then you must use the closing prices as total risk in your expectancy calculation.

If you are rarely hitting stops though and closing early often then that would call into question how the system is determining stops (risk) to begin with. You would then need to analyse the impact of actually letting the trades continue to see if they hit their stop losses to determine the effectiveness of closing the losers early.

Ah ok position size. At the moment I am leaving position size completely out of my development. I understand that increasing size when you have indication the trade has a better chance of being profitable and vice versa can increase the expectancy of the system but im trying to stay focused and work on only a few things at one time.

In my backtesting I assume a fixed position size of $5000 and a total equity of $100,000. Position size does not change with total equity. The only way this would pose a problem in the backtesting would be if I had a large drawdown at the start of the period and then my sample size was reduced for the rest of the period skewing the data to the beginning of the period. I have been on the lookout for this, and it has not been a factor in the thousands of system permutations I have tested in the last month.

Back to the original reason for starting this thread, can you give me your opinion of my original problem?

Expectancy over a back tested period is not enough to indicate that you have a good chance of the system being profitable in the future.

I experiment with allot of different exit strategies, stops at every level, only stops as an exit, all sorts of technical exits, nbar exits. Exits is probably where I spend most of my development time.

Also please remember I am only working on systems with average holding lengths of 5-20 days generating about 1000 signals a year. An edge of 1.11R may look small for a trend following system but for a 10day momentum system 1.11R would be wildly profitable, the key seems to be in the distribution of returns(both in time and magnitude).

 

[AlterEgo]

So it has an expectancy of 6.5%
NOT $6.50 for every dollar invested sorry huge difference
We appear to value expectancy quit differently!

Greetings from Spain!

6.5% is very poor
6.5:1 is very good hence my questions
Thanks for your reply

There’s nothing wrong with 6.5% (by which I take is an average profit per trade of 6.5%). That could be extremely high, if it's for a short-term system with a holding timeframe of a day or two! Expectancy, by itself, is a poor indicator of performance. It only really tells you whether a system is profitable or not - doesn’t tell you how profitable a system is. There’s no point in ranking systems by expectancy and choosing the system with the highest expectancy, as that system may in fact be the worst performer. As long as it’s positive, a small expectancy could produce an extremely high % profit per annum, or likewise a very large expectancy could produce a very low % profit per annum return, depending on how many trades the system produces per year, what amount of equity is allocated to each position, number of positions held, etc.

What’s more important is the average % per annum return. After all, that’s what we’re in this game for – to produce the highest profit possible. However, I am certainly not advocating optimizing a system for the highest profit irrespective of all other factors. But unless the system produces a high average % profit per annum it’s pointless trading it, whether the expectancy is high or not. All other factors being equal (similar drawdown characteristics, etc), I’d take the higher % return system over the lower % return system.

Like the OP though, I also like to see a very consistent return over time (eg. I don’t want to see yearly returns of +50%, -20%, +30%, -15%, etc – too much variation. +30%, +15%, +35%, +20% though would be very good). Ideally I’d like the equity curve to be close to a straight line. The system shouldn’t spend a lot of time in drawdown – system should quickly recover from drawdown periods. Maximum drawdown shouldn’t be excessive. I like to see most months produce a positive return – very few negative months. This may not be appropriate for a long term system, but I’m talking of shorter term systems.

 

[TabJockey]

Thanks AlterEgo that's what im talking about.

I have not yet been able to come up with a viable short system, so I always have a negative return in 2008 and 2011.

How many profitable months per year on average is your minimum? At the moment im getting 7-8 but that includes 2008. 10 is probably a good target right?