Newbie... portfolio backtesting results

[whacky]

Hi, I am newbie here so please dont get worked up if my queries appear to range between absurd and stupid
I would like to get some feedback on the portfolio backtest results I am currently getting.

I am quite concerned with "Max % Trade DD = -38%"!! However, the "Max % System DD = -19%" seems reasonable. Do you think this could be realistic? or do you think there is scope for improvement?

From: 01/01/2000 - 27/01/2014
Statistics
All trades Long trades Short trades
Initial capital 500000.00 500000.00 500000.00
Ending capital 2338287.00 2338287.00 500000.00
Net Profit 1838287.00 1838287.00 0.00
Net Profit % 367.66 % 367.66 % 0.00 %
Exposure % 35.05 % 35.05 % 0.00 %
Net Risk Adjusted Return % 1048.86 % 1048.86 % N/A
Annual Return % 11.58 % 11.58 % 0.00 %
Risk Adjusted Return % 33.03 % 33.03 % N/A
Total transaction costs 210500.00 210500.00 0.00
All trades 2105 2105 (100.00 %) 0 (0.00 %)
Avg. Profit/Loss 873.30 873.30 N/A
Avg. Profit/Loss % 0.78 % 0.78 % N/A
Avg. Bars Held 6.91 6.91 N/A
Winners 1355 (64.37 %) 1355 (64.37 %) 0 (0.00 %)
Total Profit 6100744.67 6100744.67 0.00
Avg. Profit 4502.39 4502.39 N/A
Avg. Profit % 3.47 % 3.47 % N/A
Avg. Bars Held 5.72 5.72 N/A
Max. Consecutive 18 18 0
Largest win 47081.18 47081.18 0.00
# bars in largest win 3 3 0
Losers 750 (35.63 %) 750 (35.63 %) 0 (0.00 %)
Total Loss -4262457.67 -4262457.67 0.00
Avg. Loss -5683.28 -5683.28 N/A
Avg. Loss % -4.07 % -4.07 % N/A
Avg. Bars Held 9.08 9.08 N/A
Max. Consecutive 11 11 0
Largest loss -62847.68 -62847.68 0.00
# bars in largest loss 12 12 0
Max. trade drawdown -67695.98 -67695.98 0.00
Max. trade % drawdown -38.64 % -38.64 % 0.00 %
Max. system drawdown -396773.46 -396773.46 0.00
Max. system % drawdown -19.01 % -19.01 % 0.00 %
Recovery Factor 4.63 4.63 N/A
CAR/MaxDD 0.61 0.61 N/A
RAR/MaxDD 1.74 1.74 N/A
Profit Factor 1.43 1.43 N/A
Payoff Ratio 0.79 0.79 N/A
Standard Error 229001.12 229001.12 0.00
Risk-Reward Ratio 0.62 0.62 N/A
Ulcer Index 6.73 6.73 0.00
Ulcer Performance Index 0.92 0.92 N/A
Sharpe Ratio of trades 0.75 0.75 0.00
K-Ratio 0.0422 0.0422 N/A
Expectancy ($) 873.30

Expectancy (%) 0.78

 

[whacky]

bump!

 

[g-topa-g]

bump!

I'm as Newbie as they come ...today in fact, with an interest in backtesting. What is bump?

And what system requires all that input in your original post?

 

[Wysiwyg]

I would like to get some feedback on the portfolio backtest results I am currently getting.

For a start you're running a test using a group of stocks that is different from the group of stocks 14 years ago. For example there might be 50% of the present stocks in the group from the test start 14 years ago which gives you a bias toward stock survivors for that 50%. (E.G. CBA $25 and BHP $9). In reality the Index constituents are changed quarterly.

To me, backtests are a rough idea of how a system might work. The proof is in the present market.

 

[Habakkuk]

To me, backtests are a rough idea of how a system might work. The proof is in the present market.

On that subject, Wysi (and others), assume you have a mildly profitable system, but nothing to write home about, single-digit returns. Hardly worth the effort. That may be on the ASX 200 or whatever stocks.
Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.

But is it real or am I fooling myself?

Is it reasonable to expect that most of these stocks will reverse their behaviour from the last 15 years? Winners turn into losers? Some of them can be expected to do so, obviously.

I don't know the answer but would like to find out without risking my money. Any ideas?

As you said, survivorship is a big factor in testing, a real no-no to use the current index constituents. But what if you used only the top 10? The 4 banks, 2 miners and TLS, WES, WOW, WPL, maybe a few more. They have been in the ASX20 for many years.
Does survivorship bias still apply then?

 

[tech/a]

If you run a Monty Carlo test on a system you'll get a chart of standard deviations
IE The highest return and lowest and all in between.

Its like giving your system to 10000 people and asking them to trade it for X years and then come back with the result.

This will tell you if you just happened to pick a portfolio that was traded which was mediocre.
I'm sure many a great system has been thrown out because the back testing and statistical data weren't
exhaustive.

 

[g-topa-g]

Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.

But is it real or am I fooling myself?

Is it reasonable to expect that most of these stocks will reverse their behaviour from the last 15 years? Winners turn into losers? Some of them can be expected to do so, obviously.

As you said, survivorship is a big factor in testing, a real no-no to use the current index constituents. But what if you used only the top 10? The 4 banks, 2 miners and TLS, WES, WOW, WPL, maybe a few more. They have been in the ASX20 for many years.
Does survivorship bias still apply then?

I too would be interested to know the answers to these questions teach/a...especially the last question.

 

[g-topa-g]

Oops sorry...tech/a...my bad

 

[Wysiwyg]

On that subject, Wysi (and others), assume you have a mildly profitable system, but nothing to write home about, single-digit returns. Hardly worth the effort. That may be on the ASX 200 or whatever stocks.
Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.

But is it real or am I fooling myself?

You would have to consider many periods of testing. One year results differs greatly or not to five year results? Tech/a Monte Carlo test suggestion. Test real time?

I'm afraid I can't offer much advice because I'm over backtest dreaming. At present simply using what I have learned so far to make decisions on what to buy, when to buy and when to sell.

As you said, survivorship is a big factor in testing, a real no-no to use the current index constituents. But what if you used only the top 10? The 4 banks, 2 miners and TLS, WES, WOW, WPL, maybe a few more. They have been in the ASX20 for many years.
Does survivorship bias still apply then?

You could 'reasonably' expect the top ten to be around for awhile so that universe would give a closer to reality result because your system is not benefiting from these stocks over other stocks that come and go.

Really though, to get an accurate backtest of an Index, the constituents would have to be as they were at the start date of the test and changed quarterly from the test start date forward. Otherwise you aren't testing all of the Index back then rather the stocks that have survived and are still in the Index today.

 

[g-topa-g]

You would have to consider many periods of testing. One year results differs greatly or not to five year results? Tech/a Monte Carlo test suggestion. Test real time?

I'm afraid I can't offer much advice because I'm over backtest dreaming. At present simply using what I have learned so far to make decisions on what to buy, when to buy and when to sell.

You could 'reasonably' expect the top ten to be around for awhile so that universe would give a closer to reality result because your system is not benefiting from these stocks over other stocks that come and go.

Really though, to get an accurate backtest of an Index, the constituents would have to be as they were at the start date of the test and changed quarterly from the test start date forward. Otherwise you aren't testing all of the Index back then rather the stocks that have survived and are still in the Index today.

Could this be why "they" keep changing the constituents i.e to skew any backtest result and isn't this why one would backtest a portfolio of stocks that have been around for as far back as one would choose to backtest?

Your portfolio of stocks becomes your Index so to speak. What's wrong with that?

 

[DeepState]

1. Could this be why "they" keep changing the constituents i.e to skew any backtest result

2. Your portfolio of stocks becomes your Index so to speak. What's wrong with that?

1. Uh, no.
2. Your sample is biased. It will consist of those companies that have survived through the period and you could not have known this at the start period. It is amongst the most egregious forms of over-fitting to do this.

On backtests, if you throw enough efforts, even by chance, a stack will look good. All sorts of rules can be invented and backfitted to produce unbelievable outcomes....even on random noise. There is no skill (beyond coding it) to producing an outstanding backtest. It is a large distance between finding one that looked good and it actually being predictive.

Not exactly sure how you would Monte Carlo a set decision rule on a set piece of data. Running 1 million of those yields a million identical outcomes.

Enjoy the journey of discovery that's ahead....

 

[g-topa-g]

1. Uh, no.
2. Your sample is biased. It will consist of those companies that have survived through the period and you could not have known this at the start period. It is amongst the most egregious forms of over-fitting to do this.

On backtests, if you throw enough efforts, even by chance, a stack will look good. All sorts of rules can be invented and backfitted to produce unbelievable outcomes....even on random noise. There is no skill (beyond coding it) to producing an outstanding backtest. It is a large distance between finding one that looked good and it actually being predictive.

Not exactly sure how you would Monte Carlo a set decision rule on a set piece of data. Running 1 million of those yields a million identical outcomes.

Enjoy the journey of discovery that's ahead....

No past performance is a guarantee of future performance. It is impossible to know for sure whether any company will survive at any time but surely one could pick some companies that one could reasonably expect to survive in future for as far back as they are being backtested. So maybe its not all about survivor bias (which one cannot argue against) but survivor probability.

 

[DeepState]

1. No past performance is a guarantee of future performance.

2. It is impossible to know for sure whether any company will survive at any time but surely one could pick some companies that one could reasonably expect to survive in future for as far back as they are being backtested. So maybe its not all about survivor bias (which one cannot argue against) but survivor probability.

1. Past performance conducted by backtest, based on methods which are pretty much data-mining in the guise of pattern recognition, will have virtually no relationship with the future. The class of strategies most frequently written of (longitudinal, time-series oriented, entirely price-based) is a total minefield of disaster for this effect. Just dig around the cemetery (the threads) and you'll find the bodies. New plots are being dug by the day.

2. So you'll only backtest those companies which have survived and that you expect to survive. That reduces the extent of bias in the backtest that might be conducted? This might be true with perfect foresight. Without any foresight, it is obviously laden with bias. The basis under which you choose to forecast survivorship will introduce new bias. You will have a very hard time understanding the impact of this.

I guess you'll have to jump in live/sim and learn these things over time. Others can jump in and explain the cost of that education and offer ways to circumvent it. For everyone who is still here posting, there would be around twenty or more who started at the same time that didn't make it. Of course, you'll be able to figure out who - of those - will still be around in three years and only look at what they have done in the past....

I guess they key message here is that your data will be imperfect. Bias is very important to understand and avoid. The short-cut you are proposing to take is pretty material for all the thought of concepts of probability of survival etc. Doing so introduces all sorts of hindsight biases and style biases of the type which is hard to observe and very fraught ("oh, I would have known that this measure would lead to a better outcome 10 years ago.."). Backtests are just a tool, yet, with the type of stuff usually attempted, where rules are developed on the basis of backtests and applied algorithmically and in a time series fashion, the key thing you will hopefully learn is that the mortality rate is through the roof. There will be a subset who attempt to do so that will succeed. Of those, you will have to discern which were genuinely skillful or the result of type-2 error, survived despite negative odds. Your probability of doing so, whilst it may feel rock solid without experience, will be closer to pure chance.

There is a reason why brokers produce programmable platforms and signals for customer use....I'll let you guess why. Proceed with caution. Good on you, though, for posting questions.

 

[tech/a]

Not exactly sure how you would Monte Carlo a set decision rule on a set piece of data. Running 1 million of those yields a million identical outcomes

He used the word Portfolio which implied to me that it was a group of stocks. As such Monte Carlo would be beneficial--as for survivorship---from 100s of tests on various strategies my findings have been that if there is a repeatable and profitable set of rules in one chart if the same setup occurs in another you'll see similar results.

By running 1000s of portfolios you will then be able to determine a set of likely results over those 1000s.
Out of sample test/walk forward test/live test---and provided there isn't a dramatic change in the real-time trading V the tested data set/s you'll get a similar profitable result.

You certainly have more time than me to answer in depth
But then again your probably a touch typist!

 

[DeepState]

He used the word Portfolio which implied to me that it was a group of stocks. As such Monte Carlo would be beneficial--as for survivorship---from 100s of tests on various strategies my findings have been that if there is a repeatable and profitable set of rules in one chart if the same setup occurs in another you'll see similar results.

By running 1000s of portfolios you will then be able to determine a set of likely results over those 1000s.
Out of sample test/walk forward test/live test---and provided there isn't a dramatic change in the real-time trading V the tested data set/s you'll get a similar profitable result.

You certainly have more time than me to answer in depth
But then again your probably a touch typist!

Building a process to develop a portfolio via algo on a fixed dataset will produce the same outcome no matter how many times you run it. This class of problems does not lend itself to Monte Carlo. If you are running a stack of algos through a dataset on whatever basis (walk-forward, backward, inverted etc...) that's not Monte Carlo. That's outright data-mining/exploration. You are describing a form of parameter surface creation for the purposes of finding something that shows a desired outcome.

As I have mentioned in exchanges with you elsewhere, there is zero skill (aside from coding) to jam a billion variations through whatever dataset you want including coin tosses or static electricity from outer space and finding a set that works. That process is data mining or, to make it seem more legitimate, signal processing (with apols to the engineers who use this in the right way for the right problems). Whatever your view on this, it is not Monte Carlo.

This response took 2 minutes to compile.

 

[AlterEgo]

Hi, I am newbie here so please dont get worked up if my queries appear to range between absurd and stupid
I would like to get some feedback on the portfolio backtest results I am currently getting.

I am quite concerned with "Max % Trade DD = -38%"!! However, the "Max % System DD = -19%" seems reasonable. Do you think this could be realistic? or do you think there is scope for improvement?

I wouldn't be concerned with a 38% trade drawdown. As long as you have your equity spread over several different stocks, a large drawdown in any one of them shouldn't have too large an impact on the overall portfolio. The system drawdown of 19% is the figure that I'd pay more attention to, and 19% is pretty reasonable.

But the annual return is fairly low. Avg Profit/Loss is quite low too. What trade universe are you using? The entire ASX, or the constituents of an index? I'd advise against using the constituents of an index for backtesting, as there would be a large amount of survivorship bias built in to that, as others have mentioned. And I hope you have allowed a realistic amount for commissions.

 

[AlterEgo]

For a start you're running a test using a group of stocks that is different from the group of stocks 14 years ago. For example there might be 50% of the present stocks in the group from the test start 14 years ago which gives you a bias toward stock survivors for that 50%. (E.G. CBA $25 and BHP $9). In reality the Index constituents are changed quarterly.

To me, backtests are a rough idea of how a system might work. The proof is in the present market.

Well to be fair, it doesn't actually say what group of stocks were used for the backtest. Yes, index constituents are changed quarterly, which is why I wouldn't base a backtest on them. There is no indication in the original post that the backtest used index constituents.

 

[AlterEgo]

On that subject, Wysi (and others), assume you have a mildly profitable system, but nothing to write home about, single-digit returns. Hardly worth the effort. That may be on the ASX 200 or whatever stocks.
Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.

But is it real or am I fooling myself?

You would be fooling yourself

Is it reasonable to expect that most of these stocks will reverse their behaviour from the last 15 years? Winners turn into losers? Some of them can be expected to do so, obviously.

I don't know the answer but would like to find out without risking my money. Any ideas?

As you said, survivorship is a big factor in testing, a real no-no to use the current index constituents. But what if you used only the top 10? The 4 banks, 2 miners and TLS, WES, WOW, WPL, maybe a few more. They have been in the ASX20 for many years.
Does survivorship bias still apply then?

Yes

Far better, IMO, to use the entire ASX as the tradable universe. That is, to allow the system to buy any stocks in the ASX that meet the buy criteria (and meet some minimum liquidity requirements). Don't restrict the tradable universe down to some select few stocks, or index constituents.

 

[AlterEgo]

Building a process to develop a portfolio via algo on a fixed dataset will produce the same outcome no matter how many times you run it.

No, it doesn't. You can have many different portfolios between runs. There are often several trade signals on a given day, but due to limited system equity your system may only be able to take say 1 or 2 of those signals. You can change the portfolio by making the system select a different set of valid signals each run.

 

[Wysiwyg]

Well to be fair, it doesn't actually say what group of stocks were used for the backtest. Yes, index constituents are changed quarterly, which is why I wouldn't base a backtest on them. There is no indication in the original post that the backtest used index constituents.

14 year test is almost certainly a stock portfolio using yahoo data and maybe even a trend trader using tech/a's conditions. You are correct though, it doesn't actually state this. :silly: