# Newbie... portfolio backtesting results



## whacky (19 February 2014)

Hi, I am newbie here so please dont get worked up if my queries appear to range between absurd and stupid  
I would like to get some feedback on the portfolio backtest results I am currently getting.

I am quite concerned with "Max % Trade DD = -38%"!!  However, the "Max % System DD = -19%" seems reasonable. Do you think this could be realistic? or do you think there is scope for improvement?

From: 01/01/2000 - 27/01/2014
Statistics
 	All trades	Long trades	Short trades
Initial capital	500000.00	500000.00	500000.00
Ending capital	2338287.00	2338287.00	500000.00
Net Profit	1838287.00	1838287.00	0.00
Net Profit %	367.66 %	367.66 %	0.00 %
Exposure %	35.05 %	35.05 %	0.00 %
Net Risk Adjusted Return %	1048.86 %	1048.86 %	N/A
Annual Return %	11.58 %	11.58 %	0.00 %
Risk Adjusted Return %	33.03 %	33.03 %	N/A
Total transaction costs	210500.00	210500.00	0.00
All trades	2105	2105 (100.00 %)	0 (0.00 %)
 Avg. Profit/Loss	873.30	873.30	N/A
 Avg. Profit/Loss %	0.78 %	0.78 %	N/A
 Avg. Bars Held	6.91	6.91	N/A
Winners	1355 (64.37 %)	1355 (64.37 %)	0 (0.00 %)
 Total Profit	6100744.67	6100744.67	0.00
 Avg. Profit	4502.39	4502.39	N/A
 Avg. Profit %	3.47 %	3.47 %	N/A
 Avg. Bars Held	5.72	5.72	N/A
 Max. Consecutive	18	18	0
 Largest win	47081.18	47081.18	0.00
 # bars in largest win	3	3	0
Losers	750 (35.63 %)	750 (35.63 %)	0 (0.00 %)
 Total Loss	-4262457.67	-4262457.67	0.00
 Avg. Loss	-5683.28	-5683.28	N/A
 Avg. Loss %	-4.07 %	-4.07 %	N/A
 Avg. Bars Held	9.08	9.08	N/A
 Max. Consecutive	11	11	0
 Largest loss	-62847.68	-62847.68	0.00
 # bars in largest loss	12	12	0
Max. trade drawdown	-67695.98	-67695.98	0.00
Max. trade % drawdown	-38.64 %	-38.64 %	0.00 %
Max. system drawdown	-396773.46	-396773.46	0.00
Max. system % drawdown	-19.01 %	-19.01 %	0.00 %
Recovery Factor	4.63	4.63	N/A
CAR/MaxDD	0.61	0.61	N/A
RAR/MaxDD	1.74	1.74	N/A
Profit Factor	1.43	1.43	N/A
Payoff Ratio	0.79	0.79	N/A
Standard Error	229001.12	229001.12	0.00
Risk-Reward Ratio	0.62	0.62	N/A
Ulcer Index	6.73	6.73	0.00
Ulcer Performance Index	0.92	0.92	N/A
Sharpe Ratio of trades	0.75	0.75	0.00
K-Ratio	0.0422	0.0422	N/A
Expectancy ($)	873.30	

Expectancy (%)	0.78


----------



## whacky (24 February 2014)

bump!


----------



## g-topa-g (17 March 2015)

whacky said:


> bump!




I'm as Newbie as they come ...today in fact, with an interest in backtesting. What is bump?

And what system requires all that input in your original post?


----------



## Wysiwyg (17 March 2015)

whacky said:


> I would like to get some feedback on the portfolio backtest results I am currently getting.



For a start you're running a test using a group of stocks that is different from the group of stocks 14 years ago. For example there might be 50% of the present stocks in the group from the test start 14 years ago which gives you a bias toward stock survivors for that 50%. (E.G. CBA $25 and BHP $9). In reality the Index constituents are changed quarterly.

To me, backtests are a rough idea of how a system might work. The proof is in the present market.


----------



## Habakkuk (17 March 2015)

Wysiwyg said:


> To me, backtests are a rough idea of how a system might work. The proof is in the present market.




On that subject, Wysi (and others), assume you have a mildly profitable system, but nothing to write home about, single-digit returns. Hardly worth the effort. That may be on the ASX 200 or whatever stocks.
Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.

But is it real or am I fooling myself?

Is it reasonable to expect that most of these stocks will reverse their behaviour from the last 15 years? Winners turn into losers? Some of them can be expected to do so, obviously.

I don't know the answer but would like to find out without risking my money. Any ideas?

As you said, survivorship is a big factor in testing, a real no-no to use the current index constituents. But what if you used only the top 10? The 4 banks, 2 miners and TLS, WES, WOW, WPL, maybe a few more. They have been in the ASX20 for many years.
Does survivorship bias still apply then?


----------



## tech/a (17 March 2015)

If you run a Monty Carlo test on a system you'll get a chart of standard deviations
IE The highest return and lowest and all in between.

Its like giving your system to 10000 people and asking them to trade it for X years and then come back with the result.

This will tell you if you just happened to pick a portfolio that was traded which was mediocre.
I'm sure many a great system has been thrown out because the back testing and statistical data weren't
exhaustive.


----------



## g-topa-g (17 March 2015)

Habakkuk said:


> Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.
> 
> But is it real or am I fooling myself?
> 
> ...




I too would be interested to know the answers to these questions teach/a...especially the last question.


----------



## g-topa-g (17 March 2015)

Oops sorry...tech/a...my bad


----------



## Wysiwyg (17 March 2015)

Habakkuk said:


> On that subject, Wysi (and others), assume you have a mildly profitable system, but nothing to write home about, single-digit returns. Hardly worth the effort. That may be on the ASX 200 or whatever stocks.
> Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.
> 
> But is it real or am I fooling myself?



You would have to consider many periods of testing. One year results differs greatly or not to five year results? Tech/a Monte Carlo test suggestion. Test real time? 

I'm afraid I can't offer much advice because I'm over backtest dreaming. At present simply using what I have learned so far to make decisions on what to buy, when to buy and when to sell. 



> As you said, survivorship is a big factor in testing, a real no-no to use the current index constituents. But what if you used only the top 10? The 4 banks, 2 miners and TLS, WES, WOW, WPL, maybe a few more. They have been in the ASX20 for many years.
> Does survivorship bias still apply then?



You could 'reasonably' expect the top ten to be around for awhile so that universe would give a closer to reality result because your system is not benefiting from these stocks over other stocks that come and go. 

Really though, to get an accurate backtest of an Index, the constituents would have to be as they were at the start date of the test and changed quarterly from the test start date forward. Otherwise you aren't testing all of the Index back then rather the stocks that have survived and are still in the Index today.


----------



## g-topa-g (17 March 2015)

Wysiwyg said:


> You would have to consider many periods of testing. One year results differs greatly or not to five year results? Tech/a Monte Carlo test suggestion. Test real time?
> 
> I'm afraid I can't offer much advice because I'm over backtest dreaming. At present simply using what I have learned so far to make decisions on what to buy, when to buy and when to sell.
> 
> ...





Could this be why "they" keep changing the constituents i.e to skew any backtest result and isn't this why one would backtest a portfolio of stocks that have been around for as far back as one would choose to backtest?

Your portfolio of stocks becomes your Index so to speak. What's wrong with that?


----------



## DeepState (17 March 2015)

g-topa-g said:


> 1. Could this be why "they" keep changing the constituents i.e to skew any backtest result
> 
> 2. Your portfolio of stocks becomes your Index so to speak. What's wrong with that?




1. Uh, no.
2. Your sample is biased.  It will consist of those companies that have survived through the period and you could not have known this at the start period.  It is amongst the most egregious forms of over-fitting to do this.

On backtests, if you throw enough efforts, even by chance, a stack will look good. All sorts of rules can be invented and backfitted to produce unbelievable outcomes....even on random noise.  There is no skill (beyond coding it) to producing an outstanding backtest.  It is a large distance between finding one that looked good and it actually being predictive.

Not exactly sure how you would Monte Carlo a set decision rule on a set piece of data.  Running 1 million of those yields a million identical outcomes.

Enjoy the journey of discovery that's ahead....


----------



## g-topa-g (17 March 2015)

DeepState said:


> 1. Uh, no.
> 2. Your sample is biased.  It will consist of those companies that have survived through the period and you could not have known this at the start period.  It is amongst the most egregious forms of over-fitting to do this.
> 
> On backtests, if you throw enough efforts, even by chance, a stack will look good. All sorts of rules can be invented and backfitted to produce unbelievable outcomes....even on random noise.  There is no skill (beyond coding it) to producing an outstanding backtest.  It is a large distance between finding one that looked good and it actually being predictive.
> ...






No past performance is a guarantee of future performance. It is impossible to know for sure whether any company will survive at any time but surely one could pick some companies that one could reasonably expect to survive in future for as far back as they are being backtested. So maybe its not all about survivor bias (which one cannot argue against) but survivor probability.


----------



## DeepState (18 March 2015)

g-topa-g said:


> 1. No past performance is a guarantee of future performance.
> 
> 2. It is impossible to know for sure whether any company will survive at any time but surely one could pick some companies that one could reasonably expect to survive in future for as far back as they are being backtested. So maybe its not all about survivor bias (which one cannot argue against) but survivor probability.





1. Past performance conducted by backtest, based on methods which are pretty much data-mining in the guise of pattern recognition, will have virtually no relationship with the future.  The class of strategies most frequently written of (longitudinal, time-series oriented, entirely price-based) is a total minefield of disaster for this effect.  Just dig around the cemetery (the threads) and you'll find the bodies.  New plots are being dug by the day.

2. So you'll only backtest those companies which have survived and that you expect to survive.  That reduces the extent of bias in the backtest that might be conducted?  This might be true with perfect foresight.  Without any foresight, it is obviously laden with bias.  The basis under which you choose to forecast survivorship will introduce new bias.  You will have a very hard time understanding the impact of this.

I guess you'll have to jump in live/sim and learn these things over time.  Others can jump in and explain the cost of that education and offer ways to circumvent it.  For everyone who is still here posting, there would be around twenty or more who started at the same time that didn't make it.  Of course, you'll be able to figure out who - of those - will still be around in three years and only look at what they have done in the past....

I guess they key message here is that your data will be imperfect.  Bias is very important to understand and avoid.  The short-cut you are proposing to take is pretty material for all the thought of concepts of probability of survival etc.  Doing so introduces all sorts of hindsight biases and style biases of the type which is hard to observe and very fraught ("oh, I would have known that this measure would lead to a better outcome 10 years ago..").  Backtests are just a tool, yet, with the type of stuff usually attempted, where rules are developed on the basis of backtests and applied algorithmically and in a time series fashion, the key thing you will hopefully learn is that the mortality rate is through the roof.  There will be a subset who attempt to do so that will succeed.  Of those, you will have to discern which were genuinely skillful or the result of type-2 error, survived despite negative odds.  Your probability of doing so, whilst it may feel rock solid without experience, will be closer to pure chance.

There is a reason why brokers produce programmable platforms and signals for customer use....I'll let you guess why.  Proceed with caution.  Good on you, though, for posting questions.


----------



## tech/a (18 March 2015)

> Not exactly sure how you would Monte Carlo a set decision rule on a set piece of data. Running 1 million of those yields a million identical outcomes




He used the word Portfolio which implied to me that it was a group of stocks. As such Monte Carlo would be beneficial--as for survivorship---from 100s of tests on various strategies my findings have been that if there is a repeatable and profitable set of rules in one chart if the same setup occurs in another you'll see similar results.

By running 1000s of portfolios you will then be able to determine a set of likely results over those 1000s.
Out of sample test/walk forward test/live test---and provided there isn't a dramatic change in the real-time trading V the tested data set/s you'll get a similar profitable result.

You certainly have more time than me to answer in depth
But then again your probably a touch typist!


----------



## DeepState (18 March 2015)

tech/a said:


> He used the word Portfolio which implied to me that it was a group of stocks. As such Monte Carlo would be beneficial--as for survivorship---from 100s of tests on various strategies my findings have been that if there is a repeatable and profitable set of rules in one chart if the same setup occurs in another you'll see similar results.
> 
> By running 1000s of portfolios you will then be able to determine a set of likely results over those 1000s.
> Out of sample test/walk forward test/live test---and provided there isn't a dramatic change in the real-time trading V the tested data set/s you'll get a similar profitable result.
> ...




Building a process to develop a portfolio via algo on a fixed dataset will produce the same outcome no matter how many times you run it.  This class of problems does not lend itself to Monte Carlo.  If you are running a stack of algos through a dataset on whatever basis (walk-forward, backward, inverted etc...) that's not Monte Carlo.  That's outright data-mining/exploration.  You are describing a form of parameter surface creation for the purposes of finding something that shows a desired outcome.  

As I have mentioned in exchanges with you elsewhere, there is zero skill (aside from coding) to jam a billion variations through whatever dataset you want including coin tosses or static electricity from outer space and finding a set that works. That process is data mining or, to make it seem more legitimate, signal processing (with apols to the engineers who use this in the right way for the right problems).  Whatever your view on this, it is not Monte Carlo. 

This response took 2 minutes to compile.


----------



## AlterEgo (18 March 2015)

whacky said:


> Hi, I am newbie here so please dont get worked up if my queries appear to range between absurd and stupid
> I would like to get some feedback on the portfolio backtest results I am currently getting.
> 
> I am quite concerned with "Max % Trade DD = -38%"!!  However, the "Max % System DD = -19%" seems reasonable. Do you think this could be realistic? or do you think there is scope for improvement?




I wouldn't be concerned with a 38% trade drawdown. As long as you have your equity spread over several different stocks, a large drawdown in any one of them shouldn't have too large an impact on the overall portfolio. The system drawdown of 19% is the figure that I'd pay more attention to, and 19% is pretty reasonable.

But the annual return is fairly low. Avg Profit/Loss is quite low too. What trade universe are you using? The entire ASX, or the constituents of an index? I'd advise against using the constituents of an index for backtesting, as there would be a large amount of survivorship bias built in to that, as others have mentioned. And I hope you have allowed a realistic amount for commissions.


----------



## AlterEgo (18 March 2015)

Wysiwyg said:


> For a start you're running a test using a group of stocks that is different from the group of stocks 14 years ago. For example there might be 50% of the present stocks in the group from the test start 14 years ago which gives you a bias toward stock survivors for that 50%. (E.G. CBA $25 and BHP $9). In reality the Index constituents are changed quarterly.
> 
> To me, backtests are a rough idea of how a system might work. The proof is in the present market.




Well to be fair, it doesn't actually say what group of stocks were used for the backtest. Yes, index constituents are changed quarterly, which is why I wouldn't base a backtest on them. There is no indication in the original post that the backtest used index constituents.


----------



## AlterEgo (18 March 2015)

Habakkuk said:


> On that subject, Wysi (and others), assume you have a mildly profitable system, but nothing to write home about, single-digit returns. Hardly worth the effort. That may be on the ASX 200 or whatever stocks.
> Now you go and backtest every stock individually with that system and keep only the good ones for your portfolio. You can easily double your return and reduce drawdown that way. The Sharpe ratio will be impressive.
> 
> But is it real or am I fooling myself?




You would be fooling yourself



Habakkuk said:


> Is it reasonable to expect that most of these stocks will reverse their behaviour from the last 15 years? Winners turn into losers? Some of them can be expected to do so, obviously.
> 
> I don't know the answer but would like to find out without risking my money. Any ideas?
> 
> ...




Yes

Far better, IMO, to use the entire ASX as the tradable universe. That is, to allow the system to buy any stocks in the ASX that meet the buy criteria (and meet some minimum liquidity requirements). Don't restrict the tradable universe down to some select few stocks, or index constituents.


----------



## AlterEgo (18 March 2015)

DeepState said:


> Building a process to develop a portfolio via algo on a fixed dataset will produce the same outcome no matter how many times you run it.




No, it doesn't. You can have many different portfolios between runs. There are often several trade signals on a given day, but due to limited system equity your system may only be able to take say 1 or 2 of those signals. You can change the portfolio by making the system select a different set of valid signals each run.


----------



## Wysiwyg (18 March 2015)

AlterEgo said:


> Well to be fair, it doesn't actually say what group of stocks were used for the backtest. Yes, index constituents are changed quarterly, which is why I wouldn't base a backtest on them. There is no indication in the original post that the backtest used index constituents.



14 year test is almost certainly a stock portfolio using yahoo data and maybe even a trend trader using tech/a's conditions. You are correct though, it doesn't actually state this.  :silly:


----------



## tech/a (18 March 2015)

AlterEgo said:


> No, it doesn't. You can have many different portfolios between runs. There are often several trade signals on a given day, but due to limited system equity your system may only be able to take say 1 or 2 of those signals. You can change the portfolio by making the system select a different set of valid signals each run.




No I think he's right.
Think of your universe as the ASX 100
You can Monte Carlo your brains out and your
still only getting a result for THAT data set.

Now go trade it on the FTSE 100
or all stocks OUTSIDE the ASX 100--see how that goes.

Haven't finished but my 2 mins is up

Then there is the issue of are in the dataset now but 10 yrs ago where were they.


----------



## DeepState (18 March 2015)

AlterEgo said:


> No, it doesn't. You can have many different portfolios between runs. There are often several trade signals on a given day, but due to limited system equity your system may only be able to take say 1 or 2 of those signals. You can change the portfolio by making the system select a different set of valid signals each run.




When you alter your decision rules ('making the system select a different...') as per above, you are creating a variant algo with different deterministic parameters.  Keep these new parameters and run this a billions times and you get the same outcome each time for this fixed set of parameters.  You have altered the algo.  What you are doing is varying decision parameters and calling it Monte Carlo.  It isn't.  It is an examination of outcomes based on varying input parameters.   

For completeness: If, for some perverse reason, you have some sort of stochastic rule to select different trades rules at different periods despite that fact the experience was identical to that point, (eg. "we'll use MA[Min(Bi(10,3),24),25] last period then regime change matrix potentially to Bollinger[2,U(5,10)] this coming period"), you will get a distribution of outcomes for a fixed dataset.  If this is how you create a decision rule whose parameters vary in mid-flight despite a fixed dataset, there may be a few more issues than definitions about Monte Carlo simulations that need to be addressed first.

In the world of finance, Monte Carlo simulations are not created by perturbing the decision rule inputs on a fixed dataset (incl the variations that Tech was talking about).  Doing so is creating a parameter hyper-surface/plane.  

Monte-Carlo simulations are conducted routinely to assess and value the price of derivatives or to estimate the outcome of some decision rule where the historical outcomes of the variable in question is used to inform some stochastic process.  For example, calculation of VaR can be calculated in several ways.  One of them is to resample (with or without replacement) from some block of history (where vertical and/or horizontal integrity might or might not be preserved).  Same calculation/decision rule, different datasets get produced via resampling or other stochastic process, voila - distributional outcome.

If you use a single set of parameters and design spec for your algo but produce a dataset that is resampled or otherwise abstracted in some fashion via some estimated stochastic model whose innovations are genuinely random, that will yield a Monte Carlo simulation.    

These two concepts are materially different and should not be interchanged.  They serve very different purposes and are used in different circumstances.  To make things hard, the two can be run concurrently.

--

Monte Carlo has a casino which is where this concept finds its roots.  They play roulette there.  A Monte-Carlo simulation helps determine what the distribution of wins and losses (alternatively - reds/blacks/green) might be for the game of roulette.  The game of roulette does not change its parameters.  The same sorts of things can be determined for all casino games whose rules do not change (yea yea with card counters and all that).  Distributional outcomes are obtained based on random perturbations with no change in design parameters.  Unless you are saying that the decision rule for trading has large random components for strategy selection, you are arguing that changing a set of deterministic decision parameters produces a Monte Carlo analysis.  It does not.  There is a world of difference between these concepts.


----------



## DeepState (18 March 2015)

tech/a said:


> Haven't finished but my 2 mins is up




you funny.... 
(5 sec)


----------



## tech/a (18 March 2015)

> When you alter your decision rules ('making the system select a different...') as per above, you are creating a variant algo with different deterministic parameters. Keep these new parameters and run this a billions times and you get the same outcome each time for this fixed set of parameters. You have altered the algo. What you are doing is varying decision parameters and calling it Monte Carlo. It isn't. It is an examination of outcomes based on varying input parameters




That's not what he's doing or saying.


----------



## DeepState (18 March 2015)

tech/a said:


> That's not what he's doing or saying.




Then what is he saying or doing if not that?


----------



## Habakkuk (18 March 2015)

DeepState said:


> Then what is he saying or doing if not that?




The technique referred to as Monte Carlo by tech/a and Alter Ego is a simple trade-substitution exercise.
There are no parameters as such being changed and no parameter hyper-surface.
It's simply the path-dependence of a system that signals multiple (buy)trades and insufficient money to take all of them.
At least that's what I understand. Whether it's valid or useful is another matter.


----------



## DeepState (18 March 2015)

Habakkuk said:


> The technique referred to as Monte Carlo by tech/a and Alter Ego is a simple trade-substitution exercise.
> There are no parameters as such being changed and no parameter hyper-surface.
> It's simply the path-dependence of a system that signals multiple (buy)trades and insufficient money to take all of them.
> At least that's what I understand. Whether it's valid or useful is another matter.




For a fixed path which is historical market realisation of any type that is factual, what exactly changes to produce different portfolio outcomes between runs if not some change to a decison rule parameter?


----------



## AlterEgo (18 March 2015)

DeepState said:


> For a fixed path which is historical market realisation of any type that is factual, what exactly changes to produce different portfolio outcomes between runs if not some change to a decison rule parameter?




What changes is which trade of the many valid signals are taken on a given day. Say you get 10 trade signals on a given day, but have funds for 2. First run may take trades A & B, next run may take trades C & D, etc. Then as C & D may have different trade durations than A & B, this alters the trading equity available in following days, so following trades will be different.


----------



## AlterEgo (18 March 2015)

Habakkuk said:


> The technique referred to as Monte Carlo by tech/a and Alter Ego is a simple trade-substitution exercise.




I'm not calling what I do Monte Carlo. But it does show a different path through the same data using the same rules, so gives a different equity curve, different stats, etc. So gives a better picture of the system performance than just looking a 1 of the possible paths.

And then you'd want to check it on a different data set as well (different time period or different market) to see if it still works. And then if all goes well, test it in the real world.


----------



## DeepState (18 March 2015)

AlterEgo said:


> What changes is which trade of the many valid signals are taken on a given day. Say you get 10 trade signals on a given day, but have funds for 2. First run may take trades A & B, next run may take trades C & D, etc. Then as C & D may have different trade durations than A & B, this alters the trading equity available in following days, so following trades will be different.




Thanks for the expansion.  I just want to make sure that I understand your perspective given the above.... 

For a given set-up in decision making and set of history (ie. fixed datablock, starting time, assets/equity etc)... if you got the chance to run the process twice without memory, do you end up doing the same thing in each run through of history/backtest (ie. 'First run may take trades A & B, next run may take trades C & D, etc. Then as C & D may have different trade durations than A & B, this alters the trading equity available in following days, so following trades will be different.')?  If so, you end up producing the same outcome for each full backtest run.  Yes, the actual strategy mix used at each point can change, but the outcome for the exact set-up is identical each time you do it. Once or a billion times.  You will use A&B, then C&D etc. each run.  You will end up with exactly the same P&L for each run. 

Yes, each bit of the underlying strategy (A, B, C, D etc.) run in pure isolation will yield its own P&L which will be fixed if nothing else is changed.  These can be helpful in deciding how to blend them.

If not, and the strategy deployment changes from the first backtest, why would two runs with identical starting conditions and exactly the same set of signal availability, equity etc. lead to a different outcome?  That is, is there some magic which changes your allocation from A&B in one run for the same period in one backtest to A&C in a second? B&D in a third backtest for the same period?  Introducing some randomness (hence changing the path of strategy deployment for different backtests despite otherwise identical starting conditions) will produce different P&L for each backtest run and will produce a Monte Carlo simulation outcome.

The process of determining which strategies to outlay given available equity at each decision point is part of an overall decision rule.  Just because you can change the sub-strategies deployed between periods doesn't produce a distribution of outcomes if the decision to deploy them remains fixed.  It is one decision rule with several bits to it.

----

If, by saying, 



AlterEgo said:


> I'm not calling what I do Monte Carlo.




...I am not to equate your approach as Monte Carlo after all.  If so, then we are on the same page.  In disagreeing with what constitutes Monte Carlo and then offering up what you do in the same paragraph as an explanation why in a prior post, it implied (to me) that your process was believed to be Monte Carlo (which it is if you randomly select sub-strategies or otherwise change them on the fly in some random way.  Not otherwise).

In the end, what matters is that tests and outcomes are understood and used properly.



AlterEgo said:


> But it does show a different path through the same data using the same rules, so gives a different equity curve, different stats, etc. So gives a better picture of the system performance than just looking a 1 of the possible paths.
> 
> And then you'd want to check it on a different data set as well (different time period or different market) to see if it still works. And then if all goes well, test it in the real world




I get the sense you do. Newbies take note.


----------



## tech/a (18 March 2015)

DeepState said:


> Then what is he saying or doing if not that?




He's not altering the algo.
Hes running (I presume) the same algo on countless trading possibilities.

For Me the Monty Carlo sim I use if from Tradesim---David Samborsky
So to attempt to explain what the software does which I think is what A/E means.

Lets say I have 10 triggers on Day 1----20 on day 2 and 30 on day three etc etc.
10 x 19 x 29 combinations of trades for a portfolio opportunity----ad infinitum---with a big enough universe.
Then----some will come off a trade get stopped out or exited all at very different times so the new trades will all be infinitely different as each portfolio trades its own path.

From the combination of a huge number of portfolio's you'll get a scatter of the results* within that universe over that period tested.*

Whether that then computes to similar results on a different data set and indeed any data set and of course going forward is not known.
But to argue against is to suggest that there is no patterns or occurrences other than flukes in any data set that will result in a quantifiable edge that can be expressed as an algo.

Which gets back to what I think your argument is and that's the Efficient Market Hypothesis
Which means this thread is going to run for a few years of 2-60 min posts.


----------



## Wysiwyg (18 March 2015)

tech/a said:


> He's not altering the algo.
> Hes running (I presume) the same algo on countless trading possibilities.



This introduction of randomness by adding an algorithm to the base code returns a best and worst case scenario. In actual trading one hopes the results lie somewhere within that range. Tomasz J. uses the "Random" algorithm in this formula ...  http://traders.com/Documentation/FEEDbk_Docs/2009/03/TradersTips.html#TT3


```
// General-purpose MC part 
HowManyMCSteps = 20000; // adjust that to change the number of MC tests 

PositionScore = 100 * mtRandomA(); 
// that is single-line that causes random picking of signals 

Step = Optimize(“Step”, 1, 1, HowManyMCSteps , 1 ); 
// this is dummy variable, not used below 

// The trading system itself 
// ( you may enter your own system below instead of one from the article ) 

NumPos = 8; // maximum number of open positions 
SetOption(“MaxOpenPositions”, NumPos ); 
SetPositionSize( GetOption(“InitialEquity”) / NumPos, spsValue ); 
// as in the article - no compounding of profits 
// SetPositionSize( 100 / NumPos, spsPercentOfEquity ); 
// uncomment this for compounding profits 

// signals 
s = Signal( 12, 26, 9 ); 
m = MACD( 12, 26 ); 
Buy = Cross( s, m ); 
Sell = Cross( m, s ); 

SetTradeDelays( 1, 1, 1, 1 ); // trade with one bar delay on open price 
BuyPrice = Open; 
SellPrice = Open;

 
—Tomasz Janeczko
amibroker.com
```


----------



## tech/a (18 March 2015)

Which is not Monty Carlo analysis
David Samborsk of Tradesim has Monty Carlo analysis which 
Is superior.


----------



## Wysiwyg (18 March 2015)

tech/a said:


> Which is not Monty Carlo analysis
> David Samborsk of Tradesim has Monty Carlo analysis which
> Is superior.



 There has to be some code added to the base code so the program can "choose" different stocks from the candidates every pass. If it is not random then it is defined which is adding a new "condition" from which to choose from the candidates. 

Amibroker has position score which ranks the candidates. This to me is the same as adding that position score condition to the buy conditions but the difference has never been explained to me.


----------



## tech/a (18 March 2015)

Wysiwyg said:


> There has to be some code added to the base code so the program can "choose" different stocks from the candidates every pass. If it is not random then it is defined which is adding a new "condition" from which to choose from the candidates.
> 
> Amibroker has position score which ranks the candidates. This to me is the same as adding that position score condition to the buy conditions but the difference has never been explained to me.




I'll contact David to see if he minds me pasting stuff up of his under copywrite
It explains what he does ( his software ) much better than I can


----------



## DeepState (18 March 2015)

tech/a said:


> He's not altering the algo.
> Hes running (I presume) the same algo on countless trading possibilities.
> 
> ---
> ...





Seems we both have a few minutes on our hands...

If he is not altering the algo then the decision rule is set in stone.  If the sign says "turn right" and the algo says "if strSIGN=="RIGHT" then fn_TURN_RIGHT end; (in Matlab-ese anyway)" it will always turn right at that point. This algo will develop a stack of levels at which trades will take place, in aggregate, whether they are sourced from sub-strategy A, B...Z.  So, you start at the start of Day 1 and have a stack of triggers (could be millions) and, at each node, there exists a pathway of further triggers (trillions, even), multiplying out as you say (to gazillions and beyond).  However many you like, however long you like.

If you do this over the same dataset, you end up going down the same path, in the garden/thicket of forking paths, every time.  The market pathway has already been pre-determined.  It actually happened already. Back-testing the decision rules trigger the same action at the same points each time because the data wasn't changed. The algo is not changed (in a deterministic sense). So it cannot motivate different outcomes on the same data. There are an incredibly large number of possibilities that could have happened had the data turned out differently, but the data turned out one way.  Having a gabagillion additional possible trading levels related to market situations that did not occur changes nothing on the outcome. History ('that universe over that period tested'). Repeat history a billion times, you get the same history.  You do not get a distribution of outcomes.  You get a distribution of one point (whatever it is, the rest are the same), realized as many times as you want to run this experiment until you give up thinking something different could happen with the next (re)run. 

Whatever it is that scatters, it won't be the outcome of a backtest run on the same data with a fixed algo which reacts the same way each time.

If you run this on a different dataset, the outcome will likely differ (unless, in the freak, the different dataset turned out to be the same one as before).  Run this new dataset through a gababillion times and it too will yield one outcome over and over again.  Just a (likely) different figure to the first dataset.

With a fixed algo (even if it consists of a stack of sub-strategies that flick on and off or otherwise vary by some fixed rule), you will only get a scatter if your data sample changes. For a Monte Carlo analysis, this is how you get distributions for a particular, deterministic, decision rule (ie. not one that changes on the fly and would do different things even with the same dataset).  Typically, this is achieved via resampling of some type or reducing the observed distributions into something compact (which is loaded with its own assumptions) and then recreating some form of alternative history.  Doing this can produce as many alternative pathways as you want.

I am not saying that all patterns are useless.  However, if you chuck a billion algo variants into random noise, you will find patterns.  Do the same on market data and you get something virtually indistinguishable.  Virtually.  Real live trading outcomes, you've seen the mortality rate, suggest that however traders determine their decision rules, it turns out to produce an outcome indistinguishable to pure luck (or worse) less expenses for the most part (the distribution of profit and what it means to be 'above average' is a whole other topic).

I make the point that backtests based on time-series, price-only, longitudinal varieties (seemingly the most common discussed) have a very poor set-up to finding meaningful patterns - particularly if you think you can outgun the hard core algos out there who use a lot more info and can trade much faster.  However, this type of setup has beautiful properties to produce patterns which appear meaningful and motivate trades.  

If this should descend into a debate on EMH, you won't find me arguing for the affirmative.


----------



## g-topa-g (18 March 2015)

DeepState said:


> 1. Past performance conducted by backtest, based on methods which are pretty much data-mining in the guise of pattern recognition, will have virtually no relationship with the future.  The class of strategies most frequently written of (longitudinal, time-series oriented, entirely price-based) is a total minefield of disaster for this effect.  Just dig around the cemetery (the threads) and you'll find the bodies.  New plots are being dug by the day.
> 
> 2. So you'll only backtest those companies which have survived and that you expect to survive.  That reduces the extent of bias in the backtest that might be conducted?  This might be true with perfect foresight.  Without any foresight, it is obviously laden with bias.  The basis under which you choose to forecast survivorship will introduce new bias.  You will have a very hard time understanding the impact of this.
> 
> ...




Not "perfect" foresight just "probable" foresight.

I think I have my answers. Thanks for the feedback


----------



## tech/a (18 March 2015)

g-topa-g said:


> Not "perfect" foresight just "probable" foresight.
> 
> I think I have my answers. Thanks for the feedback




Oh damn we stop here!


----------



## DeepState (18 March 2015)

g-topa-g said:


> Not "perfect" foresight just "probable" foresight.
> 
> I think I have my answers. Thanks for the feedback




You're welcome.  

I'm curious.  If there are 1,000 stocks out there and the mortality rate is about 5% per year:

Q1: How much more likely do you think your approach will adjust that figure towards zero for an ASX type universe of your selection?  

---

If with perfect foresight, this would make your forward looking universe 950 for a one year backtest (we run backtests only so long as we are confident that the company will survive for that long in the future)...leading to about half the original universe surviving for 14 years.  So, of the stocks which survived the last 14 years (a fairly small number relative to stocks currently alive in reality) intact, only half could be used for a full backtest if the standard of:



g-topa-g said:


> ...surely one could pick some companies that one could reasonably expect to survive in future for as far back as they are being backtested...




...is to be utilized.
----

Then, if you are a genius, and can narrow down the universe of death, by choosing a subset of the likely survivors that have a perfect chance of surviving in a given year.  With 100% foresight you halve the survivors suitable for a full back-test per above.  You choose exactly 950 stocks for Year 1 and they are the ones which expire.

Let's say that your skills are such that you can select a subset of just 900 stocks where the survival is 100% rather than the 95% for the full universe.  In other words, there are 100 stocks which you felt their survival was marginal and booted them in Year 1. All deaths for that year are drawn from this universe (pretty amazing really...because it's just a little tougher than that - just ask S&P, Merrill Lynch, Goldman Sachs etc.).

Your universe will shrink down to just 23% of the start point content for a full backtest.  If reality were any larger, and, if you will work with me at $5m per year plus upside, and I'll take you on tomorrow - heck, tonight.

--- 

Q2: How representative will those 230 stocks (or whatever fairly small number) be for the purposes of uncovering patterns which are meaningful for projection?   What new bias have you introduced by slicing the universe into half relative to perfect insight?

---

Oh, but wait, we can test on bigger universes for shorter time periods....



This is nowhere near as straightforward as might be imagined.  All the best with the adventure ahead.


----------



## tech/a (18 March 2015)

My ignorance and luck has served me well.
As I have clearly in correctly designed and traded systems profitably
Very profitably.

Your solution then is not systems developement and trading them.

What is?


----------



## DeepState (18 March 2015)

tech/a said:


> My ignorance and luck has served me well.
> As I have clearly in correctly designed and traded systems profitably
> Very profitably.
> 
> ...




So having a platform and getting some backtest that showed, you know: low drawdown, high ratios etc. and trading it is sufficient?  Apparently....that's all there is to it.  My bad!


----------



## tech/a (18 March 2015)

DeepState said:


> So having a platform and getting some backtest that showed, you know: low drawdown, high ratios etc. and trading it is sufficient?  Apparently....that's all there is to it.  My bad!




Well you have to trade it!

It has been in my case
obviously an outlier.

Still I'm interested in your alternative


----------



## DeepState (18 March 2015)

tech/a said:


> Well you have to trade it!
> 
> It has been in my case
> obviously an outlier.
> ...




Well, in order to win, you have to play.  So having to trade it is necessary for investment success.  Of course, trading often doesn't lead to success.

The missing pieces are finding something in the patterns that has a shot of actually being sustainable and predictive, re-hashing that to produce something which can be implemented, risk managing and carefully implementing it. Just those minor details. Of the number of backtests which can be produced that show a good outcome, the proportion of which contains any decent predictive ability is vanishingly small.

Something which fails backtests can reasonably be included.

---

Here's an example to just give this a tad of flavor.

We smash the database with a billion algo structures (binary, neural net, linear, n-dimensional state-dependent, CART, HORSE...) and a gazillion parameters involving a truck load of expensive data.  Amazingly, millions of combinations produce very favourable outcomes. That's because we stopped the run at 5 million outstanding backtests.  We could keep going and produce a billion. We can vary universes on a time dependent basis. Some profitable signals include things like: if a stock is up by 10% on Tuesday for the last three weeks in combination with no more than one being negative, then double down on Thursday following the Grand Final.  It works even better if I add a Fourier transform to vary the signal strength..... really?  It's even better if we adopt formulae designed to model the breeding patterns for rabbits and adapt that to market support levels. Do this and some will win. Some of those will publish, some of those books will sell. Yet, strangely, a lot of them don't seem to translate to the promises made when adopted by the masses in search of wealth which is rightfully theirs.  The profitable ones are, strangely, a similar proportion to that which would be expected from chance alone.

In the same dataset, we find that stocks which have been re-rated by the analyst community generally move in the direction of re-rating.  Wow...no Fourier, Tuesday, Grand Final...  So, if the people who influence the market via their opinions or because their opinions give reasons for HNW advisors to call their clients with a story actually impact the market via observable channels (you can see that the analyst changes his mind, calls clients and hear them changing their minds and they trade etc..) you have a visible cause and effect which is observable and testable (what if only two analysts move of ten, and one only by a bit?).  I might give a little more likelihood to this signal actually being worth something relative to our friend above.  I would give it a lot of validity if I knew I could be amongst the very first who got a hint that such a change of mind were forthcoming if these were generally followed with an actual change which was broadly announced.

People who do this for real, in the detail, tend not to publish their formulae until after the idea has been exhausted or to put off others from the scent via a false publication. The book sales are just nothing compared to the value of the formulation.  If I did leak formulae still in use and published for kudos, for example, I would have been totally cratered and deservedly so if this were not authorized.

Still....it's not enough.  You also need a free set of steak knives. You've got to check that it can be implemented.  You might wish to verify it in other markets etc.  The stuff that goes on here is somewhat special and we are hitting walls on Monte Carlo definitions.

Then you produce some sort of decision rule to give exposure to it (along with strategies B through Z) and wrap it up in some decent risk management.

---

Roughly speaking, producing a backtest which works is basically very close to worthless.  Producing one that works for reasons you can see and whose transmission channels are understood and viable for ongoing value creation is worth somewhat more.  Getting value out of it is yet another challenge.  Not blowing up along the way to getting rich off these ideas is another one on top of that.

If a $399 professional edition pack from Amibroker was all it took to go from newbie to hero, why aren't yacht sales higher than they are and why are clearly super-smart guys like WYSIWYG and KTP here talking about their backtest experience and how it varies from the actual outcomes they experienced?

If it were as easy as producing a platform and coding up something which found a favourable set of stats... yet, it can be...if you are lucky.  So, I guess, it is reasonable to limit your endeavours to finding a backtest formula that works and just trading it for substantive profit...so long as you are also lucky. Do you feel lucky.. Duck?


----------



## fiftyeight (18 March 2015)

So let me join in on this game I have 5 secs.

I may regret entering this debate and I assume someone will make me regret referencing Wikipedia but I am on site and tired....

This is how I understand what is being debated.

So I have just bought a system that promises I can get rich in just 6 months, I decide to run some tests



> Monte Carlo methods vary, but tend to follow a particular pattern:
> 
> 1.Define a domain of possible inputs.
> 2.Generate inputs randomly from a probability distribution over the domain
> ...





1.I want to test my new fancy system and I want to see how it preformed over 6 month periods within the last 20 years. My system buys/sells and can hold for seconds to weeks and can be switched on at any time. We therefore have very large number of 6 month periods to test which would be impracticable 

2. So I generate some inputs randomly from a probability distribution over the domain

3. I run my new fancy get rich in 6 months System on the inputs from above

4. I realise I have been scammed.

Is this a Monte Carlo Test?

Maybe my system will give better results if I run it for a longer period?

We scale up, make our period say 10 years, still giving a larger number 10 year periods to test. Then to 19 year periods, still a larger number initial inputs but.....

We have now reduced our initial conditions to those present in year 1. We do not have inputs randomly generated from the entire domain, rather only those present in the first year.

So we are no longer running a true Monte Carlo Test but an approximation of one using restrictions on the initial conditions.....

That took much longer than 5 secs


----------



## fiftyeight (18 March 2015)

DeepState said:


> If it were as easy as producing a platform and coding up something which found a favourable set of stats... yet, it can be...if you are lucky.  So, I guess, it is reasonable to limit your endeavours to finding a backtest formula that works and just trading it for substantive profit...so long as you are also lucky. Do you feel lucky.. Duck?




You are hard to keep up with DS

To make things short and sweet are you saying that Techs system success is *ONLY* due to luck?


----------



## AlterEgo (19 March 2015)

DeepState said:


> So having a platform and getting some backtest that showed, you know: low drawdown, high ratios etc. and trading it is sufficient?  Apparently....that's all there is to it.  My bad!




So you think it's not possible? I beg to differ. See attached plot of the profit of the system I developed in Amibroker. This is my real world trades with real money, NOT a backtest. There is some leverage involved in achieving this, however the real world results have been very consistent and in line with what I was getting in backtesting.


----------



## AlterEgo (19 March 2015)

DeepState said:


> Here's an example to just give this a tad of flavor.
> 
> We smash the database with a billion algo structures (binary, neural net, linear, n-dimensional state-dependent, CART, HORSE...) and a gazillion parameters involving a truck load of expensive data.  Amazingly, millions of combinations produce very favourable outcomes. That's because we stopped the run at 5 million outstanding backtests.  We could keep going and produce a billion. We can vary universes on a time dependent basis. Some profitable signals include things like: if a stock is up by 10% on Tuesday for the last three weeks in combination with no more than one being negative, then double down on Thursday following the Grand Final.  It works even better if I add a Fourier transform to vary the signal strength..... really?  It's even better if we adopt formulae designed to model the breeding patterns for rabbits and adapt that to market support levels. Do this and some will win. Some of those will publish, some of those books will sell. Yet, strangely, a lot of them don't seem to translate to the promises made when adopted by the masses in search of wealth which is rightfully theirs.  The profitable ones are, strangely, a similar proportion to that which would be expected from chance alone.




No, you don't do anything of the sort. That is curve fitting.


----------



## DeepState (19 March 2015)

AlterEgo said:


> So you think it's not possible? I beg to differ. See attached plot of the profit of the system I developed in Amibroker. This is my real world trades with real money, NOT a backtest. There is some leverage involved in achieving this, however the real world results have been very consistent and in line with what I was getting in backtesting.




I think you'll find that I did not say it was impossible.  In fact, my implications were that it was most defininitely possible.  Much as it is possible for a flipped coin to deliver profit...as an example of how pure luck can lead to profit. It would be very very hard to argue that a flipped coin could not develop a profit under any circumstance. I am, most certainly, not making such a statement.

Congratulations on developing a profit.


----------



## DeepState (19 March 2015)

AlterEgo said:


> No, you don't do anything of the sort. That is curve fitting.




Well, I don't. From your posts, it's pretty clear you don't either. But, given that warnings about over-fitting exist, it seems rather likely that the activity takes place somewhere and in some form.  From some of what is posted or in discussions I have otherwise had elsewhere, it is pretty clear that pattern search is seen to be largely sufficient by at least some....


----------



## DeepState (19 March 2015)

fiftyeight said:


> You are hard to keep up with DS
> 
> To make things short and sweet are you saying that Techs system success is *ONLY* due to luck?




Don't know.


----------



## DeepState (19 March 2015)

fiftyeight said:


> I may regret entering this debate....
> 
> 2. So I generate some inputs randomly from a probability distribution over the domain
> 
> Is this a Monte Carlo Test?




I am getting that feeling now... 

Pls expand on 2. Exactly how is this done?

From this, I should be able to ascertain whether this is Monte Carlo or something else.


----------



## tech/a (19 March 2015)

> So, I guess, it is reasonable to limit your endeavours to finding a backtest formula that works and just trading it for substantive profit...so long as you are also lucky. Do you feel lucky.. Duck?




Yeh. Made a lot of errors in life but made a lot of lucky calls as well.
Haven't needed a Doctorate in Economics
Nor have I pondered a balance sheet.

Done a lot of things that my more educated friends thought at the time
was/are lunacy. Strangely when invited to the Caribbean none can join us?

I'm convinced that education is not a pre requisite to financial security.
Can even be a hindrance.

Luck comes and goes.
Always has and always will.
I agree that my very biggest and very best gains in everything
Business
Property and
Trading 
Have been Right place/Right time/Right decision---Luck if you like.
DAX trading lately has been amazing luck.

To all those who haven't a degree and not likely to get one.

"Limit Risk and take advantage of quantified opportunity."


----------



## DeepState (19 March 2015)

tech/a said:


> Yeh. Made a lot of errors in life but made a lot of lucky calls as well.
> Haven't needed a Doctorate in Economics
> Nor have I pondered a balance sheet.
> 
> ...




Cheers to that Tech/A. Many roads to Rome/Barbados. Might as well take the interesting route.  If we don't make it, at least the journey was interesting.

I also think your advice to the not formally tertiary educated has wider scope....


----------

