Newbie... portfolio backtesting results

[tech/a]

No, it doesn't. You can have many different portfolios between runs. There are often several trade signals on a given day, but due to limited system equity your system may only be able to take say 1 or 2 of those signals. You can change the portfolio by making the system select a different set of valid signals each run.

No I think he's right.
Think of your universe as the ASX 100
You can Monte Carlo your brains out and your
still only getting a result for THAT data set.

Now go trade it on the FTSE 100
or all stocks OUTSIDE the ASX 100--see how that goes.

Haven't finished but my 2 mins is up

Then there is the issue of are in the dataset now but 10 yrs ago where were they.

 

[DeepState]

No, it doesn't. You can have many different portfolios between runs. There are often several trade signals on a given day, but due to limited system equity your system may only be able to take say 1 or 2 of those signals. You can change the portfolio by making the system select a different set of valid signals each run.

When you alter your decision rules ('making the system select a different...') as per above, you are creating a variant algo with different deterministic parameters. Keep these new parameters and run this a billions times and you get the same outcome each time for this fixed set of parameters. You have altered the algo. What you are doing is varying decision parameters and calling it Monte Carlo. It isn't. It is an examination of outcomes based on varying input parameters.

For completeness: If, for some perverse reason, you have some sort of stochastic rule to select different trades rules at different periods despite that fact the experience was identical to that point, (eg. "we'll use MA[Min(Bi(10,3),24),25] last period then regime change matrix potentially to Bollinger[2,U(5,10)] this coming period"), you will get a distribution of outcomes for a fixed dataset. If this is how you create a decision rule whose parameters vary in mid-flight despite a fixed dataset, there may be a few more issues than definitions about Monte Carlo simulations that need to be addressed first.

In the world of finance, Monte Carlo simulations are not created by perturbing the decision rule inputs on a fixed dataset (incl the variations that Tech was talking about). Doing so is creating a parameter hyper-surface/plane.

Monte-Carlo simulations are conducted routinely to assess and value the price of derivatives or to estimate the outcome of some decision rule where the historical outcomes of the variable in question is used to inform some stochastic process. For example, calculation of VaR can be calculated in several ways. One of them is to resample (with or without replacement) from some block of history (where vertical and/or horizontal integrity might or might not be preserved). Same calculation/decision rule, different datasets get produced via resampling or other stochastic process, voila - distributional outcome.

If you use a single set of parameters and design spec for your algo but produce a dataset that is resampled or otherwise abstracted in some fashion via some estimated stochastic model whose innovations are genuinely random, that will yield a Monte Carlo simulation.

These two concepts are materially different and should not be interchanged. They serve very different purposes and are used in different circumstances. To make things hard, the two can be run concurrently.

--

Monte Carlo has a casino which is where this concept finds its roots. They play roulette there. A Monte-Carlo simulation helps determine what the distribution of wins and losses (alternatively - reds/blacks/green) might be for the game of roulette. The game of roulette does not change its parameters. The same sorts of things can be determined for all casino games whose rules do not change (yea yea with card counters and all that). Distributional outcomes are obtained based on random perturbations with no change in design parameters. Unless you are saying that the decision rule for trading has large random components for strategy selection, you are arguing that changing a set of deterministic decision parameters produces a Monte Carlo analysis. It does not. There is a world of difference between these concepts.

 

[DeepState]

Haven't finished but my 2 mins is up

you funny....
(5 sec)

 

[tech/a]

When you alter your decision rules ('making the system select a different...') as per above, you are creating a variant algo with different deterministic parameters. Keep these new parameters and run this a billions times and you get the same outcome each time for this fixed set of parameters. You have altered the algo. What you are doing is varying decision parameters and calling it Monte Carlo. It isn't. It is an examination of outcomes based on varying input parameters

That's not what he's doing or saying.

 

[DeepState]

That's not what he's doing or saying.

Then what is he saying or doing if not that?

 

[Habakkuk]

Then what is he saying or doing if not that?

The technique referred to as Monte Carlo by tech/a and Alter Ego is a simple trade-substitution exercise.
There are no parameters as such being changed and no parameter hyper-surface.
It's simply the path-dependence of a system that signals multiple (buy)trades and insufficient money to take all of them.
At least that's what I understand. Whether it's valid or useful is another matter.

 

[DeepState]

The technique referred to as Monte Carlo by tech/a and Alter Ego is a simple trade-substitution exercise.
There are no parameters as such being changed and no parameter hyper-surface.
It's simply the path-dependence of a system that signals multiple (buy)trades and insufficient money to take all of them.
At least that's what I understand. Whether it's valid or useful is another matter.

For a fixed path which is historical market realisation of any type that is factual, what exactly changes to produce different portfolio outcomes between runs if not some change to a decison rule parameter?

 

[AlterEgo]

For a fixed path which is historical market realisation of any type that is factual, what exactly changes to produce different portfolio outcomes between runs if not some change to a decison rule parameter?

What changes is which trade of the many valid signals are taken on a given day. Say you get 10 trade signals on a given day, but have funds for 2. First run may take trades A & B, next run may take trades C & D, etc. Then as C & D may have different trade durations than A & B, this alters the trading equity available in following days, so following trades will be different.

 

[AlterEgo]

The technique referred to as Monte Carlo by tech/a and Alter Ego is a simple trade-substitution exercise.

I'm not calling what I do Monte Carlo. But it does show a different path through the same data using the same rules, so gives a different equity curve, different stats, etc. So gives a better picture of the system performance than just looking a 1 of the possible paths.

And then you'd want to check it on a different data set as well (different time period or different market) to see if it still works. And then if all goes well, test it in the real world.

 

[DeepState]

What changes is which trade of the many valid signals are taken on a given day. Say you get 10 trade signals on a given day, but have funds for 2. First run may take trades A & B, next run may take trades C & D, etc. Then as C & D may have different trade durations than A & B, this alters the trading equity available in following days, so following trades will be different.

Thanks for the expansion. I just want to make sure that I understand your perspective given the above....

For a given set-up in decision making and set of history (ie. fixed datablock, starting time, assets/equity etc)... if you got the chance to run the process twice without memory, do you end up doing the same thing in each run through of history/backtest (ie. 'First run may take trades A & B, next run may take trades C & D, etc. Then as C & D may have different trade durations than A & B, this alters the trading equity available in following days, so following trades will be different.')? If so, you end up producing the same outcome for each full backtest run. Yes, the actual strategy mix used at each point can change, but the outcome for the exact set-up is identical each time you do it. Once or a billion times. You will use A&B, then C&D etc. each run. You will end up with exactly the same P&L for each run.

Yes, each bit of the underlying strategy (A, B, C, D etc.) run in pure isolation will yield its own P&L which will be fixed if nothing else is changed. These can be helpful in deciding how to blend them.

If not, and the strategy deployment changes from the first backtest, why would two runs with identical starting conditions and exactly the same set of signal availability, equity etc. lead to a different outcome? That is, is there some magic which changes your allocation from A&B in one run for the same period in one backtest to A&C in a second? B&D in a third backtest for the same period? Introducing some randomness (hence changing the path of strategy deployment for different backtests despite otherwise identical starting conditions) will produce different P&L for each backtest run and will produce a Monte Carlo simulation outcome.

The process of determining which strategies to outlay given available equity at each decision point is part of an overall decision rule. Just because you can change the sub-strategies deployed between periods doesn't produce a distribution of outcomes if the decision to deploy them remains fixed. It is one decision rule with several bits to it.

----

If, by saying,

I'm not calling what I do Monte Carlo.

...I am not to equate your approach as Monte Carlo after all. If so, then we are on the same page. In disagreeing with what constitutes Monte Carlo and then offering up what you do in the same paragraph as an explanation why in a prior post, it implied (to me) that your process was believed to be Monte Carlo (which it is if you randomly select sub-strategies or otherwise change them on the fly in some random way. Not otherwise).

In the end, what matters is that tests and outcomes are understood and used properly.

But it does show a different path through the same data using the same rules, so gives a different equity curve, different stats, etc. So gives a better picture of the system performance than just looking a 1 of the possible paths.

And then you'd want to check it on a different data set as well (different time period or different market) to see if it still works. And then if all goes well, test it in the real world

I get the sense you do. Newbies take note.

 

[tech/a]

Then what is he saying or doing if not that?

He's not altering the algo.
Hes running (I presume) the same algo on countless trading possibilities.

For Me the Monty Carlo sim I use if from Tradesim---David Samborsky
So to attempt to explain what the software does which I think is what A/E means.

Lets say I have 10 triggers on Day 1----20 on day 2 and 30 on day three etc etc.
10 x 19 x 29 combinations of trades for a portfolio opportunity----ad infinitum---with a big enough universe.
Then----some will come off a trade get stopped out or exited all at very different times so the new trades will all be infinitely different as each portfolio trades its own path.

From the combination of a huge number of portfolio's you'll get a scatter of the results within that universe over that period tested.

Whether that then computes to similar results on a different data set and indeed any data set and of course going forward is not known.
But to argue against is to suggest that there is no patterns or occurrences other than flukes in any data set that will result in a quantifiable edge that can be expressed as an algo.

Which gets back to what I think your argument is and that's the Efficient Market Hypothesis
Which means this thread is going to run for a few years of 2-60 min posts.

 

[Wysiwyg]

He's not altering the algo.
Hes running (I presume) the same algo on countless trading possibilities.

This introduction of randomness by adding an algorithm to the base code returns a best and worst case scenario. In actual trading one hopes the results lie somewhere within that range. Tomasz J. uses the "Random" algorithm in this formula ... http://traders.com/Documentation/FEEDbk_Docs/2009/03/TradersTips.html#TT3

// General-purpose MC part 
HowManyMCSteps = 20000; // adjust that to change the number of MC tests 

PositionScore = 100 * mtRandomA(); 
// that is single-line that causes random picking of signals 

Step = Optimize(“Step”, 1, 1, HowManyMCSteps , 1 ); 
// this is dummy variable, not used below 

// The trading system itself 
// ( you may enter your own system below instead of one from the article ) 

NumPos = 8; // maximum number of open positions 
SetOption(“MaxOpenPositions”, NumPos ); 
SetPositionSize( GetOption(“InitialEquity”) / NumPos, spsValue ); 
// as in the article - no compounding of profits 
// SetPositionSize( 100 / NumPos, spsPercentOfEquity ); 
// uncomment this for compounding profits 

// signals 
s = Signal( 12, 26, 9 ); 
m = MACD( 12, 26 ); 
Buy = Cross( s, m ); 
Sell = Cross( m, s ); 

SetTradeDelays( 1, 1, 1, 1 ); // trade with one bar delay on open price 
BuyPrice = Open; 
SellPrice = Open;

 
—Tomasz Janeczko
amibroker.com

 

[tech/a]

Which is not Monty Carlo analysis
David Samborsk of Tradesim has Monty Carlo analysis which
Is superior.

 

[Wysiwyg]

Which is not Monty Carlo analysis
David Samborsk of Tradesim has Monty Carlo analysis which
Is superior.

There has to be some code added to the base code so the program can "choose" different stocks from the candidates every pass. If it is not random then it is defined which is adding a new "condition" from which to choose from the candidates.

Amibroker has position score which ranks the candidates. This to me is the same as adding that position score condition to the buy conditions but the difference has never been explained to me.

 

[tech/a]

There has to be some code added to the base code so the program can "choose" different stocks from the candidates every pass. If it is not random then it is defined which is adding a new "condition" from which to choose from the candidates.

Amibroker has position score which ranks the candidates. This to me is the same as adding that position score condition to the buy conditions but the difference has never been explained to me.

I'll contact David to see if he minds me pasting stuff up of his under copywrite
It explains what he does ( his software ) much better than I can

 

[DeepState]

He's not altering the algo.
Hes running (I presume) the same algo on countless trading possibilities.

---

Lets say I have 10 triggers on Day 1----20 on day 2 and 30 on day three etc etc.
10 x 19 x 29 combinations of trades for a portfolio opportunity----ad infinitum---with a big enough universe.
Then----some will come off a trade get stopped out or exited all at very different times so the new trades will all be infinitely different as each portfolio trades its own path.

From the combination of a huge number of portfolio's you'll get a scatter of the results within that universe over that period tested.

---

Whether that then computes to similar results on a different data set and indeed any data set and of course going forward is not known.
But to argue against is to suggest that there is no patterns or occurrences other than flukes in any data set that will result in a quantifiable edge that can be expressed as an algo.

Which gets back to what I think your argument is and that's the Efficient Market Hypothesis
Which means this thread is going to run for a few years of 2-60 min posts.

Seems we both have a few minutes on our hands...

If he is not altering the algo then the decision rule is set in stone. If the sign says "turn right" and the algo says "if strSIGN=="RIGHT" then fn_TURN_RIGHT end; (in Matlab-ese anyway)" it will always turn right at that point. This algo will develop a stack of levels at which trades will take place, in aggregate, whether they are sourced from sub-strategy A, B...Z. So, you start at the start of Day 1 and have a stack of triggers (could be millions) and, at each node, there exists a pathway of further triggers (trillions, even), multiplying out as you say (to gazillions and beyond). However many you like, however long you like.

If you do this over the same dataset, you end up going down the same path, in the garden/thicket of forking paths, every time. The market pathway has already been pre-determined. It actually happened already. Back-testing the decision rules trigger the same action at the same points each time because the data wasn't changed. The algo is not changed (in a deterministic sense). So it cannot motivate different outcomes on the same data. There are an incredibly large number of possibilities that could have happened had the data turned out differently, but the data turned out one way. Having a gabagillion additional possible trading levels related to market situations that did not occur changes nothing on the outcome. History ('that universe over that period tested'). Repeat history a billion times, you get the same history. You do not get a distribution of outcomes. You get a distribution of one point (whatever it is, the rest are the same), realized as many times as you want to run this experiment until you give up thinking something different could happen with the next (re)run.

Whatever it is that scatters, it won't be the outcome of a backtest run on the same data with a fixed algo which reacts the same way each time.

If you run this on a different dataset, the outcome will likely differ (unless, in the freak, the different dataset turned out to be the same one as before). Run this new dataset through a gababillion times and it too will yield one outcome over and over again. Just a (likely) different figure to the first dataset.

With a fixed algo (even if it consists of a stack of sub-strategies that flick on and off or otherwise vary by some fixed rule), you will only get a scatter if your data sample changes. For a Monte Carlo analysis, this is how you get distributions for a particular, deterministic, decision rule (ie. not one that changes on the fly and would do different things even with the same dataset). Typically, this is achieved via resampling of some type or reducing the observed distributions into something compact (which is loaded with its own assumptions) and then recreating some form of alternative history. Doing this can produce as many alternative pathways as you want.

I am not saying that all patterns are useless. However, if you chuck a billion algo variants into random noise, you will find patterns. Do the same on market data and you get something virtually indistinguishable. Virtually. Real live trading outcomes, you've seen the mortality rate, suggest that however traders determine their decision rules, it turns out to produce an outcome indistinguishable to pure luck (or worse) less expenses for the most part (the distribution of profit and what it means to be 'above average' is a whole other topic).

I make the point that backtests based on time-series, price-only, longitudinal varieties (seemingly the most common discussed) have a very poor set-up to finding meaningful patterns - particularly if you think you can outgun the hard core algos out there who use a lot more info and can trade much faster. However, this type of setup has beautiful properties to produce patterns which appear meaningful and motivate trades.

If this should descend into a debate on EMH, you won't find me arguing for the affirmative.

 

[g-topa-g]

1. Past performance conducted by backtest, based on methods which are pretty much data-mining in the guise of pattern recognition, will have virtually no relationship with the future. The class of strategies most frequently written of (longitudinal, time-series oriented, entirely price-based) is a total minefield of disaster for this effect. Just dig around the cemetery (the threads) and you'll find the bodies. New plots are being dug by the day.

2. So you'll only backtest those companies which have survived and that you expect to survive. That reduces the extent of bias in the backtest that might be conducted? This might be true with perfect foresight. Without any foresight, it is obviously laden with bias. The basis under which you choose to forecast survivorship will introduce new bias. You will have a very hard time understanding the impact of this.

I guess you'll have to jump in live/sim and learn these things over time. Others can jump in and explain the cost of that education and offer ways to circumvent it. For everyone who is still here posting, there would be around twenty or more who started at the same time that didn't make it. Of course, you'll be able to figure out who - of those - will still be around in three years and only look at what they have done in the past....

I guess they key message here is that your data will be imperfect. Bias is very important to understand and avoid. The short-cut you are proposing to take is pretty material for all the thought of concepts of probability of survival etc. Doing so introduces all sorts of hindsight biases and style biases of the type which is hard to observe and very fraught ("oh, I would have known that this measure would lead to a better outcome 10 years ago.."). Backtests are just a tool, yet, with the type of stuff usually attempted, where rules are developed on the basis of backtests and applied algorithmically and in a time series fashion, the key thing you will hopefully learn is that the mortality rate is through the roof. There will be a subset who attempt to do so that will succeed. Of those, you will have to discern which were genuinely skillful or the result of type-2 error, survived despite negative odds. Your probability of doing so, whilst it may feel rock solid without experience, will be closer to pure chance.

There is a reason why brokers produce programmable platforms and signals for customer use....I'll let you guess why. Proceed with caution. Good on you, though, for posting questions.

Not "perfect" foresight just "probable" foresight.

I think I have my answers. Thanks for the feedback

 

[tech/a]

Not "perfect" foresight just "probable" foresight.

I think I have my answers. Thanks for the feedback

Oh damn we stop here!

 

[DeepState]

Not "perfect" foresight just "probable" foresight.

I think I have my answers. Thanks for the feedback

You're welcome.

I'm curious. If there are 1,000 stocks out there and the mortality rate is about 5% per year:

Q1: How much more likely do you think your approach will adjust that figure towards zero for an ASX type universe of your selection?

---

If with perfect foresight, this would make your forward looking universe 950 for a one year backtest (we run backtests only so long as we are confident that the company will survive for that long in the future)...leading to about half the original universe surviving for 14 years. So, of the stocks which survived the last 14 years (a fairly small number relative to stocks currently alive in reality) intact, only half could be used for a full backtest if the standard of:

...surely one could pick some companies that one could reasonably expect to survive in future for as far back as they are being backtested...

...is to be utilized.
----

Then, if you are a genius, and can narrow down the universe of death, by choosing a subset of the likely survivors that have a perfect chance of surviving in a given year. With 100% foresight you halve the survivors suitable for a full back-test per above. You choose exactly 950 stocks for Year 1 and they are the ones which expire.

Let's say that your skills are such that you can select a subset of just 900 stocks where the survival is 100% rather than the 95% for the full universe. In other words, there are 100 stocks which you felt their survival was marginal and booted them in Year 1. All deaths for that year are drawn from this universe (pretty amazing really...because it's just a little tougher than that - just ask S&P, Merrill Lynch, Goldman Sachs etc.).

Your universe will shrink down to just 23% of the start point content for a full backtest. If reality were any larger, and, if you will work with me at $5m per year plus upside, and I'll take you on tomorrow - heck, tonight.

---

Q2: How representative will those 230 stocks (or whatever fairly small number) be for the purposes of uncovering patterns which are meaningful for projection? What new bias have you introduced by slicing the universe into half relative to perfect insight?

---

Oh, but wait, we can test on bigger universes for shorter time periods....



This is nowhere near as straightforward as might be imagined. All the best with the adventure ahead.

 

[tech/a]

My ignorance and luck has served me well.
As I have clearly in correctly designed and traded systems profitably
Very profitably.

Your solution then is not systems developement and trading them.

What is?