Newbie... portfolio backtesting results

tech/a said:

My ignorance and luck has served me well.
As I have clearly in correctly designed and traded systems profitably
Very profitably.

Your solution then is not systems developement and trading them.

What is?

Click to expand...

So having a platform and getting some backtest that showed, you know: low drawdown, high ratios etc. and trading it is sufficient? Apparently....that's all there is to it. My bad!

DeepState said:

So having a platform and getting some backtest that showed, you know: low drawdown, high ratios etc. and trading it is sufficient? Apparently....that's all there is to it. My bad!

Click to expand...

Well you have to trade it!

It has been in my case
obviously an outlier.

Still I'm interested in your alternative

tech/a said:

Well you have to trade it!

It has been in my case
obviously an outlier.

Still I'm interested in your alternative

Click to expand...

Well, in order to win, you have to play. So having to trade it is necessary for investment success. Of course, trading often doesn't lead to success.

The missing pieces are finding something in the patterns that has a shot of actually being sustainable and predictive, re-hashing that to produce something which can be implemented, risk managing and carefully implementing it. Just those minor details. Of the number of backtests which can be produced that show a good outcome, the proportion of which contains any decent predictive ability is vanishingly small.

Something which fails backtests can reasonably be included.

---

Here's an example to just give this a tad of flavor.

We smash the database with a billion algo structures (binary, neural net, linear, n-dimensional state-dependent, CART, HORSE...) and a gazillion parameters involving a truck load of expensive data. Amazingly, millions of combinations produce very favourable outcomes. That's because we stopped the run at 5 million outstanding backtests. We could keep going and produce a billion. We can vary universes on a time dependent basis. Some profitable signals include things like: if a stock is up by 10% on Tuesday for the last three weeks in combination with no more than one being negative, then double down on Thursday following the Grand Final. It works even better if I add a Fourier transform to vary the signal strength..... really? It's even better if we adopt formulae designed to model the breeding patterns for rabbits and adapt that to market support levels. Do this and some will win. Some of those will publish, some of those books will sell. Yet, strangely, a lot of them don't seem to translate to the promises made when adopted by the masses in search of wealth which is rightfully theirs. The profitable ones are, strangely, a similar proportion to that which would be expected from chance alone.

In the same dataset, we find that stocks which have been re-rated by the analyst community generally move in the direction of re-rating. Wow...no Fourier, Tuesday, Grand Final... So, if the people who influence the market via their opinions or because their opinions give reasons for HNW advisors to call their clients with a story actually impact the market via observable channels (you can see that the analyst changes his mind, calls clients and hear them changing their minds and they trade etc..) you have a visible cause and effect which is observable and testable (what if only two analysts move of ten, and one only by a bit?). I might give a little more likelihood to this signal actually being worth something relative to our friend above. I would give it a lot of validity if I knew I could be amongst the very first who got a hint that such a change of mind were forthcoming if these were generally followed with an actual change which was broadly announced.

People who do this for real, in the detail, tend not to publish their formulae until after the idea has been exhausted or to put off others from the scent via a false publication. The book sales are just nothing compared to the value of the formulation. If I did leak formulae still in use and published for kudos, for example, I would have been totally cratered and deservedly so if this were not authorized.

Still....it's not enough. You also need a free set of steak knives. You've got to check that it can be implemented. You might wish to verify it in other markets etc. The stuff that goes on here is somewhat special and we are hitting walls on Monte Carlo definitions.

Then you produce some sort of decision rule to give exposure to it (along with strategies B through Z) and wrap it up in some decent risk management.

---

Roughly speaking, producing a backtest which works is basically very close to worthless. Producing one that works for reasons you can see and whose transmission channels are understood and viable for ongoing value creation is worth somewhat more. Getting value out of it is yet another challenge. Not blowing up along the way to getting rich off these ideas is another one on top of that.

If a $399 professional edition pack from Amibroker was all it took to go from newbie to hero, why aren't yacht sales higher than they are and why are clearly super-smart guys like WYSIWYG and KTP here talking about their backtest experience and how it varies from the actual outcomes they experienced?

If it were as easy as producing a platform and coding up something which found a favourable set of stats... yet, it can be...if you are lucky. So, I guess, it is reasonable to limit your endeavours to finding a backtest formula that works and just trading it for substantive profit...so long as you are also lucky. Do you feel lucky.. Duck?

So let me join in on this game I have 5 secs.

I may regret entering this debate and I assume someone will make me regret referencing Wikipedia but I am on site and tired....

This is how I understand what is being debated.

So I have just bought a system that promises I can get rich in just 6 months, I decide to run some tests

Monte Carlo methods vary, but tend to follow a particular pattern:

1.Define a domain of possible inputs.
2.Generate inputs randomly from a probability distribution over the domain
3.Perform a deterministic computation on the inputs.
4.Aggregate the results.

Click to expand...

1.I want to test my new fancy system and I want to see how it preformed over 6 month periods within the last 20 years. My system buys/sells and can hold for seconds to weeks and can be switched on at any time. We therefore have very large number of 6 month periods to test which would be impracticable

2. So I generate some inputs randomly from a probability distribution over the domain

3. I run my new fancy get rich in 6 months System on the inputs from above

4. I realise I have been scammed.

Is this a Monte Carlo Test?

Maybe my system will give better results if I run it for a longer period?

We scale up, make our period say 10 years, still giving a larger number 10 year periods to test. Then to 19 year periods, still a larger number initial inputs but.....

We have now reduced our initial conditions to those present in year 1. We do not have inputs randomly generated from the entire domain, rather only those present in the first year.

So we are no longer running a true Monte Carlo Test but an approximation of one using restrictions on the initial conditions.....

That took much longer than 5 secs

DeepState said:

If it were as easy as producing a platform and coding up something which found a favourable set of stats... yet, it can be...if you are lucky. So, I guess, it is reasonable to limit your endeavours to finding a backtest formula that works and just trading it for substantive profit...so long as you are also lucky. Do you feel lucky.. Duck?

Click to expand...

You are hard to keep up with DS

To make things short and sweet are you saying that Techs system success is ONLY due to luck?

DeepState said:

So having a platform and getting some backtest that showed, you know: low drawdown, high ratios etc. and trading it is sufficient? Apparently....that's all there is to it. My bad!

Click to expand...

So you think it's not possible? I beg to differ. See attached plot of the profit of the system I developed in Amibroker. This is my real world trades with real money, NOT a backtest. There is some leverage involved in achieving this, however the real world results have been very consistent and in line with what I was getting in backtesting.

DeepState said:

Here's an example to just give this a tad of flavor.

We smash the database with a billion algo structures (binary, neural net, linear, n-dimensional state-dependent, CART, HORSE...) and a gazillion parameters involving a truck load of expensive data. Amazingly, millions of combinations produce very favourable outcomes. That's because we stopped the run at 5 million outstanding backtests. We could keep going and produce a billion. We can vary universes on a time dependent basis. Some profitable signals include things like: if a stock is up by 10% on Tuesday for the last three weeks in combination with no more than one being negative, then double down on Thursday following the Grand Final. It works even better if I add a Fourier transform to vary the signal strength..... really? It's even better if we adopt formulae designed to model the breeding patterns for rabbits and adapt that to market support levels. Do this and some will win. Some of those will publish, some of those books will sell. Yet, strangely, a lot of them don't seem to translate to the promises made when adopted by the masses in search of wealth which is rightfully theirs. The profitable ones are, strangely, a similar proportion to that which would be expected from chance alone.

Click to expand...

No, you don't do anything of the sort. That is curve fitting.

AlterEgo said:

So you think it's not possible? I beg to differ. See attached plot of the profit of the system I developed in Amibroker. This is my real world trades with real money, NOT a backtest. There is some leverage involved in achieving this, however the real world results have been very consistent and in line with what I was getting in backtesting.

Click to expand...

I think you'll find that I did not say it was impossible. In fact, my implications were that it was most defininitely possible. Much as it is possible for a flipped coin to deliver profit...as an example of how pure luck can lead to profit. It would be very very hard to argue that a flipped coin could not develop a profit under any circumstance. I am, most certainly, not making such a statement.

Congratulations on developing a profit.

AlterEgo said:

No, you don't do anything of the sort. That is curve fitting.

Click to expand...

Well, I don't. From your posts, it's pretty clear you don't either. But, given that warnings about over-fitting exist, it seems rather likely that the activity takes place somewhere and in some form. From some of what is posted or in discussions I have otherwise had elsewhere, it is pretty clear that pattern search is seen to be largely sufficient by at least some....

fiftyeight said:

You are hard to keep up with DS

To make things short and sweet are you saying that Techs system success is ONLY due to luck?

Click to expand...

Don't know.

fiftyeight said:

I may regret entering this debate....

2. So I generate some inputs randomly from a probability distribution over the domain

Is this a Monte Carlo Test?

Click to expand...

I am getting that feeling now...

Pls expand on 2. Exactly how is this done?

From this, I should be able to ascertain whether this is Monte Carlo or something else.

So, I guess, it is reasonable to limit your endeavours to finding a backtest formula that works and just trading it for substantive profit...so long as you are also lucky. Do you feel lucky.. Duck?

Click to expand...

Yeh. Made a lot of errors in life but made a lot of lucky calls as well.
Haven't needed a Doctorate in Economics
Nor have I pondered a balance sheet.

Done a lot of things that my more educated friends thought at the time
was/are lunacy. Strangely when invited to the Caribbean none can join us?

I'm convinced that education is not a pre requisite to financial security.
Can even be a hindrance.

Luck comes and goes.
Always has and always will.
I agree that my very biggest and very best gains in everything
Business
Property and
Trading
Have been Right place/Right time/Right decision---Luck if you like.
DAX trading lately has been amazing luck.

To all those who haven't a degree and not likely to get one.

"Limit Risk and take advantage of quantified opportunity."

tech/a said:

Yeh. Made a lot of errors in life but made a lot of lucky calls as well.
Haven't needed a Doctorate in Economics
Nor have I pondered a balance sheet.

Done a lot of things that my more educated friends thought at the time
was/are lunacy. Strangely when invited to the Caribbean none can join us?

I'm convinced that education is not a pre requisite to financial security.
Can even be a hindrance.

Luck comes and goes.
Always has and always will.
I agree that my very biggest and very best gains in everything
Business
Property and
Trading
Have been Right place/Right time/Right decision---Luck if you like.
DAX trading lately has been amazing luck.

To all those who haven't a degree and not likely to get one.

"Limit Risk and take advantage of quantified opportunity."

Click to expand...

Cheers to that Tech/A. Many roads to Rome/Barbados. Might as well take the interesting route. If we don't make it, at least the journey was interesting.

I also think your advice to the not formally tertiary educated has wider scope....