This is a mobile optimized page that loads fast, if you want to load the real page, click this text.

Backtested Systems, Lies, Damned Lies, & Statistics

Jump to new
Hi again Snake

Snake Pliskin said:
Bullmarket,

How is all of this determined?
Click to expand...


How all those parameters are determined obviously depends solely on the individual and will also be determined by what fundamental and/or technical experience and skills they posess and finally their risk tolerances.

I have already stated what my personal investment objective is in other posts and in my signature below and I have my own investment plan to suit.

There are many different techniques one can use for things like entry/exit signals. I touched on some possibilities when I posted about the maths that drive the stochastic and MACD indicators in other posts last week.

There are also different techniques on setting position sizes some of which are discussed in "Trading with a Plan" by Compton and Kendall which I have often posted as a suggested. I'm sure Daryl Guppy also talks about position sizing in his books. The same sources apply for the various techniques for risk management.

Now, I could spend half an hour or more giving details on how I do things as an investor whose #1 priority is income but I suspect that would be a complete waste of time since I get the impression that the vast majority in here see themselves as traders. And also mrs bullmarket has already made it clear to me that I have spent far too much time in here today already

So if you want to discuss further I'll be happy to discuss further on Tuesday as I better spend the day with mrs bullmarket tomorrow

Have a nice evening and I'll pop back in on Tuesday.

cheers

bullmarket
 

So it`s horses for courses is it Tech?

The goal of optimisation is profitability. Therefore if one is to rely on the optimisation of a tested system, one needs to have confirmed their optimised system with data not used in backtesting and optimising the system. This data or sample can destroy an optimised system`s profitablility and render it unusable.

So the moral is: don`t be greedy in backtesting and leave some for later.

But what if future data which is actual trading, turns out to be different again and stuffs you right up? There is that chance.
:alien2:
 

Hi tech/a

Agree with your comments above.

Optimisation is one of those subject areas that can be argued ad infinitum.

Optimisation, when trading on a portfolio basis is a questionable practise, due to each stock having its own charactertics or be affected by different factors, whether they are in the same sector or in different sectors. At best all that you can really hope to achieve is some form of compromise or averaging, but never full/complete optimisation.

One other point tech/a mentions is related to what may be referred to as a 'system stop'.

This is an often overlooked part of trading plans. The main focus is usually with respect to 'stops' related to trades taken using the plan, but not taking into account that the plan itself may go through periods where it is not effective in particular market conditions,hence consideration should be given to a 'system/plan' stop.
 

Thanks Bullmarket :sleeping:

Enjoy the time with the wife.:remybussi

Snake
 

Snake.
Id say the goal is more efficiency. I'm a bit lost as to why you would optimise variables on data not used in the backtesting.You would pretty well always get a different result. Optimisation is only accurate to NOW thats why I say re do it at every new entry assuming that each new entery comes after a close in a single entity,Future,index etc.

To ponder--the argument is that optimisation cannot be guaranteed and I agree particularly in portfolio trading. A selected variable is just that---the variables selected and used for T/T are nothing scientific they are just ones that were seen to be "logical" even when throwing in various different frequencies for those with variables the change to bottomline and Drawdowns Risk reward was/is negligible.
So my point is why are parameters in any method acceptable selected by "logic"?
Why are optimised parameters so unacceptable?

I'll answer that with they are simply entries and exits which in a succesful system/method provide a positive expectancy and a "Blue print" with which we can trade.
Are they the best possible---in a singular entity possibly----In a portfolio situation----No they are just numbers in a formula. Well thats what I have found.
 
Duc
"I have yet to see any P values generated by the software in regards to this statistical value."

My stats is a little rusty, and certainly I will be eaten alive in this area but here goes.

What is the statistical hypothesis that you want to test? (Statistical hypothesis defined as a claim about a population that can be put to the test by drawing a random sample).

I could compare a sample of backtest results with the population of backtest results (probably determined by using Monte Carlo) for the same system to see if it falls within the confidence interval but that doesn't prove anything.

What is the null hypothesis: the profitability of mechanical system A is no different from random trading? I am not too sure that this approach gets us far, or even that the p value is of much value as a statistical test in this area.

One use for p-Value could be to compare real trading results with the backtested results for the same system to see if they are statistically significant.

stevo

Say not "I have found the truth," but rather, "I have found a truth."
Kahlil Gibran
 
happytrader said:
Hi and thanks Snake for a great site.

I love those trading tips by Joe Ross especially his views on 'market opinions and 'final tips'

Cheers Happytrader
Click to expand...

No worries!
I was googling and come across it.
Happyreading Happytrader :bekloppt:
 

Tech,

I may not have been too clear with what I meant. Basically, the data used for optimising and developing the system is for that purpose only. I was alluding that a fresh sample should be used to test the results of the system on after development. I`m referring to an out sample as some may call it.

Yes, I agree with re-doing it.

Snake :aus:
 
stevo


This would seem to be the case.
Certainly it would seem to be the case that the methodologies are not statistical at all, but relying on finding a chart pattern that has worked in the past and proceeding on the assumption that it may work in the future.

Fortunately we don't have to simulate trading data - we can use actual price and volume data and draw on thousands of stocks for our databases.
Click to expand...

The first problem of course is that the time series for the data used is so short. TT I believe utilized 1996 as the earliest source of Price & Volume data.
This time frame is wholly inadequate, 10yrs, is just about, or a little over one business cycle, certainly within the ASX there has been no Bear market within the data, just the odd correction, and a secular Bull market.

How far back you would need to go is a moot point, but certainly to at least the mid 1960's, if not further. Of course there just is not the data available for entering into the software, or not cheap data anyway, and therefore 99% of users only use the bare minimum, and think that their results will be robust through the future.....................

How so?
If you have no data that pertains to the variety of Bear markets that exist, you cannot, and have not designed nor performed a thorough test, irrespective of the absence of statistical significance.

You are therefore in effect saying, I tested on conditions XYZ, and as long as XYZ remain, I should be ok, and thats fine, and I can accept that as a premise.

That however does not seem to be the case;



The danger lies within the "start period". That is when you first start trading the methodology, particularly if utilizing leverage. That the conditions change from XYZ, to ABC, can cause much mischief.

Mechanical systems have gone from being a useful tool, for supercharging a paper-trading testing period, into a cult-mantra that proposes a holy grail result, based on wholly inadequate data.

Which leads into the "expectancy calculation"
If we examine the portion (#wins/#losses) first.

jog on
d998
 
stevo

What is the statistical hypothesis that you want to test? (Statistical hypothesis defined as a claim about a population that can be put to the test by drawing a random sample).
Click to expand...

This in of itself is part and parcel of the problem. Only one set of criteria ever seem to be tested, and that constitutes the chart pattern under consideration, therefore, almost by definition, an incomplete study has been designed, using wholly inapplicable methodologies, that will have zero statistical significance.

Therefore, by examining what backtesting does extremely well, is probably the way forward, but with the caveat that it is not statistically based, or not in any readily admittable form

jog on
d998
 
lesm


I don't really have a problem with positive expectancy as a concept, more really with the way that it is calculated, inferring that there is a mathematical certainity somehow bestowed upon the result. But I'll go into greater detail.

jog on
d998
 
Duc
"The first problem of course is that the time series for the data used is so short."

Obviously this can be a problem for any sort of analysis. The amount of data available is "hindered" by the tendency to merger, acquisition and bankruptcy!

If you look at the Small Ords index on the ASX you will find a wonderful bear market from 2000 to 2003 with some beautiful spikes down - this index dropped by more than a third over the period. We also had the Asian Crisis and a couple of other of reasonably volatile occurances over the last 10 years. So it is possible to design some strategies to utilise what data we do have rather than dismissing the approach altogether.

We could debate data adequacy forever, fundamental or technical. Most fundamental data available is also sadly lacking. The least we can do is be aware of the weaknesses in whatever approach we adopt.

With regards to statistical significance;
If we need to use something like p-Values, or any other measure, to test the significance of a strategy then the strategy is not worth trading. A strategy should jump off the page; we should try to destroy the strategy by excluding the best trades and using the worst entry/exit points. But, even after our best efforts it should still jump off the page. We are not looking for small differences.

There is quite a bit of research on traders and trading - have a look at the current research section at http://faculty.haas.berkeley.edu/odean/index.html for starters. Make sure that you run your mouse over Odean's picture.

stevo
 
stevo

Obviously this can be a problem for any sort of analysis. The amount of data available is "hindered" by the tendency to merger, acquisition and bankruptcy!
Click to expand...

Recapitalizations [Stock buybacks, Stock Splits, Stock reverse splits, Conversions (Preferred, Bonds, Options)] will change the Price & Volume data significantly.


2000 - 2003 is not really a secular bear market, and will therefore present a very significant bias in any methodology that looks at longer term positions and results.

30% only just qualifies as a bear market.
Until you hit an 80%+ decline I wouldn't want to place too much emphasis on the results that have been generated on backtesting, hell I had a 20% drop overnight last week in a position, volatility of only a 30% magnitude is just not looking at market reality.

We could debate data adequacy forever, fundamental or technical. Most fundamental data available is also sadly lacking. The least we can do is be aware of the weaknesses in whatever approach we adopt.
Click to expand...

Which was really the underlying point of the thread.
So many have seemingly swallowed the hook of expectancy, that I thought it was worth a look.

Nice link, right up my alley!!
jog on
d998
 

ducati, how well will did you go in the last secular bear market? For me(being 25), I've never been alive to try and trade one, so I can't comment on how I'd go. You sound like you've traded one before, so I'd be curious to hear how much more difficult it is to be profitable in a period like that.
 
Duc
As you state yourself biases are present regardless of the method.

I quite like the paper on day traders in Taiwan - to quote from the abstract;

"Moreover, in the typical six month period, more than eight out of ten day traders lose money. Despite these bleak findings, there is strong evidence of persistent ability for a relatively small group of day traders. Traders with strong past performance continue to earn strong returns."

So whilst the majority of day traders lose there is evidence that a very small minority consistently win. Averages tell a very average picture, but some traders appear to have an edge. Unfortunately we don't know what methods the exceptional day traders were using - and I doubt if they are going to tell us.

The last paragraph is great;
"Our analysis makes clear the need for comprehensive risk disclosure. Prospective day traders should be apprised of their likelihood of success: only two out of ten make money; fewer do so consistently."

stevo

Quotes from the paper;
Do Individual Day Traders Make Money? Evidence from Taiwan with Brad Barber, Yi-Tsung Lee and Yu-Jane Liu.
 
I ask myself this question whenever a thread gets really long when all it is is someone wanting to start an argument. Why?
I guess being argumentative and confrontational is still a popular method of entertainment.
 
kave there is some great stuff here and I dont think that it is arguementative other than the Professor looking for a bit of a stouch.

Duc.
Why is it necessary for a method designed to trade long/long trends be expected to perform as well in a bear market as a bull market?

Why is it that people like yourself seem to require a method designed to do only one thing---need to be "robust" in that it should trade ALL markets at ALL times?

From what youve written due to data problems both Technical and Fundamental your basically saying that its not possible to trade a plan giving meaningful numbers.

I dont agree my numbers have been meaningful for 4 yrs others who started 2 yrs ago have had them for 2 yrs and those that start today will have them meaningful "In their" method for X period.

Just like the beginning of ANY trade there needs to be a BEGINNING to every traded trading plan.
 

Could have agreed more. All my successful systems have been like this. Maybe only one or two out of twenty or thirty I'll seriously look at each year will be like this. You tweak the parameters a bit but the system will be tradable across a wide range of values.

I think that Duc is inventing problems that aren't significant. True there are dividends and mergers and takeovers but still all of the systems I have run over the years have been pretty true to the original backtesting for expectancy.

As for the limited period of data. I agree that this is an issue. There has been no real bear market for the last couple of decades. However, I got daily Dow data back to the 1930s and had a look at what we can expect. It is true that a bear market can move down a significant percentage but what I found is that the actual down movement is relatively quick. From a couple of months in 1987 to at most a couple of years. Most seem to be less than 12 months. After this the market moves sideways. However, there is enough movement up and down that a short term swing trader could make money. Certainly any person who made money 2000-2003 would make money in the sideway bits.

So the trick is to survive the down bit with most of your money intact. I think even a longer term system should manage this as buy signals should dry up. Some people will make money playing the short side.

So I am pretty optimistic about my systems as well as being able to make money even during a secular bear market. The worst will be waiting for buy signals during the down bit.

MIT
 
Mit
I tend to design longer term systems with some measure of the general market - when the market is weak the systems stop taking on buy signals. A sudden drop is a lot harder to handle than a drop that gives some warning though.

Just for interest I developed a random entry and exit strategy and compared it with a simple weekly MACD on the current All Ords stocks for the last 7 years (back to 1999). If there is any validity in the MACD approach we should get a result significantly better than the random entry and exit approach. Chart below says it all. Compared to the random system the MACD jumps off the page - forget the stats.

I am not suggesting that a simple MACD system is worth trading, just that it is certainly better than randomly buying and selling - certainly don't need a p-Value to tell us that.

stevo
 

Attachments

  • MACD vs Random.gif
    35.4 KB · Views: 56
You must log in or register to reply here.

Similar threads

R
21 Lessons in Investing
Replies
6
Views
3K
divs4ever
R
The 7 Moats of Quality Companies
Replies
7
Views
7K
divs4ever
Ultimate Acronym List (UAL)
Replies
0
Views
5K
Dividend
F
AMT Model & Methodology
49 50 51
Replies
1K
Views
238K
Frank D
F
When FAILURE is simply NOT an option
Replies
17
Views
5K
wayneL
Menu
Log in
Register
Cookies are required to use this site. You must accept them to continue using the site. Learn more...