Backtested Systems, Lies, Damned Lies, & Statistics

[bullmarket]

Hi again Snake

Bullmarket,

How is all of this determined?

How all those parameters are determined obviously depends solely on the individual and will also be determined by what fundamental and/or technical experience and skills they posess and finally their risk tolerances.

I have already stated what my personal investment objective is in other posts and in my signature below and I have my own investment plan to suit.

There are many different techniques one can use for things like entry/exit signals. I touched on some possibilities when I posted about the maths that drive the stochastic and MACD indicators in other posts last week.

There are also different techniques on setting position sizes some of which are discussed in "Trading with a Plan" by Compton and Kendall which I have often posted as a suggested. I'm sure Daryl Guppy also talks about position sizing in his books. The same sources apply for the various techniques for risk management.

Now, I could spend half an hour or more giving details on how I do things as an investor whose #1 priority is income but I suspect that would be a complete waste of time since I get the impression that the vast majority in here see themselves as traders. And also mrs bullmarket has already made it clear to me that I have spent far too much time in here today already

So if you want to discuss further I'll be happy to discuss further on Tuesday as I better spend the day with mrs bullmarket tomorrow

Have a nice evening and I'll pop back in on Tuesday.

cheers

bullmarket

 

[It's Snake Pliskin]

Glad you asked.

Optimisation has its place for singular entities,be it a stock,index,or future.
Provided that the optimised variable is varified at and after each trade.

To optimise a set of variables and then apply that optimisation to a portfolio or group is plain crazy as the variables within the group basically render the optimisation impractical.

Again of course running an optimised methodology over an entity will give a set of Numbers---your Blueprint---so while trading is within the blueprint all OK when outside STOP---.

Thats my veiw but of course but its up to the individual,what suits some may not suit others,its subject to various points of veiw and interpretation,it could be right and could be wrong,I knew a taxi driver who once optimised his rides,he's still a taxi driver today so it must have worked for him.
Can I phone a friend??

Thats should get them going.

So it`s horses for courses is it Tech?

The goal of optimisation is profitability. Therefore if one is to rely on the optimisation of a tested system, one needs to have confirmed their optimised system with data not used in backtesting and optimising the system. This data or sample can destroy an optimised system`s profitablility and render it unusable.

So the moral is: don`t be greedy in backtesting and leave some for later.

But what if future data which is actual trading, turns out to be different again and stuffs you right up? There is that chance.
:alien2:

 

[lesm]

Glad you asked.

Optimisation has its place for singular entities,be it a stock,index,or future.
Provided that the optimised variable is varified at and after each trade.

To optimise a set of variables and then apply that optimisation to a portfolio or group is plain crazy as the variables within the group basically render the optimisation impractical.

Again of course running an optimised methodology over an entity will give a set of Numbers---your Blueprint---so while trading is within the blueprint all OK when outside STOP---.

Thats should get them going.

Hi tech/a

Agree with your comments above.

Optimisation is one of those subject areas that can be argued ad infinitum.

Optimisation, when trading on a portfolio basis is a questionable practise, due to each stock having its own charactertics or be affected by different factors, whether they are in the same sector or in different sectors. At best all that you can really hope to achieve is some form of compromise or averaging, but never full/complete optimisation.

One other point tech/a mentions is related to what may be referred to as a 'system stop'.

This is an often overlooked part of trading plans. The main focus is usually with respect to 'stops' related to trades taken using the plan, but not taking into account that the plan itself may go through periods where it is not effective in particular market conditions,hence consideration should be given to a 'system/plan' stop.

 

[It's Snake Pliskin]

Hi again Snake

How all those parameters are determined obviously depends solely on the individual and will also be determined by what fundamental and/or technical experience and skills they posess and finally their risk tolerances.

I have already stated what my personal investment objective is in other posts and in my signature below and I have my own investment plan to suit.

There are many different techniques one can use for things like entry/exit signals. I touched on some possibilities when I posted about the maths that drive the stochastic and MACD indicators in other posts last week.

There are also different techniques on setting position sizes some of which are discussed in "Trading with a Plan" by Compton and Kendall which I have often posted as a suggested. I'm sure Daryl Guppy also talks about position sizing in his books. The same sources apply for the various techniques for risk management.

Now, I could spend half an hour or more giving details on how I do things as an investor whose #1 priority is income but I suspect that would be a complete waste of time since I get the impression that the vast majority in here see themselves as traders. And also mrs bullmarket has already made it clear to me that I have spent far too much time in here today already

So if you want to discuss further I'll be happy to discuss further on Tuesday as I better spend the day with mrs bullmarket tomorrow

Have a nice evening and I'll pop back in on Tuesday.

cheers

bullmarket

Thanks Bullmarket :sleeping:

Enjoy the time with the wife.:remybussi

Snake

 

[tech/a]

So it`s horses for courses is it Tech?

The goal of optimisation is profitability. Therefore if one is to rely on the optimisation of a tested system, one needs to have confirmed their optimised system with data not used in backtesting and optimising the system. This data or sample can destroy an optimised system`s profitablility and render it unusable.


:alien2:

Snake.
Id say the goal is more efficiency. I'm a bit lost as to why you would optimise variables on data not used in the backtesting.You would pretty well always get a different result. Optimisation is only accurate to NOW thats why I say re do it at every new entry assuming that each new entery comes after a close in a single entity,Future,index etc.

To ponder--the argument is that optimisation cannot be guaranteed and I agree particularly in portfolio trading. A selected variable is just that---the variables selected and used for T/T are nothing scientific they are just ones that were seen to be "logical" even when throwing in various different frequencies for those with variables the change to bottomline and Drawdowns Risk reward was/is negligible.
So my point is why are parameters in any method acceptable selected by "logic"?
Why are optimised parameters so unacceptable?

I'll answer that with they are simply entries and exits which in a succesful system/method provide a positive expectancy and a "Blue print" with which we can trade.
Are they the best possible---in a singular entity possibly----In a portfolio situation----No they are just numbers in a formula. Well thats what I have found.

 

[happytrader]

Read this article on paper trading
http://www.actionforex.com/articles...r_trading_can_pump_you_up,_but_it's_not_real/

:alien2:

Hi and thanks Snake for a great site.

I love those trading tips by Joe Ross especially his views on 'market opinions and 'final tips'

Cheers Happytrader

 

[stevo]

Duc
"I have yet to see any P values generated by the software in regards to this statistical value."

My stats is a little rusty, and certainly I will be eaten alive in this area but here goes.

What is the statistical hypothesis that you want to test? (Statistical hypothesis defined as a claim about a population that can be put to the test by drawing a random sample).

I could compare a sample of backtest results with the population of backtest results (probably determined by using Monte Carlo) for the same system to see if it falls within the confidence interval but that doesn't prove anything.

What is the null hypothesis: the profitability of mechanical system A is no different from random trading? I am not too sure that this approach gets us far, or even that the p value is of much value as a statistical test in this area.

One use for p-Value could be to compare real trading results with the backtested results for the same system to see if they are statistically significant.

stevo

Say not "I have found the truth," but rather, "I have found a truth."
Kahlil Gibran

 

[It's Snake Pliskin]

Hi and thanks Snake for a great site.

I love those trading tips by Joe Ross especially his views on 'market opinions and 'final tips'

Cheers Happytrader

No worries!
I was googling and come across it.
Happyreading Happytrader :bekloppt:

 

[It's Snake Pliskin]

Snake.
Id say the goal is more efficiency. I'm a bit lost as to why you would optimise variables on data not used in the backtesting.You would pretty well always get a different result. Optimisation is only accurate to NOW thats why I say re do it at every new entry assuming that each new entery comes after a close in a single entity,Future,index etc.

Tech,

I may not have been too clear with what I meant. Basically, the data used for optimising and developing the system is for that purpose only. I was alluding that a fresh sample should be used to test the results of the system on after development. I`m referring to an out sample as some may call it.

Yes, I agree with re-doing it.

Snake :aus:

 

[ducati916]

stevo

It would be nice to have a definitive text on system testing but any I have seen are sadly lacking. Most people don't have the skills, knowledge and perseverance to carry out comprehensive testing of any sort of trading strategies. Many strategies touted are untestable - something the seller of the method probably appreciates.

This would seem to be the case.
Certainly it would seem to be the case that the methodologies are not statistical at all, but relying on finding a chart pattern that has worked in the past and proceeding on the assumption that it may work in the future.

Fortunately we don't have to simulate trading data - we can use actual price and volume data and draw on thousands of stocks for our databases.

The first problem of course is that the time series for the data used is so short. TT I believe utilized 1996 as the earliest source of Price & Volume data.
This time frame is wholly inadequate, 10yrs, is just about, or a little over one business cycle, certainly within the ASX there has been no Bear market within the data, just the odd correction, and a secular Bull market.

How far back you would need to go is a moot point, but certainly to at least the mid 1960's, if not further. Of course there just is not the data available for entering into the software, or not cheap data anyway, and therefore 99% of users only use the bare minimum, and think that their results will be robust through the future.....................

How so?
If you have no data that pertains to the variety of Bear markets that exist, you cannot, and have not designed nor performed a thorough test, irrespective of the absence of statistical significance.

You are therefore in effect saying, I tested on conditions XYZ, and as long as XYZ remain, I should be ok, and thats fine, and I can accept that as a premise.

That however does not seem to be the case;

Actually Duc, If you/anyone has a positive expectancy model and they trade it within the numbers generated by that model then yes I would guarantee that you would profit. Provided of course that sufficient testing had taken place.I think that the 6 mths testing in the case of T/T has proven to be sufficient. Of course what is sufficient can be argued ad infinitum. I'm not the first or last to come to that conclusion.Stevo has a weekly method that is doing as well as T/T. As has Andrew.

Where Duc and I differ (In part at least) is the accuracy and sample testing of any method.
I dont believe every concievable condition or variable need be met. I think and so far have proven (The future may prove me wrong),That provided you trade the Numbers of the tested positive expectancy method that have been generated during testing,and cease to trade it if actual trading figures fall outside those generated in testing---you'll make a profit within the parameters returned in testing.

The danger lies within the "start period". That is when you first start trading the methodology, particularly if utilizing leverage. That the conditions change from XYZ, to ABC, can cause much mischief.

Mechanical systems have gone from being a useful tool, for supercharging a paper-trading testing period, into a cult-mantra that proposes a holy grail result, based on wholly inadequate data.

Which leads into the "expectancy calculation"
If we examine the portion (#wins/#losses) first.

jog on
d998

 

[ducati916]

stevo

What is the statistical hypothesis that you want to test? (Statistical hypothesis defined as a claim about a population that can be put to the test by drawing a random sample).

This in of itself is part and parcel of the problem. Only one set of criteria ever seem to be tested, and that constitutes the chart pattern under consideration, therefore, almost by definition, an incomplete study has been designed, using wholly inapplicable methodologies, that will have zero statistical significance.

Therefore, by examining what backtesting does extremely well, is probably the way forward, but with the caveat that it is not statistically based, or not in any readily admittable form

jog on
d998

 

[ducati916]

lesm

Duc,

Always enjoy your posts, including the variation from informative to argumentative and sometimes baiting of TAs.

At times, I wonder if you are attempting to over analyse/rationalise the approach to the development or the application of trading systems.

You seem to have difficulty with the term of 'positive expectancy' would the probability of a 'positive outcome' or 'positive return on investment' be more palatable?

I don't really have a problem with positive expectancy as a concept, more really with the way that it is calculated, inferring that there is a mathematical certainity somehow bestowed upon the result. But I'll go into greater detail.

jog on
d998

 

[stevo]

Duc
"The first problem of course is that the time series for the data used is so short."

Obviously this can be a problem for any sort of analysis. The amount of data available is "hindered" by the tendency to merger, acquisition and bankruptcy!

If you look at the Small Ords index on the ASX you will find a wonderful bear market from 2000 to 2003 with some beautiful spikes down - this index dropped by more than a third over the period. We also had the Asian Crisis and a couple of other of reasonably volatile occurances over the last 10 years. So it is possible to design some strategies to utilise what data we do have rather than dismissing the approach altogether.

We could debate data adequacy forever, fundamental or technical. Most fundamental data available is also sadly lacking. The least we can do is be aware of the weaknesses in whatever approach we adopt.

With regards to statistical significance;
If we need to use something like p-Values, or any other measure, to test the significance of a strategy then the strategy is not worth trading. A strategy should jump off the page; we should try to destroy the strategy by excluding the best trades and using the worst entry/exit points. But, even after our best efforts it should still jump off the page. We are not looking for small differences.

There is quite a bit of research on traders and trading - have a look at the current research section at http://faculty.haas.berkeley.edu/odean/index.html for starters. Make sure that you run your mouse over Odean's picture.

stevo

 

[ducati916]

stevo

Obviously this can be a problem for any sort of analysis. The amount of data available is "hindered" by the tendency to merger, acquisition and bankruptcy!

Recapitalizations [Stock buybacks, Stock Splits, Stock reverse splits, Conversions (Preferred, Bonds, Options)] will change the Price & Volume data significantly.

If you look at the Small Ords index on the ASX you will find a wonderful bear market from 2000 to 2003 with some beautiful spikes down - this index dropped by more than a third over the period. We also had the Asian Crisis and a couple of other of reasonably volatile occurances over the last 10 years. So it is possible to design some strategies to utilise what data we do have rather than dismissing the approach altogether.

2000 - 2003 is not really a secular bear market, and will therefore present a very significant bias in any methodology that looks at longer term positions and results.

30% only just qualifies as a bear market.
Until you hit an 80%+ decline I wouldn't want to place too much emphasis on the results that have been generated on backtesting, hell I had a 20% drop overnight last week in a position, volatility of only a 30% magnitude is just not looking at market reality.

We could debate data adequacy forever, fundamental or technical. Most fundamental data available is also sadly lacking. The least we can do is be aware of the weaknesses in whatever approach we adopt.

Which was really the underlying point of the thread.
So many have seemingly swallowed the hook of expectancy, that I thought it was worth a look.

Nice link, right up my alley!!
jog on
d998

 

[professor_frink]

stevo



2000 - 2003 is not really a secular bear market, and will therefore present a very significant bias in any methodology that looks at longer term positions and results.

30% only just qualifies as a bear market.
Until you hit an 80%+ decline I wouldn't want to place too much emphasis on the results that have been generated on backtesting, hell I had a 20% drop overnight last week in a position, volatility of only a 30% magnitude is just not looking at market reality.

ducati, how well will did you go in the last secular bear market? For me(being 25), I've never been alive to try and trade one, so I can't comment on how I'd go. You sound like you've traded one before, so I'd be curious to hear how much more difficult it is to be profitable in a period like that.

 

[stevo]

Duc
As you state yourself biases are present regardless of the method.

I quite like the paper on day traders in Taiwan - to quote from the abstract;

"Moreover, in the typical six month period, more than eight out of ten day traders lose money. Despite these bleak findings, there is strong evidence of persistent ability for a relatively small group of day traders. Traders with strong past performance continue to earn strong returns."

So whilst the majority of day traders lose there is evidence that a very small minority consistently win. Averages tell a very average picture, but some traders appear to have an edge. Unfortunately we don't know what methods the exceptional day traders were using - and I doubt if they are going to tell us.

The last paragraph is great;
"Our analysis makes clear the need for comprehensive risk disclosure. Prospective day traders should be apprised of their likelihood of success: only two out of ten make money; fewer do so consistently."

stevo

Quotes from the paper;
Do Individual Day Traders Make Money? Evidence from Taiwan with Brad Barber, Yi-Tsung Lee and Yu-Jane Liu.

 

[kaveman]

I ask myself this question whenever a thread gets really long when all it is is someone wanting to start an argument. Why?
I guess being argumentative and confrontational is still a popular method of entertainment.

 

[tech/a]

kave there is some great stuff here and I dont think that it is arguementative other than the Professor looking for a bit of a stouch.

Duc.
Why is it necessary for a method designed to trade long/long trends be expected to perform as well in a bear market as a bull market?

Why is it that people like yourself seem to require a method designed to do only one thing---need to be "robust" in that it should trade ALL markets at ALL times?

From what youve written due to data problems both Technical and Fundamental your basically saying that its not possible to trade a plan giving meaningful numbers.

I dont agree my numbers have been meaningful for 4 yrs others who started 2 yrs ago have had them for 2 yrs and those that start today will have them meaningful "In their" method for X period.

Just like the beginning of ANY trade there needs to be a BEGINNING to every traded trading plan.

 

[mit]

With regards to statistical significance;
If we need to use something like p-Values, or any other measure, to test the significance of a strategy then the strategy is not worth trading. A strategy should jump off the page; we should try to destroy the strategy by excluding the best trades and using the worst entry/exit points. But, even after our best efforts it should still jump off the page. We are not looking for small differences.

stevo

Could have agreed more. All my successful systems have been like this. Maybe only one or two out of twenty or thirty I'll seriously look at each year will be like this. You tweak the parameters a bit but the system will be tradable across a wide range of values.

I think that Duc is inventing problems that aren't significant. True there are dividends and mergers and takeovers but still all of the systems I have run over the years have been pretty true to the original backtesting for expectancy.

As for the limited period of data. I agree that this is an issue. There has been no real bear market for the last couple of decades. However, I got daily Dow data back to the 1930s and had a look at what we can expect. It is true that a bear market can move down a significant percentage but what I found is that the actual down movement is relatively quick. From a couple of months in 1987 to at most a couple of years. Most seem to be less than 12 months. After this the market moves sideways. However, there is enough movement up and down that a short term swing trader could make money. Certainly any person who made money 2000-2003 would make money in the sideway bits.

So the trick is to survive the down bit with most of your money intact. I think even a longer term system should manage this as buy signals should dry up. Some people will make money playing the short side.

So I am pretty optimistic about my systems as well as being able to make money even during a secular bear market. The worst will be waiting for buy signals during the down bit.

MIT

 

[stevo]

Mit
I tend to design longer term systems with some measure of the general market - when the market is weak the systems stop taking on buy signals. A sudden drop is a lot harder to handle than a drop that gives some warning though.

Just for interest I developed a random entry and exit strategy and compared it with a simple weekly MACD on the current All Ords stocks for the last 7 years (back to 1999). If there is any validity in the MACD approach we should get a result significantly better than the random entry and exit approach. Chart below says it all. Compared to the random system the MACD jumps off the page - forget the stats.

I am not suggesting that a simple MACD system is worth trading, just that it is certainly better than randomly buying and selling - certainly don't need a p-Value to tell us that.

stevo