# Backtested Systems, Lies, Damned Lies, & Statistics



## ducati916 (9 March 2006)

It was this quote from *tech/a* that really seemed to highlight the belief that *mechanical systems* somehow present a trading plan, or methodology that based on statistics and mathematics, offered what is commonly referred to as *the edge* 



> Discretionary trading at best is Maverick style.
> There is much talk about plans and I often see plans that are like jelly ---constantly changing.
> 
> 
> Sound plans have statistics on entry exit,stops and position size variables so that when trading you KNOW wether what you are doing is within your results---reslts that determine profitability if you trade a particular plan.




We see that within this definition, *discretionary trading* has been portrayed as trading without a consistent plan, viz. trading on emotion.

Contrasted with the mechanical advantage, or statistically tested methodology that *results that determine profitability if you trade a particular plan. * 

Therefore it would really behoove anyone who is contemplating trading a mechanical methodology to closely examine some or all of the *assumptions that underlie the NUMBERS* 

To give us a benchmark, and a point of comparable standards, I shall utilize Medical Research gold standards as the benchmark.
In medical research statistical modeling is used extensively in the research of new drugs, diagnostic criteria, treatment methodologies and everything inbetween.

Why medicine?
Several reasons, and all related to the assessment of methodological quality.

1...It utilizes human beings, who though possessed of the same, or similar anatomy & physiology, have very different outcomes due to many *variables* 
2...Medical research is required to be predictive.
3...Medical research must be predictive for the majority, within clearly defined *confidence parametres* and examination of *causation*  to be accepted *(Correlation, regression, and causation)* 
4...The statistical techniques utilized are in many instances the same as utilized in these *backtested mechanical methodologies* 
5...Shoddy research can be hazardous to your health, or wealth in trading terms.
6...The increasing use of *Surrogate endpoints* 
7...Paired data, tails, and outliers
8...Study designs

jog on
d998


----------



## ducati916 (9 March 2006)

As with anything, starting at the beginning is probably the best place to make a start.

Was the study original?
Using my favorite mechanical system as a benchmark, the ubiquitous TechTrader, we can ask the question.

I personally think no, it wasn't, it had many similarities to the *TURTLE system* Now this is not a criticism (not yet anyway). Was the methodology (possibly undisclosed) examined in anyway for variables that were similar or the same within TechTrader?

From this starting point we can ask the following questions;

Is this study larger, continued for longer, or otherwise more substantial than previous studies (studies equating to trading methodologies)

Is the methodology of this study any more rigorous (in particular, does it cover any specific methodological criticisms of previous studies)

Will the numerical results of this study add significantly to a meta-analysis of previous studies?

Is the population studied different in any way (has the study examined capitalization structures, duration of listing, economic environment, market environment, inflation, interest, alternate investment classes etc)

I'll give tech/a some time to have a think about some of the preliminary questions before continuing any further.

jog on
d998


----------



## bullmarket (9 March 2006)

Hi ducati

Sorry but my attention span at this time of the morning is limited to posts no more than half  a screen in length   but I get the jist of what you're saying - I think   

But anyway, I'm a firm believer in having a written investment/trading plan which should be paper traded (especially for traders) before committing real funds.

*Imo a sound plan should include but not be limited to things like:*

1) Your objectives - long and/or short term

2) What markets and types of securities you will trade/invest in - shares, bonds, options, hybrids, fixed interest etc etc

3) How you will search/scan for investment/trading candidates - technical/fundamental or other criteria.

4) How you will determine entry/exit points - technical/fundamental criteria or a combination of both

5) How you will determine the position size for each investment/trade.

6) How you will manage risk - diversification, stop losses etc etc

*Now how complicated or simple someone makes their plan is solely up to them - but imo a key purpose of a plan is to remove emotion as much as possible from decision making by making the plan as comprehensive as required to suit ones objectives and circumstances.*

_I think it is very important to paper trade a plan for however long it takes (and that will vary according to each individual) until the person is satisfied the plan is generating the objectives/returns specified in the plan before committing funds._

cheers

bullmarket


----------



## ducati916 (9 March 2006)

*bullmarket* 

While I acknowledge the effort that you have taken in your answer, and certainly the information that you have provided is within its context materially correct, you have I'm afraid sort of missed the point of the thread.
This may in point of fact be my error.

The purpose of the thread is as follows;
Mechanically traded systems claim statistical evidence of their efficacy.
We know this from the methodology that is followed within the software that takes thousands of data inputs and spews back a statistically derived ratio.

These ratios form the basis of a trading plan, and delineated via further calculations, one example being the *expectancy calculation which is as (1+($win/$loss)*(#wins/#losses)-1)* 

These calculated ratios, and equation based outcomes infer that the trading methodology is in point of fact robust, logical, tested, mathematically determined, and therefore provides one of the most cliched trading phrases ever promulgated...........*an edge* 

The purpose of this thread is to examine the methodology behind *the numbers* to examine just how reliable, robust, accurate, or manipulated that they actually are. After all before you place all your cash, and leverage yourself higher than might be prudent, wouldn't you examine the assumptions and facts behind the software?

jog on
d998


----------



## bullmarket (9 March 2006)

ok no problem ducati 



			
				ducati916 said:
			
		

> *bullmarket*
> 
> 
> Mechanically traded systems claim statistical evidence of their efficacy.
> ...




re the above extract from your post the way I look at it is that mechanical systems, or whatever you want to call them, with their various ratios or whatever are largely just a way of monitoring your trading performance and keeping an eye on your trading performance to see if your plan's objectives are consistantly being met is critical for obvious reasons.  

Keeping a record in a spreadsheet or whatever of things like:

1) number of trades
2) number of winning trades
3) number of losing trades
4) $ profit per winning trade
5) $ loss per losing trade
6) time in both winning and losing trades etc etc etc

and calculating various ratios from the above and other data to then compare against preset bench-marks is an important part of portfolio and trading performance monitoring imo.

Now whether doing any or more of the above gives someone an 'edge' is simply up to one's own perception of what they call an edge so to speak.

But using the above as a base starting point to monitor trading performance should certainly help identify potential flaws in someone's trading strategy sooner rather than later when compared to someone whose record keeping and/or monitoring of their performance is not as thorough.

And finally, all of what I mention above imo should be included in someone's trading plan as part of their portfolio/trading performance monitoring process....and hence the importance imo of paper trading a plan for however long it takes prior to commiting funds so that your trading performance monitoring process can show you if your trading strategies are generating the returns you expected.

cheers

bullmarket 

ooops....mrs bullmarket is calling   I'll log in tonight if you would like to discuss further....cheers..


----------



## wayneL (9 March 2006)

I sense a crowd gathering ......  

Duc,

Interesting, and I'm probably going to find myself agreeing with you here. Watching with interest....


----------



## tech/a (9 March 2006)

I'm tied up with business issues today and will become "involved" when I can.

Suprising to some I'm with duc to a degree on some matters.


----------



## ducati916 (9 March 2006)

*bullmarket* 



> re the above extract from your post the way I look at it is that mechanical systems, or whatever you want to call them, with their various ratios or whatever are largely just a way of monitoring your trading performance and keeping an eye on your trading performance to see if your plan's objectives are consistantly being met is critical for obvious reasons.




Yes, this is true.
However you are running way ahead of the curve at the moment.
Implicit within your statement is that the trading plan built with a statistically backtested result is legitimate because it has been built mathematically or statistically, which assumes the statistical methodology is of a high methodological design.

Lets reverse this for a moment.
I as a new trader have an idea of a methodology that I wish to trade.
I formulate a trading plan.
I trade the plan.
I record the results.
Early returns are most unsatisfactory, I give up on this plan, and go to another plan and repeat.

This is your method, but you qualify it by saying paper-trade it, so that if its horrible, you don't lose too much cash.

tech/a and others are saying;
1...Statistically, you have no idea whether your failed plan, is in *reality no good, because the sample size is inconsequential, and it may just be in drawdown* 



> And finally, all of what I mention above imo should be included in someone's trading plan as part of their portfolio/trading performance monitoring process....and hence the importance imo of paper trading a plan for however long it takes prior to commiting funds so that your trading performance monitoring process can show you if your trading strategies are generating the returns you expected.




2...To test any idea by empirically, is so slow and drawn-out that by the time you have enough data to statistically draw meaningful conclusions, you'll be too old to care, to rich too care, or busted out living in a dustbin.

Therefore, enter the sandman..............
Computer software that runs monte-carlo simulations and all sorts of nifty timesaving bolt-ons, and hey presto, instant results.

This is where I come in....................
If you are going to base your trading plan on statistically generated results you must first *assess the methodological quality of the statistical models that you are building and running* if you do not, then how reliable are the trading systems that you will presumably place cash, and potentially leverage it, you could be in dustbin city far sooner than planned.

Therefore, utilizing medicine, and medical research, that utilize almost the same models, we can benchmark the methodological quality of the statistical tests, results, and conclusions.

To do so, we must start at the beginning, and build the concept of the model to be utilized, and look for potential faults, and potential remedies.



> Now whether doing any or more of the above gives someone an 'edge' is simply up to one's own perception of what they call an edge so to speak.




Which brings us directly into the psychological models that are being built within behavioral finance currently. However, that is somewhat a separate topic, and will muddy the water currently.

*WayneL* 



> Interesting, and I'm probably going to find myself agreeing with you here. Watching with interest....




Oh dear, that will never do.

*tech/a* 



> I'm tied up with business issues today and will become "involved" when I can.
> 
> Suprising to some I'm with duc to a degree on some matters.




Having an afternoon nap more likely.

jog on
d998


----------



## wayneL (9 March 2006)

ducati916 said:
			
		

> *WayneL*
> 
> Oh dear, that will never do.




Ahhhhhh Ahhhlright!

We can argue about the expectancy equation (which I maintain is a misnomer)... if you insist


----------



## ducati916 (9 March 2006)

Just so that the initial questions that require consideration don't get too buried, here is a quick reminder.



> Is this study larger, continued for longer, or otherwise more substantial than previous studies (studies equating to trading methodologies)
> 
> Is the methodology of this study any more rigorous (in particular, does it cover any specific methodological criticisms of previous studies)
> 
> ...




These are the questions that need to be considered prior to moving forward.
So all you mechanical backtesting advocates, don't be shy.

jog on
d998


----------



## ducati916 (9 March 2006)

*WayneL* 



> Ahhhhhh Ahhhlright!
> 
> We can argue about the expectancy equation (which I maintain is a misnomer)... if you insist




Indeed we can, but we are a long way from that point currently.
We still need to work our way through some of the assumptions that lie at the heart of the software designed to produce the statistics.

jog on
d998


----------



## It's Snake Pliskin (9 March 2006)

ducati916 said:
			
		

> Just so that the initial questions that require consideration don't get too buried, here is a quick reminder.
> 
> 
> 
> ...




Duc,

So curve fitting and overoptimisation should be included in the discussion then.

Curve fitting: matching a system to the data. To make a system look great for the period and data given.

Overoptimisation: fudging the parameters of a sytem until it works. Much like changing the moving averages until you find one that gives the best exit entry etc. So as a result it only works on the data tested on and is no guarantee of future success.

How does one overcome these biases in the search for *the edge* ?:nono:


----------



## bullmarket (9 March 2006)

Hi ducati




			
				ducati916 said:
			
		

> *bullmarket*
> 
> 
> 
> ...





If you paper trade your plan then you don't loose any cash at all so I'm not sure what you're getting at with _"This is your method, but you qualify it by saying paper-trade it, so that if its horrible, you don't lose too much cash."_

Also if the initial results are most unsatisfactory, as you put it, you don't have to give up on the plan at all as you suggest and move on to another.

I'd suggest looking into why the results are unsatisfactory and try to remedy as their are a multitude of possible reasons for the poor results during paper trading.

Some could be:

1) Poor selection criteria/parameters (fundamental and/or technical) when scanning the market for trading candidates either manually or preferably with market scanning software which should be included in any reputable charting software thus resulting in low grade trading opps.

2) Poor entry/exit criteria and parameters if using charts.  Maybe the trader just needs to read up or whatever a bit more on charting techniques to hopefully improve their charting skills.

3) Maybe the trader needs to fine tune how he/she sets their position size

4)  etc etc etc

I'd be surprised if there wasn't some modifications to the plan during paper trading.  Basically, setting up and fine tuning a trading plan will require a lot of time and effort to get it to a stage where the trader will feel comfortable with starting to commit real funds.  Now how long that takes will depend on the attitiude of the trader, their temperament how often they will likely trade etc etc.

But now after my having said the above, someone could also ask in reply to your _"Early returns are most unsatisfactory, I give up on this plan, and go to another plan and repeat."_  -  *What if the initial results ARE satisfactory?* I don't see anything wrong in the trader then commencing to commit real funds to their plan 

But then, even after trading with real funds you might recall in other threads that I have suggested periodically reviewing one's trading plan to firstly monitor its performance against preset bench marks and then fine tuning it if required.  

Don't get me wrong - the ideas/concepts I describe in this and earlier posts are by no means going to guarantee any significant profits in the long run, human nature and weaknesses being what they are, _but there is no way you are going to convince me that following what I have posted earlier will not at least improve the probability of a trader being in the 10% or so of traders that end up being profitable in the long term. _ 

How profitable they end up being in the long run will depend on things like their attitude, willingness to put in the large amount of time and effort required to be a successful professional trader and their skill in technical and/or fundamental analysis.  Personally I don't have the temperament to be a full time trader.

Finally, now I'm sure others will have different views and methods on what they believe is the best way to approach trading - *but imo the best approach is what the trader is most comfortable with.*  After all, if you're not happy and hence reluctant to put in the time and effort in whatever plan structure you choose there is a high probability that you will most likely simply loose interest and fail in the long run.

cheers

bullmarket


----------



## ducati916 (9 March 2006)

*Snake* 



> So curve fitting and overoptimisation should be included in the discussion then.
> 
> Curve fitting: matching a system to the data. To make a system look great for the period and data given.
> 
> Overoptimisation: fudging the parameters of a sytem until it works. Much like changing the moving averages until you find one that gives the best exit entry etc. So as a result it only works on the data tested on and is no guarantee of future success




Absolutely.
Curve fitting and overoptimization sound very much the same as data mining.
These definitely need to be examined, and if present pretty much nullify any conclusions drawn from the data, and potentially open the trader/investor to serious losses.



> How does one overcome these biases in the search for the edge




By having a very high standard within the methodological design.
By starting from the bottom up, as opposed from the top down (seemingly the route taken by current software, although, currently this is simply my *feeling* ) we can examine the methodological quality, and decide as to the quality of the conclusions drawn

jog on
d998


----------



## happytrader (9 March 2006)

Hello again Ducati

Why not go with the 'Best Practice' model and keep doing what you know works most of the time?

Sure it doesn't GUARANTEE perfection, but hey thats you, me, everyone else and life. Who really needs to be RIGHT all the time anyway to succeed?

We've all got to roll with the punches anyway, dust ourselves off and get up.
Anyone here know a trader that hasn't taken a decent hit? I don't.

Cheers
Happytrader


----------



## bullmarket (9 March 2006)

hi happytrader 



			
				happytrader said:
			
		

> Hello again Ducati
> 
> Why not go with the 'Best Practice' model and keep doing what you know works most of the time?
> 
> ...




:iagree: with the jist of your post and its similar to the point I was making earlier re traders/investors should work with what works best for them according to their personalities, temperament, objectives etc etc.

*I think it's a good idea to toss around different views and ideas in a forum like this but imo when people start insisting that their way is the best or whatever then from my point of view they leave themselves wide open to being laughed at by those who are much more successful than them and who didn't use their 'methodolgies' or whatever you want to call them *

Imo choosing a trading/investment strategy and how you implement that is a case of horses for courses to some extent as there is no single 1 way of trading/investing which is the best because if there was then everybody would be using it 

cheers

bullmarket


----------



## ducati916 (9 March 2006)

*bullmarket* 

I can see that you have a very firm grip on the wrong end of the stick, and are not going to let go without a fight. Ok, here we go.

You can test a trading or investment methodology several different ways.
We shall call your outlined methodology *method #1* 

In method #1
We manually test a trading plan by forward testing the plan.
Or, we backtest the plan using historical data.
In both instances, we do it ourselves.

Forward testing has one major flaw, and that is it is time consuming.
It cannot progress faster than one day at a time.
The second flaw (potential flaw) is the statistically insignificant sample size.
Therefore, statistically speaking, you are trading, and will trading for a very long period of time, a purely *discretionary method* 

If you backtest, using historical data, then you must ensure that you have a methodological design quality that will deliver statistically significant outcomes.

This is what this thread is going to discuss.

When backtesting, you can go the slow, labour intensive method, that is time consuming, *method #2*  but has one major advantage in that you understand exactly the inputs into the methodological design, or, you go the automated route, that is fast, but, is not exactly transparent within the methodology and this can be *method #3* 



> Don't get me wrong - the ideas/concepts I describe in this and earlier posts are by no means going to guarantee any significant profits in the long run, human nature and weaknesses being what they are, but there is no way you are going to convince me that following what I have posted earlier will not at least improve the probability of a trader being in the 10% or so of traders that end up being profitable in the long term.




Focus on the word *probability* 
Probability has nothing whatsoever to do with the context in which you are using it. Probabilities within financial markets are argued over endlessly, and it really depends on how you view *linear and non-linear theory, complexity theory, power laws, equilibrium, fractals, determinism and other quite fascinating theories* 

Probability does have relevance, to a degree, in that it has as a concept been so misused, that it has been accepted into trader lore, *high probability set-up, high probability trade, etc, etc* 
Also, with mechanical systems, there has been the hint of an inference that the "numbers" portray a high probability of success..........
We'll see.



> How profitable they end up being in the long run will depend on things like their attitude, willingness to put in the large amount of time and effort required to be a successful professional trader and their skill in technical and/or fundamental analysis. Personally I don't have the temperament to be a full time trader.




I tend to agree with you on this one.
The mecchies, however will not, it will they argue take as long as it takes you to generate positive numbers.
Will it?




> Finally, now I'm sure others will have different views and methods on what they believe is the best way to approach trading - but imo the best approach is what the trader is most comfortable with. After all, if you're not happy and hence reluctant to put in the time and effort in whatever plan structure you choose there is a high probability that you will most likely simply loose interest and fail in the long run.




Comfort zone............
"Losers do what is comfortable, Winners do what is necessary"
Psychological compatibility, intellectual compatibility, emotional compatibility
All areas of comfort.
Comfort can be a mind-set. What starts out as "work" can, with the right attitude become pleasurable, or can it?
Interesting.

jog on
d998


----------



## ducati916 (9 March 2006)

*happytrader* 



> Why not go with the 'Best Practice' model and keep doing what you know works most of the time?




Don't really follow your post here.
Best Practice model, refers to a benchmark, for comparative values.

*bullmarket* 



> I think it's a good idea to toss around different views and ideas in a forum like this but imo when people start insisting that their way is the best or whatever then from my point of view they leave themselves wide open to being laughed at by those who are much more successful than them and who didn't use their 'methodolgies' or whatever you want to call them




Yes agree.
Does that mean that I will not be subjected to another post on your method?



> Imo choosing a trading/investment strategy and how you implement that is a case of horses for courses to some extent as there is no single 1 way of trading/investing which is the best because if there was then everybody would be using it




That is most likely true, but it still doesn't exclude the possibility that the vast majority do, and you're just the odd one out.

Intrestingly, when you start to examine the successes in the financial markets, you start to notice that there are far more similarities, than differences................but that really is off topic.

jog on
d998


----------



## bullmarket (9 March 2006)

no problem ducati 

I just posted my views with supporting reasons in reply to your posts and that is all I can do.....if we disagree then that is fine - I don't have a problem with that 

*I'm not out to try to prove that you are wrong and I am right and I don't accept that you are right and I am wrong.*

As I said before its horses for courses and I will simply stick with what works for me.  Having said that, obviously what works for me will also work for some and not work for others....it's as simple as that   I will just continue to call things as I see them with supporting reasons.

As I said before, those who try to insist they are right and everyone else is wrong (and I'm not saying you are doing that here) leave themselves wide open to being laughed at by those much more successful than they are and who also don't agree with them.

You have your views and supporting reasons and I have mine with my supporting reasons and that's all there is to it....both will have pros and cons 

cheers

bullmarket


----------



## professor_frink (9 March 2006)

hi ducati, interesting thread you've started here.
quick question- Is it technical analysis or pure mechanical trading that you don't like?


----------



## tech/a (9 March 2006)

ducati916 said:
			
		

> I personally think no, it wasn't, it had many similarities to the *TURTLE system* Now this is not a criticism (not yet anyway). Was the methodology (possibly undisclosed) examined in anyway for variables that were similar or the same within TechTrader?




Initially there was discussion about the Turtle Trading method and there was a method designed around that method--its far from the final T/T being used today.The only similarity is that the method is a longterm trend following method.



> From this starting point we can ask the following questions;
> 
> Is this study larger, continued for longer, or otherwise more substantial than previous studies (studies equating to trading methodologies)




No.Its the longest I have seen walking forward though and public,there maybe more somewhere that are public and I like most would love to follow them.I was originally going to pull the public trading of T/T after year 1 but was convinced to leave it running.



> Is the methodology of this study any more rigorous (in particular, does it cover any specific methodological criticisms of previous studies)




No. Infact it is involved in many of the critisisms I and many others have in system testing portfolio methods in particular.Single entity methods IE Futures can be studied more rigorously--but I'll get to that in a moment.



> Will the numerical results of this study add significantly to a meta-analysis of previous studies?




No,but it has gone a long way in helping some find a methodology that suits them as there are a few hybrids I know of.



> Is the population studied different in any way (has the study examined capitalization structures, duration of listing, economic environment, market environment, inflation, interest, alternate investment classes etc)




Yes and No.Yes by way of culling a universe of stocks simply based upon selection of allowable traded stock by the margin lender in the case in question that was/is BT (Bankers Trust). No in that the restrictions due to data are common to 99% of systems testers relying on retail data.


*The single biggest* problem a systems or method trade/developer has is the one of survivorship.Todays list is used for testing whether that be the BT list the ASX or ASX300 or any group of stocks you wish to include.That list is only valid for today.
Testing data 10 yrs ago is impossible we just dont have the list of stocks that were either included in the margin list or the ASX and your criteria could have selected/traded and been caught out by delisted stocks not in your data base---today.

So that brings the question which I have asked myself often as to how accurate then are the results of the method and could my "Blueprint" (The statistics which form the basis for comparison of performance as I trade the system forward) be out far enough to risk ruin?

I have perhaps answered my own question (Albeit convieniently) by reasoning the following.
Buy placing a set of conditions with a set of variables into a method and testing it over a great deal of timelines and stocks (Both in the AUST bourse and Overseas) results are consistantly within acceptable parameters.I have not tested one case where ruin would occur.
Before you get all tied up in accuracy I feel one aspect of this and any low risk parameter set method will distribute the risk broadly enough to avoid ruin if even 2 or 3 outliers of delisting hit it at once.No I dont have the stats on that but I do know that there are even fewer delistings of stocks in a sustained up trend than downtrend (From observations of charts).Thats what the method is designed to trade.
So by applying the method rules to all universes I have tested---as of yet I havent found a case for ruin.

Forward trading of 16 versions of this method from reports from those who keep in contact with me report that they are having similar results to the "Numbers" found in the original design traded on Reef.

This doesnt mean that there couldnt be a case of ruin waiting out there for anyone who got their numbers from trading at the worst time in history.
Yes, Ive also heard that even idiots can profit in a bullmarket--well I'm living proof,there are a great deal of us however that no matter how bullish a market is consistant profit avoids us and ruin is familiar.

While not statistically perfect due to the in ability to gain correct data.The entry,exit,stop and position sizing model has shown in every case I have found to be profitable and without a single case of ruin.

I'm also yet to see a test or a traded method which has numbers which fall below those in the "Blueprint" spat out in the original method.
If actual forward trading returned figures outside (To the negative) to those found in the "blueprint" then I would stop trading it.I'm also yet to see trading above the higher end of the Montecarlo results of the method,although I have achieved that end.

Now is it the best method around---I doubt it.
Could it be better---yes I'm sure it can as I have a better than "Original" I trade myself. I also trade the original.
Is it the most efficient---hardly.
Does it suit me ---absolutely.

Having said this I also believe that data has exactly the same effect on any method designed for portfolio trading and as such results will be similar to my own in respect of accuracy and acceptability to the designer.
Acceptable to some maybe not others.

Back to sleep.


----------



## Milk Man (9 March 2006)

Quick question: what is the Turtle Traders method? Ive heard of it before but Ive never seen exactly how it works.

I do see your point Ducati. I have the same reservations myself about backtested systems. However, I still believe that a backtested, mechanical system is the best way to consitently return a profit. As tech said, if parameters fall outside those of your testing then its back to the drawing board. 

Having said this, is it then a bad idea to start up your own business? The figures are similarly bad to trading and they only have, what you call, methodological testing procedures. As I see it, people who start their own business have a pretty firm grip on the methodological side of things as they generally would have worked in the industry first. It is then, my theory, that it is the lack of risk management (and backtesting) that cannot assure everyone a profit.

Is it then a bad idea to start your own business? No...
Everything in life involves an element of risk. The way risk is managed directly influences our success; using backtesting can only be a positive for this. However, the methodology is an important part of designing a successful system.


----------



## Bobby (10 March 2006)

Milk Man said:
			
		

> Quick question: what is the Turtle Traders method? Ive heard of it before but Ive never seen exactly how it works.
> 
> I do see your point Ducati. I have the same reservations myself about backtested systems. However, I still believe that a backtested, mechanical system is the best way to consitently return a profit. As tech said, if parameters fall outside those of your testing then its back to the drawing board.
> 
> ...



Hullo Milkman,
The term turtle traders was what a great trader (Richard Dennis ) called his new apprentices to the world of trading back in 1983, as most young turtles don't make it to an adult .
The original method is on the net just google it.
Bob.


----------



## wayneL (10 March 2006)

Bobby said:
			
		

> Hullo Milkman,
> The term turtle traders was what a great trader (Richard Dennis ) called his new apprentices to the world of trading back in 1983, as most young turtles don't make it to an adult .
> The original method is on the net just google it.
> Bob.



http://www.tradingblox.net/Rules/turtlerules.pdf


----------



## Milk Man (10 March 2006)

Thanks guys.


----------



## ducati916 (10 March 2006)

*Prof.* 



> quick question- Is it technical analysis or pure mechanical trading that you don't like?




Mostly technical analysis.
I actually think that most people would be far better off with some form of mechanical or formula methodology.

*tech/a* 



> Initially there was discussion about the Turtle Trading method and there was a method designed around that method--its far from the final T/T being used today.The only similarity is that the method is a longterm trend following method.




The discussion that revolved around the Turtle system identified flaws that TT sought to rectify?
If that is the case, then this would be a positive.
It also answers the question



> Is the methodology of this study any more rigorous (in particular, does it cover any specific methodological criticisms of previous studies)




As the methodology was modified, or completely revamped, is there is no point of comparison with previous methodologies? If there is no point of commonality this would thereby reduce the total gross number of the population, as we cannot complete a meta-analysis thereby increasing the total population, and complying with a power calculation that may push the study into the *statistically significant*
If there are however points of commonality, viz. the *good bits*, how did you utilize this data?



> Is this study larger, continued for longer, or otherwise more substantial than previous studies (studies equating to trading methodologies)






> Forward trading of 16 versions of this method from reports from those who keep in contact with me report that they are having similar results to the "Numbers" found in the original design traded on Reef.




Therefore have you actually looked at the number of Turtle Traders that there were, how long they traded the system etc?
This is definitely a positive as more data is gathered, as I understand none of the other 16 portfolios are the same. However, how many do actually replicate positions. By way of example UTB was a 190% return in your public TT. Have any of the other 16 replicated the same results?
If not why not? What accounts for the variability? Is it systematic, or individual? Did testing indicate this variability prior to taking it live? If not why not? And so on.

The problem, and this has nothing to do with your research, is that trading methodologies, and research of trading methodologies are extremely secretive
People, Institutions, do not make available proprietary trading information.
Publically listed Investment companys have to disclose results, and positions, but do not have to disclose in any meaningful way their methodology.

This leaves very serious questions as to benchmarks.
This is a problem that can seriously bias your statistical results, and potentially cause serious consequences.



> Is the population studied different in any way (has the study examined capitalization structures, duration of listing, economic environment, market environment, inflation, interest, alternate investment classes etc)




Here I would put it to you that there are some very different fundamental factors that differentiate the populations studied;

1...ASX stocks do not utilize Market Makers in the same manner as the US
2...ASX stocks have far fewer adjustments of Capitalization structures (stock splits would be an example)
3...ASX stocks do not have anywhere near the Short-selling capabilities (you can short sell, but, it is not an easy thing to do)
4...Differences in liquidity, and thus spreads.
5...Differences in traded Options on stocks
6...Differences in volatility
7...Differences in volume
8...Differences in foreign stocks listed



> I have perhaps answered my own question (Albeit convieniently) by reasoning the following.
> Buy placing a set of conditions with a set of variables into a method and testing it over a great deal of timelines and stocks (Both in the AUST bourse and Overseas)




There are others, but just in these examples, there is enough variability to invalidate from a statistical point of view, calling them a common population.
By invalidating the gross numbers in the population, it must also call into question the validity of the found results due to the law of small numbers.
On the software, is there any *POWER CALCULATION* calculation?
This would determine the required number to generate a statistically significant result. I suspect there will not be any, as this would highlight immediately the *population* is woefully low, and no statistical accuracy can be inferred from the calculated data.

If we examine your inclusion criteria against our medical benchmark by asking the following two questions;

1......Who was included in the study?
Many medical trials routinely exclude patients with co-existing illness, those who do not speak the native language, using other medications etc.
This approach while scientifically clean has no logic behind the reasoning as clinical trial results will be used as "best practice", within wide patient groups.
It makes no sense, as, the requirement in the real world, is very different from the research protocol.

2...Who was excluded from the study?
For example, a randomised controlled trial may be restricted to patients with moderate or severe forms of heart failure, this selection policy could lead directly to false conclusions regarding patient populations with *mild* heart failure

The two questions illustrate the seriousness as to the distortions that can be induced from *selection of the population* and forms the selection bias that invalidates studies from drawing statistically derived conclusions.
It would seem that the selection of stocks to be included has some serious issues to be overcome prior to drawing statistically derived conclusions.



> The single biggest problem a systems or method trade/developer has is the one of survivorship.Todays list is used for testing whether that be the BT list the ASX or ASX300 or any group of stocks you wish to include.That list is only valid for today.




Yes, mechanical testing fails on survivorship bias.
That also invalidates the statistical data generated.
However, you are jumping ahead of the curve, I have not yet worked my way through all the potential biases.



> While not statistically perfect due to the in ability to gain correct data.The entry,exit,stop and position sizing model has shown in every case I have found to be profitable and without a single case of ruin.




Jumping ahead again...................
Data, quality, and surrounding issues will all be looked at.

*milk* 



> I do see your point Ducati. I have the same reservations myself about backtested systems. However, I still believe that a backtested, mechanical system is the best way to consitently return a profit. As tech said, if parameters fall outside those of your testing then its back to the drawing board.




Possibly it is for the majority, but then again possibly not.
The problem being that many do not really understand what they are in actuality trading. If you understand all the inherent weaknesses, and have allowances, or tolerances for the failings, then that is quite different from trading a computer print-out without any understanding.

By the time it falls outside of your parametres, you could be taking up your palatial residence at #3 dustbin. Remember some people leverage these systems to the hilt, trade them in futures markets etc.

I'm only using TT as it is a publically disclosed system.
Most are very secretive, and thus you'll never see the blow-ups.
If you are committed to a mechanical system, thats fine, I actually am quite pro-mechanical, but informed mechanical.



> Having said this, is it then a bad idea to start up your own business? The figures are similarly bad to trading and they only have, what you call, methodological testing procedures. As I see it, people who start their own business have a pretty firm grip on the methodological side of things as they generally would have worked in the industry first. It is then, my theory, that it is the lack of risk management (and backtesting) that cannot assure everyone a profit




Well I would beg to differ.
Most start-ups go belly up.
Most who start small businesses are under capitalized, clueless, and destined to failure, not lasting past the first year or two.

Trading/Investing is, or should be, a small business.
How many coming into this *industry* knew what they were doing prior to entering? How many have a clue even after 3 yrs?

tech/a
I'll come back to you with the next set of questions, after I punish the gym equipment, Friday, is my relaxation day.

jog on
d998


----------



## Milk Man (10 March 2006)

http://www.tradingblox.net/Rules/turtlerules.pdf

Wow! That is pure gold, no, platinum! That is even more insightful than Nick's book. Everyone should read that before considering trading, even if not the system itself, but just the methodology.

:bananasmi


----------



## ghotib (10 March 2006)

Please, where is the study and/or the discussion of it that Tech was talking about in the the quotes at the start of this thread? 

Hopefully if I can read that I won't have to bother you all with a stack of other questions about what we're talking about. 

Cheers,

Ghoti


----------



## tech/a (10 March 2006)

> *tech/a*
> 
> 
> 
> ...




The turtle system was never pulled apart or studied to any degree.
All I/We looked at was how it entered and exited as a basis from which to start. The Good bits were that it traded a trend over long time frames.Nothing more complex than that.



> Therefore have you actually looked at the number of Turtle Traders that there were, how long they traded the system etc?
> This is definitely a positive as more data is gathered, as I understand none of the other 16 portfolios are the same. However, how many do actually replicate positions. By way of example UTB was a 190% return in your public TT. Have any of the other 16 replicated the same results?
> If not why not? What accounts for the variability? Is it systematic, or individual? Did testing indicate this variability prior to taking it live? If not why not? And so on.




Re Turtle system answered above.
The 16 portfolios were started at various times myself and Daryl both traded UTB to its end.As for the other 15 traders using the method and or a hybrid i dont have individual trade results.But i would say that most got a good part of UTB.As start dates will obviously have an impact on which entries are triggered I would expect there to be both same and different stocks in portfolio's over different start dates. Incedently trading EVERY stock selected by the method (had you the funds) returns massive profit.



> The problem, and this has nothing to do with your research, is that trading methodologies, and research of trading methodologies are extremely secretive
> People, Institutions, do not make available proprietary trading information.
> Publically listed Investment companys have to disclose results, and positions, but do not have to disclose in any meaningful way their methodology.




True and thats why I am happy to have this public,I want people like you to pull it to pieces and many have added constructive critisism over the years.Its as much a learning process to me as anyone.With something to atleast benchmark that is working there is a starting point for discussion.To this end it has/is serving its purpose---and making a buck while I/we learn more.



> This leaves very serious questions as to benchmarks.
> This is a problem that can seriously bias your statistical results, and potentially cause serious consequences.




There is one benchmark I need and that is Profit.I dont need to have a comparitive benchmark Im happy to improve on the singular benchmark printed by the method itself.



> Here I would put it to you that there are some very different fundamental factors that differentiate the populations studied;
> 
> 1...ASX stocks do not utilize Market Makers in the same manner as the US
> 2...ASX stocks have far fewer adjustments of Capitalization structures (stock splits would be an example)
> ...




Absolutely correct but you are presuming that a method should work within its "blueprint" in every situation you place it in.I disagree the system and any system can be designed around any set condition and conditions.The 8 condition "variables" you mention above have little impact on the specifics of the method.



> There are others, but just in these examples, there is enough variability to invalidate from a statistical point of view, calling them a common population.
> By invalidating the gross numbers in the population, it must also call into question the validity of the found results due to the law of small numbers.
> On the software, is there any *POWER CALCULATION* calculation?
> This would determine the required number to generate a statistically significant result. I suspect there will not be any, as this would highlight immediately the *population* is woefully low, and no statistical accuracy can be inferred from the calculated data.




Well the population or Universe of stocks used in the test from the BT margin list is 327 stocks and the number of trades tested is over 800 (Possible but due to capital constraints only around 120 trades taken ---from memory I'm at the office). This has been replicated by countless other both using metastock/amibroker and wealthlab. Many select universes well away from the parameters set when the system was initially designed. Other than using far to small timeframes I am yet to recieve a test result showing ruin despite 100s if not 1000s of tests and various universes both in size and criteria.



> If we examine your inclusion criteria against our medical benchmark by asking the following two questions;
> 
> 1......Who was included in the study?
> Many medical trials routinely exclude patients with co-existing illness, those who do not speak the native language, using other medications etc.
> ...




Duc your again setting your own standard of one that must have a system satisfy every situation and every universe and under every condition.
I dont agree that that is necessary and think "so Far" I've been able to prove that point. I have made it clear the selection process for which the method was/is designed attempting to adapt it to other universes and in a bear market would have the same consequences as using asprin to treat diabetes.

The broader you design a method the more complex will be your criteria.

To the other extreme trading a singualr entity such as a future, is just as valid as trading every stock in every condition in every market in the world!
The simplistic and profitable method used to trade the Future would have little chance of being profitable in every concievable situation and this is the same for every method designed.They will/should be specific to their task.



> Yes, mechanical testing fails on survivorship bias.
> That also invalidates the statistical data generated.
> However, you are jumping ahead of the curve, I have not yet worked my way through all the potential biases.




Here we disagree.Incomplete in a specific set of data yes invalidates??


----------



## tech/a (10 March 2006)

Ghotib.

If you mean TechTrader then there are 100s of pages written on the topic and it is traded live with each weeks results published by Darryl,who first asked me the question on wether a mechanical system could be designed and traded profitably---then the link and weeks of reading are here.

http://lightning.he.net/cgi-bin/suid/~reefcap/ultimatebb.cgi?ubb=forum;f=74


----------



## ducati916 (10 March 2006)

*tech/a* 



> Duc your again setting your own standard of one that must have a system satisfy every situation and every universe and under every condition.
> I dont agree that that is necessary and think "so Far" I've been able to prove that point. I have made it clear the selection process for which the method was/is designed attempting to adapt it to other universes and in a bear market would have the same consequences as using asprin to treat diabetes.




No not really.
The question that bothers me is this..........can mechanical systems backtested by the software that has been developed fairly claim *statistically significant results* 

The answer already is a resounding no. However I wish to continue through the entire process to see if there are areas where the results are strong. We are at very early days.

The whole point of defining statistically significant results is rather important
There is a world of difference between a true statistically significant result, and one that is not.

Returning to the medical model for illustrative purposes; if a new drug, say Vioxx, was shown to have statistically significant correlation and causation to cardiac pathology, then, Merck could be in serious trouble, if not, then they get off.

If your tested trading methodology has a statistically significant result, that should mean that you are mathematically assured of the result, but, if it is not statistically significant, then you might get the result.

For all the above to be correct, the study design must be of a high methodological quality, otherwise it is *mathematically impossible to claim a statistically significant result* 

The big advantage that mechanical traders enjoy over *discretionary traders is a psychological buffer zone engendered by a belief that they are trading a mathematically proven methodology* 

What if this is a false premise?
Currently it certainly looks incredibly fragile.



> Well the population or Universe of stocks used in the test from the BT margin list is 327 stocks and the number of trades tested is over 800 (Possible but due to capital constraints only around 120 trades taken ---from memory I'm at the office). This has been replicated by countless other both using metastock/amibroker and wealthlab. Many select universes well away from the parameters set when the system was initially designed. Other than using far to small timeframes I am yet to recieve a test result showing ruin despite 100s if not 1000s of tests and various universes both in size and criteria.




Total ruin is pretty unlikely, given adequate money management, which I know that you employ. Rather it is a failure to replicate the *success* that the system implies, this could be a major disappointment.

Certainly the timeframe is wholly inadequate, 1996 was the earliest data that was available I believe. The earliest data that I have heard of is 1982, again just nowhere near enough. I have seen examples of just 3/4 yrs being used, this is just delusional and could well engender a blow up if the drawdown was particularly aggressive.




> 1...ASX stocks do not utilize Market Makers in the same manner as the US
> 2...ASX stocks have far fewer adjustments of Capitalization structures (stock splits would be an example)
> 3...ASX stocks do not have anywhere near the Short-selling capabilities (you can short sell, but, it is not an easy thing to do)
> 4...Differences in liquidity, and thus spreads.
> ...




But how do you know that?
The fact is you don't.
The design of the methodology is flawed and invalidates any conclusions that you draw on a statistical basis. 

It may be that you are in point of fact correct, but without testing it with a robust statistical model, you and I are both exhibiting a bias, and that is what statistical modeling seeks to eliminate, or marginalize.



> The 16 portfolios were started at various times myself and Daryl both traded UTB to its end.As for the other 15 traders using the method and or a hybrid i dont have individual trade results.But i would say that most got a good part of UTB.As start dates will obviously have an impact on which entries are triggered I would expect there to be both same and different stocks in portfolio's over different start dates.




This would imply that it was individual rather than systemic.



> The turtle system was never pulled apart or studied to any degree.
> All I/We looked at was how it entered and exited as a basis from which to start. The Good bits were that it traded a trend over long time frames.Nothing more complex than that.




I was always under the impression that TT (my mistake) was very similar to the Turtle methodology. As you know they blew it up. It would have been interesting to know if it was an individual or systemic blow-up. 

jog on
d998


----------



## tech/a (10 March 2006)

Duc

I'll answer in more detail when I can.

But from the direction your going your going to be able to prove (and most anything can be proven mathematically) that due to umpteen reasons statistics from backtesting cannot be seen as "Statistically Significant"

I'm not going to argue that.

I'm also not going to stop trading the method either.
If I cant hit a golf ball perfectly everytime I'm not going to give up golf either.If I can hit it consistantly well often enough,completely miss it evry now and again,go around in a respectable score often enough for people to enjoy playing with me and every now and again hit an eagle,I'll keep playing.

Same goes with trading a potfolio.


----------



## ducati916 (10 March 2006)

Moving swiftly on.

*Sample Size* 
One crucial prerequisite prior to embarking upon the *testing* or study is the requirement to perform a *sample size, or POWER CALCULATION* In the words of D. Altman "a trial should be big enough to have a high chance of detecting, as a statistically significant, a worthwhile effect if it exists, and thus to be reasonably sure that no benefit exists if it is not found in the trial."

In our world, this will apply to profitability.
Intrestingly, and this is currently possibly being quite strongly exhibited by TT currently, what level of difference between *profitability* & *statistically significant profitability*.

Note, that this may not be the same measurement. You can have profitability, even high profitability, but it need not, and quite possibly will be statistically insignificant........viz. unrelated to the study. The obvious variable would be the presence of a strongly trending bull market.

By using a statistical nomogram the individual can prior to running the study calculate how large a sample is required to have in order to detect a moderate, high, or very high true difference between the profitability generated via the design or criteria, and variable market conditions.

The likelihood of detecting a true difference is known as the power of the study. It is common for medical studies to require a power of between 80%-90%. I have not seen any references to this calculation in any of the computer printouts from the software generated results.

jog on
d998


----------



## ducati916 (10 March 2006)

*tech/a* 



> But from the direction your going your going to be able to prove (and most anything can be proven mathematically) that due to umpteen reasons statistics from backtesting cannot be seen as "Statistically Significant"




Quite possibly.



> I'm also not going to stop trading the method either.




And that was not my expectation, nor wish.
It was simply that when a novice trader asks, how do you make a profit in the market? That the seemingly simple answer "its in the numbers" is trotted out with some caveats, and not promulgated as some mathematic guarantee.

The number of times I have been subjected to the (1+($profit/$loss)*(#wins/#losses)-1) nonsense, as if this was some form of money back guarantee. If an experienced, profitable trader, feels the requirement to offer advice, it should be offered on a true understanding, not some half-baked dimly understood regurgitation of a concept that they have swallowed whole without really doing the work.

Of course you could argue that, caveat emptor, but, hell I'm a nice guy.

jog on
d998


----------



## tech/a (10 March 2006)

Yes well we could argue statistical significance V data infinitum.
While in a longterm method such as that which we are discussing a larger cleaner data base would be preferable,my logic tells me that the statistical relevance if you like is---

If I set the conditions and the variables and over constant testing both from backtests and forward trading the results are consistantly within "blueprint results" regardless of how accurate they are---as even as un accurate as they could be a profit is found as "expectancy" in the tests and "actual" in the forward trades.If it were to fall outside the blueprint then trading would and should cease.
Until then the expectancy is intact and manifested in profit.


----------



## wayneL (10 March 2006)

Duc,

The expectancy equation is not nonsense. It is a tool... a measuring stick.

It can be used in an innappropriate way, and I suspect this is what you are refering to. I think "expectancy" is a misnomer in the best use of the equation is not described by that term.

But it is like saying a hammer is nonsense because you don't happen to be a carpenter, even if the hammer is given the wrong name. It is clearly a silly thing to say, nonsense even, and exposes a narrowness of thought common amongst the "learned".

Cheers


----------



## tech/a (10 March 2006)

If I take on one hand the entry.exit,and stop of T/T and apply it to a say 50 trades it will have an expectancy.
If I apply it to 100 trades same again.
1000 and so on.

As it was and is positive and the Numbers thrown out from 100s of tests give me a number of "blueprints" which fit within acceptable or Not acceptable limits--

My choice due to the confidence in the amount of testing I have done and the design of the method and its moneymanagement techniques applied to it coupled with a set of parameters that I must stay within to remain as "sample" allows me to trade the method until and or if it fails to meet the criteria for positive expectancy/profit.


Its really that simple.
Dont care how accurate it is compared to the standard set by Duc or X.


As for novice traders their confidence and ability to build a plan and trade a method with positive expectancy consistantly and over a long period of time may well lie in their ability and thoroughness of testing.
However to cover every variable and situation for Joe Average is not possible nor NECESSARY in my view.

The future may prove me wrong but in the meantime I'll take whatever it gives me--.

I'm trading the Numbers Duc and when they tell me not to----I'll stop.
Theyve been sufficient for 3.5 yrs so far. Infact all 3 methods have been.


----------



## ghotib (10 March 2006)

tech/a said:
			
		

> Ghotib.
> 
> If you mean TechTrader then there are 100s of pages written on the topic and it is traded live with each weeks results published by Darryl,who first asked me the question on wether a mechanical system could be designed and traded profitably---then the link and weeks of reading are here.
> 
> http://lightning.he.net/cgi-bin/suid/~reefcap/ultimatebb.cgi?ubb=forum;f=74



Thanks Tech/a

Don't know if I can just shut up and read for weeks, but I'll try 

ghotib


----------



## tech/a (10 March 2006)

ducati916 said:
			
		

> *tech/a*
> 
> 
> 
> ...




Actually Duc, If you/anyone has a positive expectancy model and they trade it within the numbers generated by that model then yes I would guarantee that you would profit. Provided of course that sufficient testing had taken place.I think that the 6 mths testing in the case of T/T has proven to be sufficient. Of course what is sufficient can be argued ad infinitum. I'm not the first or last to come to that conclusion.Stevo has a weekly method that is doing as well as T/T. As has Andrew.

If you can offer evidence to the contrary I'm all eyes.


----------



## tech/a (11 March 2006)

*Trading Plan*

This is the _"half-baked dimly understood regurgitation of a concept that they have swallowed whole without really doing the work."_ That I know duc is *NOT* refering to.
But I am.
We are told constantly that to be successful *HAVE A TRADING PLAN* and even more dangerously ---DONT deviate from your plan--cause if you do whats the point of having a plan in the first place---Makes sence.

There is a MISSING LINK.
Without knowledge of how your plan performs you could be on the road to ruin with a poorly devised plan.

This is what this discussion is about.

The *RESULTS* of your plan.
Firstly it MUST have a proven positive expectancy over a period and number of tests that YOU have confidence in.

Where Duc and I differ (In part at least) is the accuracy and sample testing of any method.
I dont believe every concievable condition or variable need be met. I think and so far have proven (The future may prove me wrong),That provided you trade the *Numbers* of the tested positive expectancy method that have been generated during testing,and cease to trade it if actual trading figures fall outside those generated in testing---*you'll make a profit within the parameters returned in testing.*


----------



## happytrader (11 March 2006)

Well researched, well backtested and then tested on a small scale in the real market with real money is a low risk way to start. Think of this as your tuition fee as you gain insight into how you behave.

I am always amazed at how many people blatantly don't read or follow directions and instructions unless they are penalised in some say. Its like some passive aggressive behaviour where they are compelled to do the opposite. Just because most market participants are adults doesn't mean they have out grown the behaviour. Chances are if you react this way on being 'told what what to do' by an authority figure which looks to me how most people interpret instructions you may have difficulty following through on your own authority which is what you are doing when you conceive a plan 'to tell you what to do and when to do it' I'm just guessing by the way. 

Watched 'The Naked Chef' Jamie Oliver selecting would be apprentices. Point to remember here is they would be given the opportunity to rub shoulders with and learn from a very famous and successful chef. All he was really looking for is someone to do what they were told.They were given very specific instructions on how to prepare, cook and present a particular dish.  Most had memory lapses about the sequence. Another was impatient did their own thing with the timing and temperature. Another completely overrode all instructions. Another didn't trust the cooking time and second guessed it. Another did okay till it came to presentation however he decided to push his own creative bent. Out of half a dozen hopefuls, only one made a concerted effort to do 'just so'

Cheers
Happytrader


----------



## stevo (11 March 2006)

*Just read the bits in bold to get the short version of this post*  

Duc
You stated that _"Forward testing has one major flaw, and that is it is time consuming."_

*A forward test can be conducted just as quickly as a back test. *It is not necessary to trade live to forward test.

There are probably a lot of ways to forward test. One method is to develop a strategy based on 5 years of data (and lets assume that we have accurate complete data for the market in question) . The strategy is then tested going forward. *So we could test from 1998 to 2003, then forward test from 2003 to 2006. *

Obviously their are lots of permutations on forward testing. You could also develop the strategy on data from 2002 to 2006, then backtest it from 1999 to 2004. *The important thing is that the data that the "forward" test is carried out on is different to the data that the system is developed with.*

Because large databases can be used for system testing and the database can be split up in all sorts of ways it is possible to develop robust trading systems going forward. *If a medical trial (and I know little of these things) is big enough then it is possible to gain some level of confidence in the results. *

I am sure that the large drug companies would love to be able to simulate medical tests on a computer but the human body is quite complex to simulate. *Forward testing by drug companies is littered with unfortunate failures, but that doesn't mean we shouldn't test drugs. *

Fortunately we don't have to simulate trading data - we can use actual price and volume data and draw on thousands of stocks for our databases.

It would be nice to have a definitive text on system testing but any I have seen are sadly lacking. *Most people don't have the skills, knowledge and perseverance to carry out comprehensive testing of any sort of trading strategies. * Many strategies touted are untestable - something the seller of the method probably appreciates.

*You cannot prove that system testing doesn't work and is totally useless, you can just state that you haven't found any evidence of it working from the limited research that you have done.* 

regards

stevo

*ALSO* - don't even mention optimisation! The debate will never end.


----------



## It's Snake Pliskin (11 March 2006)

bullmarket said:
			
		

> Hi ducati
> 
> _I think it is very important to paper trade a plan for however long it takes (and that will vary according to each individual) until the person is satisfied the plan is generating the objectives/returns specified in the plan before committing funds._
> 
> ...





Read this article on paper trading
http://www.actionforex.com/articles...r_trading_can_pump_you_up,_but_it's_not_real/

 :alien2:


----------



## tech/a (12 March 2006)

Most people have no alternative.

They develope a plan and wish to see how it goes before committing funds to it.
This is where Duc and his accuracy in sample test has credence.You can paper trade all day everyday and record every detail but wont have anything meaningful for years.
But if you could in the end youd have a set of Numbers which tell you if your profitable of not. 90%+ of the time after years the answer will be not profitable.

Fundamentalists tend to succeed simply by buying and holding.
Try trading short term Fundamentally!
Technical methods developed around longterm trading where % won far exceeds %'s lost are also simpler to develope.

While short term trading methods can be designed and can return good profits I personally havent tested one "Plan" which results in consistant profit.I'm yet to see one based on a portfolio style of stock trading.

My personal feeling is that Futures methods and singular index and or stock trading short term have more chance of success than Attemping to portfolio trade short term.Developing methods with greater % wins and reasonable Reward to Risk ratios are harder in my veiw.Than less wins higher Reward to Risk.

Most plans are based upon short term methods as result can be seen earlier,in terms of days or weeks rather than Months and Years.
Spectacular gains are confused with consistancy.
Impatience at a possible 200% gain far outweighs the patience of attaining consistancy in return.

With excellent software like Amibroker available for under $500 I cant understand why people dont take the time to become proficient so that they can test their plans and find one that works and simply trade it within its "Numbers" from the "Blueprint" generated.

Forget the paper step into the 21st century!!


----------



## bullmarket (12 March 2006)

Morning everyone or to whoever is reading this   

I don't believe it has to take years before you can have any meaningful data from paper trading, though for some depending on their strategy I suppose it could.

I know a hand full of people who started out paper trading quite a few years ago before commiting real funds and who are still profitable now.  At the time they paper traded for anywhere from 6 to 12 months and from memory made on average a few 100 hundred trades before they were convinced the results from their paper trading were good and reliable enough for them to start trading using real funds.......but as I said in earlier posts, it's horses for courses and what suits and works for each individual.

The obvious downside of paper trading is that you are not under the same emotional stress (fear and greed) as you might be when trading with real funds and so the hard bit is to continue to allow the criteria and parameters in your trading plan to drive buy/sell decisions and not let emotions interfere after starting to trade with real funds.

Now some people will obviously be able to resist letting emotions influence buy/sell decisions during real trading better than others - that is just naturally occuring differences in peoples' personalities, temperament and general makeup etc.

Personally, based on my experiences over the years, I still believe paper trading is one good way to test and fine tune a trading plan before commiting real funds 

cheers

bullmarket


----------



## tech/a (12 March 2006)

Bulldust from someone who has never traded and who has never developed a method which is used for trading,who has never been involved in systems testing or developement,your parrot like "Each to their own" "Horses for courses" "What suits the individual" is getting as boring as hell.

If thats the case might as well not comment Bulldust as really everything is 
"Horses for courses,each to their own,what suits the individual."

*Get off the fence*

Put some meaningful content into your posts if your going to comment.
A few hundered trades as a basis for meaningful results is like saying I can fly a kite so now I'll hop in a Jumbo and fly that just as well as the kite.

T/T had 50,000 (About 30 different times) portfolios tested all with 120 trades in each,Thats 6 million trades and according to Duc even thats hardly meaningful.
Its also been tested by 100s of others over 4 yrs so there would be millions and Millions of trades. (I'm only using T/T as an example as its the only one I know of made Public),It was developed purely for discussions just like this.
And I welcome people pulling it to bits.
After all I trade it with real $$s and if there are 100s of people having critical input ,Thats far better than 1 person--me constantly looking at it.

If you dont know Bulldust you dont have to comment its not compulsory.

And before you wipe this Wayne my opinion is just as valid as Bulldusts,when it comes to input.


----------



## bullmarket (12 March 2006)

no problem tech/a 

You're clearly upset and frustrated   

I am just calling things as I see them with my supporting reason and will continue to do so here or anywhere else I see fit with no consideration whatsoever for what you think of my posts 

If you disagree with anything I say that is fine....I don't have a problem with that because I don't see how what you think of my posts can be of any interest let alone of any consequence to me.

If my posts annoy you so much, then why not put me on your ignore list......seems like a quick and easy solution to release your frustrations 

see you in the soup 

bullmarket


----------



## stevo (12 March 2006)

Why paper trade if you can trade for real? 

If someone isn't sure of their strategies they could paper trade. I would need to paper trade for about 3 years before I could say that a method looks like it might work - an impractical approach to testing for many. Shorter term traders would need to get at least 100 trades under the belt - 100 trades would take me 4 years. 

So paper trading is impractical for longer term methods and also quite frustrating. Imagine paper trading a successful system for 4 years and knowing how much money you have forgone. "If only I had put real money down!"

stevo


----------



## bullmarket (12 March 2006)

Hi steve



			
				stevo said:
			
		

> Why paper trade if you can trade for real?
> 
> If someone isn't sure of their strategies they could paper trade. I would need to paper trade for about 3 years before I could say that a method looks like it might work - an impractical approach to testing for many. Shorter term traders would need to get at least 100 trades under the belt - 100 trades would take me 4 years.
> 
> ...




yes I agree in general with you, but I am not suggesting paper trading is suitable or will be beneficial to all.

The friends I was referring to made at least a few hundred paper trades in 6 - 12 months roughly before they were satisfied their strategy was likely to be profitable.  For them, paper trading was beneficial. 

Imo paper trading is just one option a trader can use when starting out to test their strategies before commiting real funds.  I'm not suggesting at all that everyone will gain useful information from paper trading - obviously some will, some won't.

cheers

bullmarket


----------



## ducati916 (12 March 2006)

I see that the discussion is fluctuating a little, and that we are starting to discuss some rather diffuse variables.

Just to bring it back into focus a little.
The object is to really ascertain whether the software that is available on the market today is really utilizing statistical methodology to generate the results, and if they are, is the design quality of a high enough standard that you can actually claim a genuine statistical significance to the result.

If, however the answer is a negative in regards to the statistical significance, is there any benefit to the utilization of the testing methodology that is undertaken.

To my mind the answer to question one, is that the statistical quality is too low to qualify as a *statistically significant result* and therefore this would be potentially misleading to the average consumer purchasing this software.

But as *stevo* has joined the discussion, and I believe that he trades a successful mechanical system, I shall address the following questions to both *tech/a & stevo* 



> Because large databases can be used for system testing and the database can be split up in all sorts of ways it is possible to develop robust trading systems going forward. If a medical trial (and I know little of these things) is big enough then it is possible to gain some level of confidence in the results.




Introduced is the concept of *confidence* 
This has a great deal of statistical significance. Confidence is correlated to *probability* 

One of the first values a student learns to calculate is the *P value* - that is the probability that any particular outcome would have arisen by chance. Standard scientific practice which is entirely arbitrary, usually deems a P value of less than 1 in 20 (expessed as P<0.05) as statistically significant and a P value of less than 1 in 100 (P <0.01) as statistically highly significant.

Therefore, if multiple outcomes are to be analyzed from the data set, you must make a correction to try and allow for this (usually achieved by the Bonferoni method)

A result in the statistically significant range (P<0.05, P<0.01) suggests that the null hypothesis should be rejected viz. the methodology has revealed a statistically relevant, and tradeable strategy.

However, a P value in the non-significant range tells you either that, there is no difference, or, there were too few subjects to demonstrate the difference if in point of fact it existed.

I have yet to see any P values generated by the software in regards to this statistical value. An absence, reinforces the poor design quality of the statistical methodology.

Which then brings us to the point that *stevo* has raised, viz. "Confidence levels"

A confidence interval (that can be run on t tests, r values, sensitivity, specificity etc) allows you to estimate for both positive trials (those that show a statistically significant difference between the two arms of the trial) and "negative" ones (those that seem to show no difference)

Whether the the strength of the evidence is strong or weak, and whether the study is *definitive* 

If you were to repeat the same trial hundreds of times, you would not get the exact same result each time, but on aggregate, you would establish a particular level of difference, or lack thereof.

This calculation is the *confidence interval* and should read something along the lines of; The 95% confidence interval around this difference is -1.2% to +12%

In this example, as zero level is overlapped, therefore we have a dichotomy, and would classify the trial as a negative, however, in reality, there "probably is a real difference, and it probably lies closer to 5% than either -1.2%, or +12%

In all the systems results that have been generated and posted, I have never seen P values nor Confidence Intervals posted as part of the computer software generated results.

Therefore, again, the quality of the design methodology is just not of a high enough quality to derive a statistically significant result.

Combine that with all the additional flaws, data mining, optimization, hidden biases, poor selection of testing criteria, you are really invalidating any semblence of credibility if putting forward a "statistical" argument for the testing that is performed.

What we have left I believe is option #2, which is a glorified supercharged way of *testing trading plans* 

The sexiness of the *Automated trading plan tester* really isn't as marketable as the current, mathematically jazzed up version that is marketed to traders.



> A forward test can be conducted just as quickly as a back test. It is not necessary to trade live to forward test.




This is a classic.
Before I lump it in with the hated expectancy calculation, would you care to expand on this, in regards to the logical basis on which a future prediction is predicated?



> You cannot prove that system testing doesn't work and is totally useless, you can just state that you haven't found any evidence of it working from the limited research that you have done.




No indeed, quite the opposite, and all the more dangerous for this wrinkle;
*that it does work, either, some of the time, or all of the time, is the danger, you will never know which one will pop up.* 

tech/a has stated that his TT system is currently "outperforming"
That in my book qualifies as a "failure of the system" as, should it be "underperforming" it would be pulled, due to the exceeding of drawdown, and results falling outside of "expectation"

That the results are positive, seemingly do not invalidate the methodology.
No need to ask why.
Psychologically, which should be the primary area to benefit from going mechanical, the guidelines have been scrapped. Would the same guidelines in a negative scenario, also be thrown out?

There are just so many variables raised, but I shall limit this post here and see how these points are addressed.

jog on 
d998


----------



## tech/a (12 March 2006)

Duc.
I'm now out of the discussion.
You simply wont look at what I have said.You'll ignore it and plough head long into argument regardless of opinion held by those who are happy with their results.
On T/T results,"Outperforming"

When tested using Montecarlo analysis of the 50000 portfolios tested 
there were NO portfolios that were not profitable.
Of the mean there were some below the mean and some above the mean.
Outperformance of T/T that I am trading is at the upper end of those above the mean. So its within the numbers and wont be pulled. Guidelines are well in place and the method is well within them.

I'll say it again.
If someone has a set of numbers from 5 tests which are giving a positive expectancy and they trade within the numbers generated then they can expect and will get a profit.If it goes outside the tested numbers then failure is possible.
Now turn that to 10 Million Tested Trades same thing.--*Knock yourself out with arguement*---I conceed that you can and will be able to prove statistical "Insignificance" you could do that with any example.
Doesnt matter *While your arguing I'm trading * until my numbers are proven to be incorrect.

Bulldust.
Its not what you say its what you dont say!! Read your posts there is NO content. Nothing presented, argued, given up as example.
Wouldnt mind if you had something to add.
You'd frustrate a room full of Buddhist Monks.

Chances are you "Used to be indecisive but now you're not so sure."

Have a lovely afternoon.


----------



## bullmarket (12 March 2006)

no problem tech/a 



			
				tech/a said:
			
		

> Bulldust.
> Its not what you say its what you dont say!! Read your posts there is NO content. Nothing presented, argued, given up as example.
> Wouldnt mind if you had something to add.
> You'd frustrate a room full of Buddhist Monks.
> ...




As I said earlier, you are clearly upset and frustrated and your resorting to petty name calling against other chatters when they disagree with you confirms, for me at least, your immaturity and frustration and reflects poorly on your character and integrity.

But each to their own.......I'll just continue to post what I like when I like, as everyone is entitled to do, and if my posts annoy you simply put me on your ignore list as I suggested before 

cheers

bullmarket


----------



## lesm (12 March 2006)

Duc,

Always enjoy your posts, including the variation from informative to argumentative and sometimes baiting of TAs.   

At times, I wonder if you are attempting to over analyse/rationalise the approach to the development or the application of trading systems.

You seem to have difficulty with the term of 'positive expectancy' would the probability of a 'positive outcome' or 'positive return on investment' be more palatable?


----------



## happytrader (12 March 2006)

lesm said:
			
		

> Duc,
> 
> Always enjoy your posts, including the variation from informative to argumentative and sometimes baiting of TAs.
> 
> ...




Hi lesm totally agree with your post. 

Even a 99% positive expectancy trading system would be a wasted effort on anyone who lacked the courage to do the Nike thing and 'Just do it'

I notice no one ever specifically mentions those most necessary of all trading traits called courage and obedience.

Cheers
Happytrader


----------



## It's Snake Pliskin (12 March 2006)

bullmarket said:
			
		

> Hi steve
> yes I agree in general with you, but I am not suggesting paper trading is suitable or will be beneficial to all.
> 
> Imo paper trading is just one option a trader can use when starting out to test their strategies before commiting real funds.  I'm not suggesting at all that everyone will gain useful information from paper trading - obviously some will, some won't.
> ...




...a bit of fence leaning here. :dunno:


----------



## It's Snake Pliskin (12 March 2006)

> ALSO - don't even mention optimisation! The debate will never end.




"optimisation"...


----------



## tech/a (12 March 2006)

Glad you asked.

Optimisation has its place for singular entities,be it a stock,index,or future.
Provided that the optimised variable is varified at and after each trade.

To optimise a set of variables and then apply that optimisation to a portfolio or group is plain crazy as the variables within the group basically render the optimisation impractical.

Again of course running an optimised methodology over an entity will give a set of Numbers---your Blueprint---so while trading is within the blueprint all OK when outside STOP---.

Thats my veiw but of course but its up to the individual,what suits some may not suit others,its subject to various points of veiw and interpretation,it could be right and could be wrong,I knew a taxi driver who once optimised his rides,he's still a taxi driver today so it must have worked for him.
Can I phone a friend??

Thats should get them going.


----------



## It's Snake Pliskin (12 March 2006)

Snake Pliskin said:
			
		

> ...a bit of fence leaning here. :dunno:




  ...generalising or committed thought?


> And finally, all of what I mention above imo should be included in someone's trading plan as part of their portfolio/trading performance monitoring process....and hence the importance imo of paper trading a plan for however long it takes prior to commiting funds so that your trading performance monitoring process can show you if your trading strategies are generating the returns you expected.


----------



## It's Snake Pliskin (12 March 2006)

Bullmarket,

How is all of this determined?



> Imo a sound plan should include but not be limited to things like:
> 
> 1) Your objectives - long and/or short term
> 
> ...


----------



## bullmarket (12 March 2006)

Hi Snake



			
				Snake Pliskin said:
			
		

> ...generalising or committed thought?




It's a committed thought 

cheers

bullmarket


----------



## bullmarket (12 March 2006)

Hi again Snake



			
				Snake Pliskin said:
			
		

> Bullmarket,
> 
> How is all of this determined?





How all those parameters are determined obviously depends solely on the individual and will also be determined by what fundamental and/or technical experience and skills they posess and finally their risk tolerances.

I have already stated what my personal investment objective is in other posts and in my signature below and I have my own investment plan to suit. 

There are many different techniques one can use for things like entry/exit signals.  I touched on some possibilities when I posted about the maths that drive the stochastic and MACD indicators in other posts last week.

There are also different techniques on setting position sizes some of which are discussed in "Trading with a Plan" by Compton and Kendall which I have often posted as a suggested.  I'm sure Daryl Guppy also talks about position sizing in his books.  The same sources apply for the various techniques for risk management.

Now, I could spend half an hour or more giving details on how I do things as an investor whose #1 priority is income but I suspect that would be a complete waste of time since I get the impression that the vast majority in here see themselves as traders.  And also mrs bullmarket has already made it clear to me that I have spent far too much time in here today already   

So if you want to discuss further I'll be happy to discuss further on Tuesday as I better spend the day with mrs bullmarket tomorrow   

Have a nice evening and I'll pop back in on Tuesday.

cheers

bullmarket


----------



## It's Snake Pliskin (12 March 2006)

tech/a said:
			
		

> Glad you asked.
> 
> Optimisation has its place for singular entities,be it a stock,index,or future.
> Provided that the optimised variable is varified at and after each trade.
> ...




So it`s horses for courses is it Tech?  

The goal of optimisation is profitability. Therefore if one is to rely on the optimisation of a tested system, one needs to have confirmed their optimised system with data not used in backtesting and optimising the system. This data or sample can destroy an optimised system`s profitablility and render it unusable. 

So the moral is: don`t be greedy in backtesting and leave some for later. 

But what if future data which is actual trading, turns out to be different again and stuffs you right up? There is that chance. 
 :alien2:


----------



## lesm (12 March 2006)

tech/a said:
			
		

> Glad you asked.
> 
> Optimisation has its place for singular entities,be it a stock,index,or future.
> Provided that the optimised variable is varified at and after each trade.
> ...




Hi tech/a

Agree with your comments above.

Optimisation is one of those subject areas that can be argued ad infinitum.

Optimisation, when trading on a portfolio basis is a questionable practise, due to each stock having its own charactertics or be affected by different factors, whether they are in the same sector or in different sectors. At best all that you can really hope to achieve is some form of compromise or averaging, but never full/complete optimisation.

One other point tech/a mentions is related to what may be referred to as a 'system stop'. 

This is an often overlooked part of trading plans. The main focus is usually with respect to 'stops' related to trades taken using the plan, but not taking into account that the plan itself may go through periods where it is not effective in particular market conditions,hence consideration should be given to a 'system/plan' stop.


----------



## It's Snake Pliskin (12 March 2006)

bullmarket said:
			
		

> Hi again Snake
> 
> How all those parameters are determined obviously depends solely on the individual and will also be determined by what fundamental and/or technical experience and skills they posess and finally their risk tolerances.
> 
> ...




Thanks Bullmarket :sleeping:

Enjoy the time with the wife.:remybussi 

Snake


----------



## tech/a (12 March 2006)

Snake Pliskin said:
			
		

> So it`s horses for courses is it Tech?
> 
> The goal of optimisation is profitability. Therefore if one is to rely on the optimisation of a tested system, one needs to have confirmed their optimised system with data not used in backtesting and optimising the system. This data or sample can destroy an optimised system`s profitablility and render it unusable.
> 
> ...




Snake.
Id say the goal is more efficiency. I'm a bit lost as to why you would optimise variables on data not used in the backtesting.You would pretty well always get a different result. Optimisation is only accurate to NOW thats why I say re do it at every new entry assuming that each new entery comes after a close in a single entity,Future,index etc.

To ponder--the argument is that optimisation cannot be guaranteed and I agree particularly in portfolio trading. A selected variable is just that---the variables selected and used for T/T are nothing scientific they are just ones that were seen to be "logical" even when throwing in various different frequencies for those with variables the change to bottomline and Drawdowns Risk reward was/is negligible.
So my point is why are parameters in any method acceptable selected by "logic"?
Why are optimised parameters so unacceptable?

I'll answer that with *they are simply entries and exits* which in a succesful system/method provide a positive expectancy and a "Blue print" with which we can trade.
Are they the best possible---in a singular entity possibly----In a portfolio situation----No they are just numbers in a formula. Well thats what I have found.


----------



## happytrader (13 March 2006)

Snake Pliskin said:
			
		

> Read this article on paper trading
> http://www.actionforex.com/articles...r_trading_can_pump_you_up,_but_it's_not_real/
> 
> :alien2:




Hi and thanks Snake for a great site. 

I love those trading tips by Joe Ross especially his views on 'market opinions and 'final tips'

Cheers Happytrader


----------



## stevo (13 March 2006)

Duc
_"I have yet to see any P values generated by the software in regards to this statistical value."_

My stats is a little rusty, and certainly I will be eaten alive in this area but here goes.

What is the statistical hypothesis that you want to test? (Statistical hypothesis defined as a claim about a population that can be put to the test by drawing a random sample). 

I could compare a sample of backtest results with the population of backtest results (probably determined by using Monte Carlo) for the same system to see if it falls within the confidence interval but that doesn't prove anything.

What is the null hypothesis: the profitability of mechanical system A is no different from random trading? I am not too sure that this approach gets us far, or even that the p value is of much value as a statistical test in this area.

One use for p-Value could be to compare real trading results with the backtested results for the same system to see if they are statistically significant.

stevo

Say not "I have found the truth," but rather, "I have found a truth." 
_Kahlil Gibran_


----------



## It's Snake Pliskin (13 March 2006)

happytrader said:
			
		

> Hi and thanks Snake for a great site.
> 
> I love those trading tips by Joe Ross especially his views on 'market opinions and 'final tips'
> 
> Cheers Happytrader




No worries! 
I was googling and come across it.
Happyreading Happytrader :bekloppt:


----------



## It's Snake Pliskin (13 March 2006)

tech/a said:
			
		

> Snake.
> Id say the goal is more efficiency. I'm a bit lost as to why you would optimise variables on data not used in the backtesting.You would pretty well always get a different result. Optimisation is only accurate to NOW thats why I say re do it at every new entry assuming that each new entery comes after a close in a single entity,Future,index etc.




Tech,

I may not have been too clear with what I meant.  Basically, the data used for optimising and developing the system is for that purpose only. I was alluding that a fresh sample should be used to test the results of the system on after development. I`m referring to an out sample as some may call it. 

Yes, I agree with re-doing it.

Snake :aus:


----------



## ducati916 (13 March 2006)

*stevo* 



> It would be nice to have a definitive text on system testing but any I have seen are sadly lacking. Most people don't have the skills, knowledge and perseverance to carry out comprehensive testing of any sort of trading strategies. Many strategies touted are untestable - something the seller of the method probably appreciates.




This would seem to be the case.
Certainly it would seem to be the case that the methodologies are not statistical at all, but relying on finding a chart pattern that has worked in the past and proceeding on the assumption that it may work in the future.



> Fortunately we don't have to simulate trading data - we can use actual price and volume data and draw on thousands of stocks for our databases.




The first problem of course is that the time series for the data used is so short. TT I believe utilized 1996 as the earliest source of Price & Volume data.
This time frame is wholly inadequate, 10yrs, is just about, or a little over one business cycle, certainly within the ASX there has been no Bear market within the data, just the odd correction, and a secular Bull market.

How far back you would need to go is a moot point, but certainly to at least the mid 1960's, if not further. Of course there just is not the data available for entering into the software, or not cheap data anyway, and therefore 99% of users only use the bare minimum, and think that their results will be robust through the future.....................

How so?
If you have no data that pertains to the variety of Bear markets that exist, you cannot, and have not designed nor performed a thorough test, irrespective of the absence of statistical significance.

You are therefore in effect saying, I tested on conditions XYZ, and as long as XYZ remain, I should be ok, and thats fine, and I can accept that as a premise.

That however does not seem to be the case;



> Actually Duc, If you/anyone has a positive expectancy model and they trade it within the numbers generated by that model then yes I would guarantee that you would profit. Provided of course that sufficient testing had taken place.I think that the 6 mths testing in the case of T/T has proven to be sufficient. Of course what is sufficient can be argued ad infinitum. I'm not the first or last to come to that conclusion.Stevo has a weekly method that is doing as well as T/T. As has Andrew.






> Where Duc and I differ (In part at least) is the accuracy and sample testing of any method.
> I dont believe every concievable condition or variable need be met. I think and so far have proven (The future may prove me wrong),That provided you trade the Numbers of the tested positive expectancy method that have been generated during testing,and cease to trade it if actual trading figures fall outside those generated in testing---you'll make a profit within the parameters returned in testing.




The danger lies within the "start period". That is when you first start trading the methodology, particularly if utilizing leverage. That the conditions change from XYZ, to ABC, can cause much mischief.

Mechanical systems have gone from being a useful tool, for supercharging a paper-trading testing period, into a cult-mantra that proposes a holy grail result, based on wholly inadequate data.

Which leads into the "expectancy calculation"
If we examine the portion (#wins/#losses) first.

jog on
d998


----------



## ducati916 (13 March 2006)

*stevo* 



> What is the statistical hypothesis that you want to test? (Statistical hypothesis defined as a claim about a population that can be put to the test by drawing a random sample).




This in of itself is part and parcel of the problem. Only one set of criteria ever seem to be tested, and that constitutes the chart pattern under consideration, therefore, almost by definition, an incomplete study has been designed, using wholly inapplicable methodologies, that will have zero statistical significance.

Therefore, by examining what backtesting does extremely well, is probably the way forward, but with the caveat that it is not statistically based, or not in any readily admittable form

jog on
d998


----------



## ducati916 (13 March 2006)

*lesm* 



> Duc,
> 
> Always enjoy your posts, including the variation from informative to argumentative and sometimes baiting of TAs.
> 
> ...




I don't really have a problem with positive expectancy as a concept, more really with the way that it is calculated, inferring that there is a mathematical certainity somehow bestowed upon the result. But I'll go into greater detail.

jog on
d998


----------



## stevo (13 March 2006)

Duc
_"The first problem of course is that the time series for the data used is so short."_

Obviously this can be a problem for any sort of analysis. The amount of data available is "hindered" by the tendency to merger, acquisition and bankruptcy! 

If you look at the Small Ords index on the ASX you will find a wonderful bear market from 2000 to 2003 with some beautiful spikes down - *this index dropped by more than a third over the period*. We also had the Asian Crisis and a couple of other of reasonably volatile occurances over the last 10 years. So it is possible to design some strategies to utilise what data we do have rather than dismissing the approach altogether.

We could debate data adequacy forever, fundamental or technical. Most fundamental data available is also sadly lacking.* The least we can do is be aware of the weaknesses in whatever approach we adopt.*

With regards to statistical significance;
If we need to use something like p-Values, or any other measure, to test the significance of a strategy then the strategy is not worth trading. A strategy should jump off the page; we should try to destroy the strategy by excluding the best trades and using the worst entry/exit points. But, even after our best efforts it should still jump off the page. We are not looking for small differences.

There is quite a bit of research on traders and trading - have a look at the current research section at http://faculty.haas.berkeley.edu/odean/index.html for starters. Make sure that you run your mouse over Odean's picture.

stevo


----------



## ducati916 (13 March 2006)

*stevo* 



> Obviously this can be a problem for any sort of analysis. The amount of data available is "hindered" by the tendency to merger, acquisition and bankruptcy!




Recapitalizations [Stock buybacks, Stock Splits, Stock reverse splits, Conversions (Preferred, Bonds, Options)] will change the Price & Volume data significantly.



> If you look at the Small Ords index on the ASX you will find a wonderful bear market from 2000 to 2003 with some beautiful spikes down - this index dropped by more than a third over the period. We also had the Asian Crisis and a couple of other of reasonably volatile occurances over the last 10 years. So it is possible to design some strategies to utilise what data we do have rather than dismissing the approach altogether.




2000 - 2003 is not really a secular bear market, and will therefore present a very significant bias in any methodology that looks at longer term positions and results. 

30% only just qualifies as a bear market.
Until you hit an 80%+ decline I wouldn't want to place too much emphasis on the results that have been generated on backtesting, hell I had a 20% drop overnight last week in a position, volatility of only a 30% magnitude is just not looking at market reality.



> We could debate data adequacy forever, fundamental or technical. Most fundamental data available is also sadly lacking. The least we can do is be aware of the weaknesses in whatever approach we adopt.




Which was really the underlying point of the thread.
So many have seemingly swallowed the hook of expectancy, that I thought it was worth a look.

Nice link, right up my alley!!
jog on
d998


----------



## professor_frink (13 March 2006)

ducati916 said:
			
		

> *stevo*
> 
> 
> 
> ...




ducati, how well will did you go in the last secular bear market? For me(being 25), I've never been alive to try and trade one, so I can't comment on how I'd go. You sound like you've traded one before, so I'd be curious to hear how much more difficult it is to be profitable in a period like that.


----------



## stevo (13 March 2006)

Duc
As you state yourself biases are present regardless of the method. 

I quite like the paper on day traders in Taiwan - to quote from the abstract;

"Moreover, in the typical six month period, more than eight out of ten day traders lose money. Despite these bleak findings, there is strong evidence of persistent ability for a relatively small group of day traders.  Traders with strong past performance continue to earn strong returns."

So whilst the majority of day traders lose there is evidence that a very small minority consistently win. Averages tell a very average picture, but some traders appear to have an edge. Unfortunately we don't know what methods the exceptional day traders were using - and I doubt if they are going to tell us.

The last paragraph is great;
"Our analysis makes clear the need for comprehensive risk disclosure. Prospective day traders should be apprised of their likelihood of success: only two out of ten make money; fewer do so consistently."

stevo

Quotes from the paper;
_Do Individual Day Traders Make Money? Evidence from Taiwan with Brad Barber, Yi-Tsung Lee and Yu-Jane Liu._


----------



## kaveman (13 March 2006)

I ask myself this question whenever a thread gets really long when all it is is someone wanting to start an argument.          Why?
I guess being argumentative and confrontational is still a popular method of entertainment.


----------



## tech/a (13 March 2006)

kave there is some great stuff here and I dont think that it is arguementative other than the Professor looking for a bit of a stouch.

Duc.
Why is it necessary for a method designed to trade long/long trends be expected to perform as well in a bear market as a bull market?

Why is it that people like yourself seem to require a method designed to do only one thing---need to be "robust" in that it should trade ALL markets at ALL times?

From what youve written due to data problems both Technical and Fundamental your basically saying that its not possible to trade a plan giving meaningful numbers.

I dont agree my numbers have been meaningful for 4 yrs others who started 2 yrs ago have had them for 2 yrs and those that start today will have them meaningful "In their" method for X period.

Just like the beginning of ANY trade there needs to be a BEGINNING to every traded trading plan.


----------



## mit (13 March 2006)

stevo said:
			
		

> With regards to statistical significance;
> If we need to use something like p-Values, or any other measure, to test the significance of a strategy then the strategy is not worth trading. A strategy should jump off the page; we should try to destroy the strategy by excluding the best trades and using the worst entry/exit points. But, even after our best efforts it should still jump off the page. We are not looking for small differences.
> 
> stevo




Could have agreed more. All my successful systems have been like this. Maybe only one or two out of twenty or thirty I'll seriously look at each year will be like this. You tweak the parameters a bit but the system will be tradable across a wide range of values. 

I think that Duc is inventing problems that aren't significant. True there are dividends and mergers and takeovers but still all of the systems I have run over the years have been pretty true to the original backtesting for expectancy. 

As for the limited period of data. I agree that this is an issue. There has been no real bear market for the last couple of decades. However, I got daily Dow data back to the 1930s and had a look at what we can expect. It is true that a bear market can move down a significant percentage but what I found is that the actual down movement is relatively quick. From a couple of months in 1987 to at most a couple of years. Most seem to be less than 12 months. After this the market moves sideways. However, there is enough movement up and down that a short term swing trader could make money. Certainly any person who made money 2000-2003 would make money in the sideway bits.

So the trick is to survive the down bit with most of your money intact. I think even a longer term system should manage this as buy signals should dry up. Some people will make money playing the short side.

So I am pretty optimistic about my systems as well as being able to make money even during a secular bear market. The worst will be waiting for buy signals during the down bit.

MIT


----------



## stevo (13 March 2006)

Mit
I tend to design longer term systems with some measure of the general market - when the market is weak the systems stop taking on buy signals. A sudden drop is a lot harder to handle than a drop that gives some warning though.

Just for interest I developed a random entry and exit strategy and compared it with a simple weekly MACD on the current All Ords stocks for the last 7 years (back to 1999). If there is any validity in the MACD approach we should get a result significantly better than the random entry and exit approach. Chart below says it all. Compared to the random system the MACD jumps off the page - forget the stats. 

I am not suggesting that a simple MACD system is worth trading, just that it is certainly better than randomly buying and selling - certainly don't need a p-Value to tell us that.

stevo


----------



## professor_frink (13 March 2006)

tech/a said:
			
		

> kave there is some great stuff here and I dont think that it is arguementative other than the Professor looking for a bit of a stouch.




Was it that obvious  
ducati wants a period where prices decline 80% before it's a valid test- what a joke!
Why is it funny to me? If there is an 80% decline in stock prices, I personally won't care too much if my system breaks and sends me broke because we'll already be in the next great depression, there will probably be another world war coming soon, so I'll just enlist in the army to keep myself fed and clothed  
Or I'll just sit on the corner with all my professor buddies who will also be out of work and we'll busk for food stamps.
Although I do agree with you tech/a- there has been alot of good discussion on here- I just couldn't help myself with that one


----------



## ducati916 (14 March 2006)

*tech/a* 



> Duc.
> Why is it necessary for a method designed to trade long/long trends be expected to perform as well in a bear market as a bull market?Why is it that people like yourself seem to require a method designed to do only one thing---need to be "robust" in that it should trade ALL markets at ALL times?





A methodology that can return positive numbers in all and any conditions fulfills my criteria for wealth creation. Wealth creation is where I am coming from. 



> I dont agree my numbers have been meaningful for 4 yrs others who started 2 yrs ago have had them for 2 yrs and those that start today will have them meaningful "In their" method for X period.





If a methodology can only produce in a specific environment, you are immediately limited to finding those conditions, and accurately recognising them early enough to monetize a significant portion thereof.

The inflection points are where even the top guys struggle.
It is at the inflection points that instigating an entry of a *methodological system* that relies on specific market conditions is fraught with peril.



> From what youve written due to data problems both Technical and Fundamental your basically saying that its not possible to trade a plan giving meaningful numbers.




Not at all.
As you well know I have recommended TT to people.
Although I jibe you about bear markets, I actually think that it may work in a bear market, not quite as well, but you should get through.

My issue with the numbers, are several. They are essentially promulgated as statistically significant. Plainly this is nonsense.
Whats the big deal about statistical significance?
Statistical significance *is about probabilities, and is about a predivtive future result, within the defined test results.* 

The results generated by the software, would seem *not to be dealing in probabilities (as defined statistically) but dealing within a DETERMINISTIC context* this is a completely different paradigm.

It is negligent to suggest otherwise. Those new to trading, (and here I will generalise a little) are looking for the fast buck, are lazy, misinformed, and eager to snap-up anything that vaguely suggests easy money for no work.

Pumping in computer (market data) pushing a button and having the answer spew back out has great appeal. I did say I was generalising, but for every tech/a & stevo, you will get ten who fit my description.

*mit* 



> I think that Duc is inventing problems that aren't significant. True there are dividends and mergers and takeovers but still all of the systems I have run over the years have been pretty true to the original backtesting for expectancy.




To a certain extent it will depend on the market that you choose.
If the ASX, then there has been no significant bear market, and this will introduce a serious bias to any results. 



> As for the limited period of data. I agree that this is an issue. There has been no real bear market for the last couple of decades. However, I got daily Dow data back to the 1930s and had a look at what we can expect. It is true that a bear market can move down a significant percentage but what I found is that the actual down movement is relatively quick. From a couple of months in 1987 to at most a couple of years. Most seem to be less than 12 months. After this the market moves sideways. However, there is enough movement up and down that a short term swing trader could make money. Certainly any person who made money 2000-2003 would make money in the sideway bits.




Your data, what exactly does it consist of?
Please don't say a chart.

In the US, there have been the following secular bear markets
1901-1920
1929-1932
1937-1941
1966-1981
2000-current?

Part of the problem with Bear markets is the gross increase in volatility.
In Bull markets, volatility reduces, thus, trend following has greater success.
In a Bear, the increased volatility punishes stoploss based methodologies due to the whipsaw of volatility.

Unless your *testing* has survived high, protracted periods of volatility, then you are in for possibly a very rude awakening when a Bear market turns up.

jog on
d998


----------



## ducati916 (14 March 2006)

Actually this link is quite interesting as regards volatility in the US markets;

http://biz.yahoo.com/tm/060313/14038.html

jog on
d998


----------



## tech/a (14 March 2006)

ducati916 said:
			
		

> *tech/a*
> 
> 
> 
> ...


----------



## ducati916 (14 March 2006)

*tech/a* 



> AHHHH.Then Duc you should take a great deal of notice of the following.
> True wealth creation will come from those periods of outperformance,it certaintly wont come from those periods of average and underperformance.True wealth creation wont come from one source although it is possible for one source to outperform long enough for you to recognise the opportunity and create that wealth.




Or have a methodology that *outperforms in all market conditions* 



> In 1996 the plans for the Southern Expressway fell on my desk for tendering of the Reataining Walls.I knew from experience that areas needed a REASON to increase in value---better access was a proven one,




Now, did you consult a chart?
Did you wait for a breakout to new highs in the property prices?
Experience, always a vital ingredient.
A fundamental driver within the property market......being access, you recognized it, and acted upon it from an analysis of the information that became available to you.

This is a very common finding.
Property investment analysis tends towards the fundamentals far more than does Stock market investment analysis. There are several reasons for this, volatility, and the ability to analyze financial statements are but two.




> I'm selling houses now but is it necessary to be right all the time? Of course not you need only to be right ONCE if your able to recognise an opportunity and get some luck your way.(Initial capital helps a great deal)




For me, I like to be right all of the time.



> Its about minimising Loss,maximising Opportunity, and having the Kahunas to exploit that opportunity




Indeed.
Minimising loss is the key. That is exactly where I focus my attention.



> Duc while you attempt to find the statistically significant way to be 110% sure you wont lose your hard earned




I have a statistically significant methodology.



> opportunity looks you in the face and you dont see it.Nor do you know how to take advantage of opportunity without serious risk to your capital (You hate Stops)




On what evidence do you base this?
If it is simply an issue of *stoplosses* they are a requirement for technical based methodologies, as of course the whole concept of profitability is based on taking losses.

While it is possible that I may take a loss, based on a statistical probability, I will take ZERO losses, and return circa 30% compounded. That is the basis of the portfolio that I am currently running.



> The Technical approach (well mine) would mean that as of now I would be out of 4 of your selections at a 10% stop ,




Exactly.
And the stocks that replace them may also be stopped out, ad infinitum.
This is exactly the exercise in futility that I wish to avoid.



> I certaintly find I'm learning more about the Ducsta everyday.
> In all seriousness Duc and as constructive comment,I feel much of what you write are manifestations of your weaknesses.





Now you are venturing into my favorite area......psychology.
You must expand upon my manifested weaknesses, who knows you may in point of fact be quite correct.

jog on
d998


----------



## stevo (14 March 2006)

Duc
_
*"A methodology that can return positive numbers in all and any conditions fulfills my criteria for wealth creation. Wealth creation is where I am coming from."*_

Ah - so it's the Holy Grail that you seek! Or have you found it?

_*"For me, I like to be right all of the time."*_

Definitely - the Holy Grail has been found in Duc's backyard.  :dance: 
Nobody likes to admit that they get it wrong from time to time. But if the method is right then losing occasionally is not being wrong, it's just part of the process.

_*"Minimising loss is the key. That is exactly where I focus my attention."*_

If you are right all the time is there any need to minimise loss?  

_*"I have a statistically significant methodology."*_

Statistically significant on what basis?

regards

stevo


----------



## tech/a (14 March 2006)

> Now, did you consult a chart?
> Did you wait for a breakout to new highs in the property prices?
> Experience, always a vital ingredient.
> A fundamental driver within the property market......being access, you recognized it, and acted upon it from an analysis of the information that became available to you.
> ...




I actually looked to see if I could find one.(There is a ready made business poeple would pay for that sort of info graphically reported)

After 10 yrs of mediocrity blind freddy would have seen a monumental breakout from a chart of property had there been one.
not only that but a massive increase in volume and monthly increases in range and continuous new highs.
Now we would see consolidation and pullbacks in some areas
If ALL areas (postcodes)were represented by charts I'll bet that there would be a great deal more accuracy available for property buffs.



> You must expand upon my manifested weaknesses




I dont have to,youve written pages on the topic.


----------



## ducati916 (14 March 2006)

*tech/a & stevo* 

Guys, I am disappointed.
I had higher expectations. Looks like I can be wrong after all.



> Ah - so it's the Holy Grail that you seek! Or have you found it?




Found it.
*Arbitrage........100% profit...........0% losses = holy grail in my book* 



> Nobody likes to admit that they get it wrong from time to time. But if the method is right then losing occasionally is not being wrong, it's just part of the process.




True.
But that way doesn't interest me. I want my cake, and to enjoy each mouthful in addition, of course that's why I'm a fat bastard and need to jog so much.



> If you are right all the time is there any need to minimise loss?




There are no losses, *because that is where I focus my attention. I have yet to master achieving zero loss with zero effort* 



> Statistically significant on what basis?




On the basis that I have been outlining on this thread.



> I dont have to,youve written pages on the topic.




Precisely as I thought, mere opinion.
When it really boils down to providing some reasoned argument, supported with some evidence..................oh dear, it all heads south very quickly.
If the evidence is available, from my posted material it should be relatively easy to document my manifested weaknesses.

jog on
d998


----------



## tech/a (14 March 2006)

Duc.

Its pretty simple.

*The overwhelming need to be right * is the single thing that holds you back more than anything.

Accepting that you'll be wrong more often in life than correct means that being able to handle being wrong becomes more efficient.

I'm wrong all the time in business and personal life. But Ive made a decision and hence will be proven wrong or correct. Of 1000s of decisions about 10 have proven to be life altering. A few have nearly caused me bankruptcy---but let me assure you this that if you or anyone else faced bankruptcy a call to the "Duck" would be well worth the 50c---Being wrong was the best teacher I could ever wish for!

*You'll learn more being wrong (as opposed to AVOIDING being wrong)than being right.*


----------



## professor_frink (14 March 2006)

well said tech/a.
Not only do you learn more when you're wrong, you learn it very quickly too!


----------



## tech/a (14 March 2006)

professor_frink said:
			
		

> well said tech/a.
> Not only do you learn more when you're wrong, you learn it very quickly too!




If your in a Singapore,Bali jail then its never quick enough!
In cases like those I'm with DUC just dont ever ever get it that wrong!!!

Theres a difference in being Wrong and Stupid.
However many seem to have the ability to get them both RIGHT!


----------



## professor_frink (14 March 2006)

well that's a whole other level of stupidity. I won't even get started on that one


----------



## ducati916 (14 March 2006)

*tech/a* 



> Accepting that you'll be wrong more often in life than correct means that being able to handle being wrong becomes more efficient.




I see where you are coming from.
*Life* in general. I am referring to the markets.
So simply, yes I can agree that you must learn from life's mistakes, and by not committing any mistakes, you probably haven't lived. Somewhat akin to deciding to brake from 290kph @ the 50m marker, rather than the 75m marker

Now of course in that context, you are accusing me of having led a boring and sheltered life, and being spoonfed with a platinum set of cutlery.
I shall definitely visit you in Adelaide this winter, and you can decide for yourself.



> The overwhelming need to be right is the single thing that holds you back more than anything.




Here I disagree.
You see I am unwilling to accept at face value that there is not a better way, or that the way proposed is in point of fact not a total crock of s***e
Therefore I shall worry at a problem until I have solved it. In the process I invariably find things that I definitely didn't know, or understand previously.
You see if I had accepted the common concensus of the expectancy calculation, I would never have realised just how misleading it actually is.



> Theres a difference in being Wrong and Stupid.




Indeed there is.
It is when you are both, that time is ticking.

jog on
d998


----------



## It's Snake Pliskin (14 March 2006)

Duc,



> Guys, I am disappointed.
> I had higher expectations. Looks like I can be wrong after all.




Actually Stevo and Tech have had some really insightful stuff to say. 



> Found it. Arbitrage........100% profit...........0% losses = holy grail in my book




All forms of investing or trading are arbitrage of some sort - looking for  inefficiencies in the market.

Pure arbitrage has a window of opportunity to contend with. Are you always finding that window of opportunity? :headshake  Time is of the essence, as you would know.


----------



## ducati916 (14 March 2006)

*Snake* 



> All forms of investing or trading are arbitrage of some sort - looking for inefficiencies in the market.




Nonsense. As of course not all inefficiencies are risk free.



> Pure arbitrage has a window of opportunity to contend with. Are you always finding that window of opportunity?  Time is of the essence, as you would know.




No, not always.
Competition for risk free returns are after all keen.



> Actually Stevo and Tech have had some really insightful stuff to say.




Of course, these examples just weren't some of their better offerings however.

jog on
d998


----------



## It's Snake Pliskin (14 March 2006)

Duc,



> Found it. Arbitrage........100% profit...........0% losses = holy grail in my book




...and then..



> No, not always. Competition for risk free returns are after all keen.




So you haven`t found the HOLY GRAIL then.


----------



## ducati916 (14 March 2006)

*Snake* 



> So you haven`t found the HOLY GRAIL then.




Every now and then I encounter incisive thought, so penetrating, so profound, that it almost leaves me unable to respond in a manner that behooves the genius of the observation.

Now is one of those moments.
jog on
d998


----------



## wayneL (14 March 2006)

Snake Pliskin said:
			
		

> So you haven`t found the HOLY GRAIL then.




I once thought I had found it, but alas I didn't look carefully enough


----------



## It's Snake Pliskin (14 March 2006)

ducati916 said:
			
		

> *Snake*
> 
> Every now and then I encounter incisive thought, so penetrating, so profound, that it almost leaves me unable to respond in a manner that behooves the genius of the observation.
> 
> ...




The mission of arbitrage is to correct inefficiencies. People who do this pick apart strategies, concepts etc, not limited to trading/investing systems though, until they find an inefficiency or a series of inefficiencies. A bit like investing in undervalued stocks, or not?

But, inefficiencies are not always found. 

Therefore, move on to the next strategy, concept etc.etc. and start again. The window of opportunity is extremely important, because you are competing with other people doing the same, looking for inefficiencies. If they beat you to it you can miss out. 

So how can arbitrage be the holy grail if time is not an issue?  :alien2:


----------



## Shane Baker (14 March 2006)

Those interested in confidence intervals for their trading results may wish to look  at this software. 

http://www.adaptrade.com/sigtest.htm.

Amazingly this thread has a strong sense of deja vu (see Reefcap). 

The great thing about this software is that Duc will be able to arrive at his own confidence interval about whether he is speaking out of his "*fundamental*" orifice or not!


----------



## ducati916 (14 March 2006)

*Snake* 



> The mission of arbitrage is to correct inefficiencies. People who do this pick apart strategies, concepts etc, not limited to trading/investing systems though, until they find an inefficiency or a series of inefficiencies. A bit like investing in undervalued stocks, or not?




My apologies, I assumed that you knew what an arbitrage actually was.
It is a risk free trade taken between two securities of the same issuer, but priced to provide a spread.

The spread is the risk free return. By way of example;
A Preferred convertible is selling at 1000, convertible into 25 shares of common; thus the common is fairly valued at $40.00

If however, the common is selling at $43.98, there is a 9% spread.
By purchasing the Preferred at $1000.00 and selling short the common at $43.98 or above, you lock in a 9%+ risk free return in 1/day or 1,980% annualised. If you should leverage that say 10 times, and why would you not, that is a 90% risk free return/day



> But, inefficiencies are not always found.




Not every day.
But certainly enough to keep me interested.


> Therefore, move on to the next strategy, concept etc.etc. and start again. The window of opportunity is extremely important, because you are competing with other people doing the same, looking for inefficiencies. If they beat you to it you can miss out.




No, you keep a % of your cash available for this strategy, and have alternate strategies available to fill in the slower times.



> So how can arbitrage be the holy grail if time is not an issue?




Because you cannot lose money.
That's pretty cool in my book.
jog on
d998


----------



## tech/a (14 March 2006)

There are 3 issues,re arbitrage.

(1) Occurence.
(2) How long the inefficiency manifests itself.
(3) Highly correlated to (2) Finding one/them.

Even if they could be spotted once a year then $ x standing buy in an account would be risk free and your bank manager would see to it that his as well as all of your savings were available for immediate use.


----------



## markrmau (14 March 2006)

ducati916 said:
			
		

> My apologies, I assumed that you knew what an arbitrage actually was.
> It is a risk free trade taken between two securities of the same issuer, but priced to provide a spread.




That is a rather limited view on arbitrage. Arbitrage is not limited to mispricing between securities.

A few months ago, the SMH provided a coupon to get a free salad bar roll from Mcdonalds.

The SMH cost about $2 or what ever, and the roll is priced at $5 or so. So if you wanted the roll from Mcdonalds, a simple arbitrage deal was to buy the SMH. 

But the point is that it is still arbitrage. In a completely efficient market, all the SMH's would have been snapped up, and poor old Mcdonalds would have excessively long queues until the time value of the person lining up came to about $3


----------



## happytrader (14 March 2006)

ducati916 said:
			
		

> *Snake*
> 
> 
> 
> ...




Hi Duc

Do feel free to expand and educate us on your views and share with us the benefit of your experience. We are all ears and eyes here.

Thanks in advance

Cheers
Happytrader


----------



## stevo (14 March 2006)

Duc
Arbitrage opportunities, as you have mentioned before, don't come up all that often. Do I assume that these opportunities form a minor part of the strategies that you use?

You keep mentioning expectancy and I am not sure why - it's just another calculation. *What measures would you use to determine if a system has a reasonable chance of working going forward?* How did you determine that arbitrage was a goer?

Monte Carlo analysis is useful to investigate the possible outcomes of the same system. A single backtest run doesn't really tell me much but a 1000 simulations start to tell me things. I also don't test on individual stocks only - I always portfolio test 100's of stocks.

I plot monthly returns, yearly returns, drawdown, as well as the equity curve. I can then compare actual results achieved over the years with results that have been simulated and see if these results fall within the expected outcomes.

*I like graphical outcomes, and I am not talking about charts when I say this.*
A picture like the one below tells me a lot. *I ran 1000 Monte Carlo simulations on 3 systems - random buy and sell, standard weekly MACD and a system I have been working on - Rocket Science (RS). *

Just looking at the graphs you can see that random averaged around 4% compound annual return, MACD around 11% and RS around 28%. Don't get too hung up on the absolute values - look at the differences between the system. Using different position sizing strategies can have dramatic effects on returns.

What is more interesting is the range of possible results is quite tight for RS - around  25% to 31%. Both the MACD and random had enough variation in the outcomes to make them untradeable - regardless of the low returns.

Obviously there is more to system testing than one graph - I can do the same sort of graph for drawdown - the random system has horrible drawdown, as does MACD.

You can do the same simulations with buy and hold strategies - some wonderful results and some incredibly bad drawdowns. But it's the variation of possible outcomes that kills buy and hold style strategies for me.

*As I stated before if I have to do p-Values or t-tests to see the difference then the system is not worth trading.* But an equity curves and quarterly returns bar chart gives me a hint of the future if I traded a system.

stevo


----------



## tech/a (14 March 2006)

Stevo what are you using for your tables? (software)


----------



## stevo (14 March 2006)

tech
AmiBroker to generate the Monte Carlo data and Excel to get the frequency histograms, although if I want I can use Amibroker and TradeSim as well.

stevo


----------



## It's Snake Pliskin (15 March 2006)

> My apologies, I assumed that you knew what an arbitrage actually was.
> It is a risk free trade taken between two securities of the same issuer, but priced to provide a spread.



 :bs: 

Duc,

I was talking about arbitrage not "an arbitrage".

Perhaps you could go beyond your definition and explain why it is used by entrepreneurs all the time. It is not limited to securities, and is certainly rendered inneffective once the window of opportunity has passed. Maybe a text book can be quoted or something.....


----------



## Bobby (15 March 2006)

Snake Pliskin said:
			
		

> :bs:
> 
> Duc,
> 
> ...




Go Snake,
I'm not the only one sick of this humdrum- jog on humbug, hope you had a good laugh ! : 

Bob.


----------



## ducati916 (15 March 2006)

*stevo* 



> Duc
> Arbitrage opportunities, as you have mentioned before, don't come up all that often. Do I assume that these opportunities form a minor part of the strategies that you use?




No they don't.
I'll find five, six opportunities each day, but I will only get executed on average one deal per month. With enough leverage, the returns are well worth having for zero risk.



> You keep mentioning expectancy and I am not sure why - it's just another calculation. What measures would you use to determine if a system has a reasonable chance of working going forward? How did you determine that arbitrage was a goer?




I mention, or harp on about expectancy for the following reason.
The inputs to the calculation are derivatives.
Being derivatives, their adequacy is governed by the quality of their generation. ****e in ****e out. Therefore, if the statistical significance is non-existent, the expectancy calculation is worthless.

The major flaw with *monte carlo* is that the time period is too short.
If your time period is long enough, then, you should start to see statistical significance, and the derived calculations will have relevance.

I spoke to one systems tester, and he didn't even bother with monte carlo.
Yet the same calculations were used.



> Monte Carlo analysis is useful to investigate the possible outcomes of the same system. A single backtest run doesn't really tell me much but a 1000 simulations start to tell me things. I also don't test on individual stocks only - I always portfolio test 100's of stocks.




Agreed.
Monte carlo is definitely a step in the right direction.
I know that both you and tech/a utilize it.
TT only went (I believe from 1996) and that to my mind is just barely scraping a pass mark, if we say a *business cycle* is 4yrs to 7yrs.
It does not cover *market cycles* which are much more variable.
1996 to today in the ASX also does not include a secular bear market.
This I believe is important information that needs consideration within the testing parametres, if you wish to achieve true statistical significance, which will then provide you with a true expectancy or PROBABILITY.



> As I stated before if I have to do p-Values or t-tests to see the difference then the system is not worth trading. But an equity curves and quarterly returns bar chart gives me a hint of the future if I traded a system.




It will only give you a hint of the future if it is *statistically significant* 
If it is not, then you are defining a *trend* 
A trend is a direct prediction of the future, and must prove itself either correct, or incorrect, as it will either continue, or end. This is true because the paradigm that you are currently testing in is DETERMINISTIC.

*markrmau* 



> That is a rather limited view on arbitrage. Arbitrage is not limited to mispricing between securities.




Of course, but you wouldn't expect a thesis on arbitrage in the space of one post, any more than a tecchie could summarize TA in a single post.

*Snake* 



> I was talking about arbitrage not "an arbitrage".
> 
> Perhaps you could go beyond your definition and explain why it is used by entrepreneurs all the time. It is not limited to securities, and is certainly rendered inneffective once the window of opportunity has passed. Maybe a text book can be quoted or something.....




If you can find a quote directly attributable to myself, asserting, *entrepreneurs use it all the time* then I shall do so.
Of course, that will never happen, as I never said any such thing.
I was illustrating a *statistically significant methodology* something that obviously holds no interest for yourself.

In regards to its limitation to securities, you are correct, there are a number of arbitrage techniques. The opportunity lasts as long as the price inefficiency, how long is a piece of string?

*bob* 



> Go Snake,
> I'm not the only one sick of this humdrum- jog on humbug, hope you had a good laugh !
> 
> Bob.




Even the peanut gallery are jumping in. Obviously ignorance is bliss, and using a hammer for a screw insertion carries the appeal of speed, and seemingly identical results.

jog on
d998


----------



## tech/a (15 March 2006)

Duc

How do you determine a statistically significant amount of data?

Currently the discussion revolves around T/T and other longterm methods.
If you were a short term futures trader using 5 min tick data when is enough data enough data.

Fundamentally how long is enough fundamantal data?
How can you vouch for accuracy in the data presented in a balance sheet/s
and Profit and loss stataements? You cant tell me that everything presented fundamentally shows a complete picture,after all its presented to satisfy share holders and in the best possible light.

Again I'll ask the question--Why would you want a method to perform in an environment in which it is not desighned to perform well in?
Why would you trade a bullish method in a bear market?
Why would you trade a stock which has fundamentally great numbers when you knew that the market as a whole was going to be against it?
EG say a gold stock when Bank reserves are selling gold (as an example).

Whats stops you or anyone from suffering from analysis paralysis?
Yeh I know the 100% risk free mantra but in the real world while rarely possible not totally practical.


See Corporate and Social Responsibilities Thread on Reefcap.


----------



## Milk Man (15 March 2006)

tech/a said:
			
		

> Duc
> 
> How do you determine a statistically significant amount of data?
> 
> ...




Reminds me of that post you stated was "the best ive seen on any forum". Basically says only to trade long in bulls and be careful in sideways, but either stay out or go short in bears. Some form of portfolio stop could be utilized but it may prove to be "statistically insignificant"; just like my bank a/c. esok:


----------



## tech/a (15 March 2006)

Milk.

This is a tact I am currently looking into.But more on that later.


----------



## stevo (15 March 2006)

Duc


> TT only went (I believe from 1996) and that to my mind is just barely scraping a pass mark, if we say a *business cycle* is 4yrs to 7yrs.
> It does not cover *market cycles* which are much more variable.




10 years is actually pretty good, but obviously doesn't include all past market activity. If you test 500 stocks over 10 years you get roughly 1.25 million trading days worth of data. But the more the better. I personally don't trust data back more than about ten years, and even then it is getting a little hairy. It's better not to test than to test on bad data.

I like to think of all the data for all the stocks joined end to end as a single stock - *10 years of data on 500 stocks is the equivalent of 5000 years worth of data for one stock.* I am sure that you will point out the error in this thought but it does highlight the value of portfolio testing versus single stock back-testing for a given set of criteria.

I have been looking for price data out to 2020 but nobody seems to have it! If anyone knows a supplier I would be prepared to pay substantial sums for it - although not my soul.

stevo


----------



## ducati916 (15 March 2006)

*tech/a* 



> How do you determine a statistically significant amount of data?




From pg2



> Sample Size
> One crucial prerequisite prior to embarking upon the *testing* or study is the requirement to perform a sample size, or POWER CALCULATION In the words of D. Altman "a trial should be big enough to have a high chance of detecting, as a statistically significant, a worthwhile effect if it exists, and thus to be reasonably sure that no benefit exists if it is not found in the trial."




As regards duration, common sense is enough of a guide to determine the appropriate time period to be used

TT was designed as a long term methodology, therefore long term duration of data should be utilized.

Intrestingly, you may if you subscribe to the *fractal theory of financial markets* utilize 1mins charts for your long term TT methodology, and you would starting from 1996 - 2005 have far more data available.

As for daytrading who knows, who cares.
Obviously daytraders.
I would suspect in actuality they need the same amount of data as TT.
What you are looking for is statistical significance, the *law of large numbers* rules within statistical methodologies.



> Fundamentally how long is enough fundamantal data?




13 weeks, or Q1.



> How can you vouch for accuracy in the data presented in a balance sheet/s
> and Profit and loss stataements? You cant tell me that everything presented fundamentally shows a complete picture,after all its presented to satisfy share holders and in the best possible light.




They would never seek to mislead, or lie to me.
The data is 100% reliable.



> Again I'll ask the question--Why would you want a method to perform in an environment in which it is not desighned to perform well in?
> Why would you trade a bullish method in a bear market?




Statistically significant methodologies are robust methodologies.
Therefore, if it performs in all markets, under all conditions, even adverse ones, this is a methodology I can back with cash.



> Why would you trade a stock which has fundamentally great numbers when you knew that the market as a whole was going to be against it?




Because that is when your risk is at its lowest, and your potential returns are at their highest.



> Whats stops you or anyone from suffering from analysis paralysis?




An interesting question.
First and foremost, there must be a differentiation of quantitative methods from qualitative methods. The quantitative methods must be consistent, comparable, show causation, correlation and be based or founded in the simplest calculations possible. The qualitative methods must support the quantitative on a common sense basis, and again be based on simple logical arguments, that are easily understood by the layman.

Increasing use of esoteric methodologies, higher math, quantum math, elaborate paradigms, etc will almost certainly lead to a mental meltdown, and analysis paralysis, or extreme emotional bias.

*stevo* 



> 10 years is actually pretty good, but obviously doesn't include all past market activity. If you test 500 stocks over 10 years you get roughly 1.25 million trading days worth of data. But the more the better. I personally don't trust data back more than about ten years, and even then it is getting a little hairy. It's better not to test than to test on bad data.




Agreed.
If the data is corrupt, the output must by definition be corrupt, and worthless



> I like to think of all the data for all the stocks joined end to end as a single stock - 10 years of data on 500 stocks is the equivalent of 5000 years worth of data for one stock. I am sure that you will point out the error in this thought but it does highlight the value of portfolio testing versus single stock back-testing for a given set of criteria.




This would be fine, if, all the variables within individual stocks were consistent. That they are not, must therefore invalidate this convenient theory.


> I have been looking for price data out to 2020 but nobody seems to have it! If anyone knows a supplier I would be prepared to pay substantial sums for it - although not my soul.




I of course have it.
But if I send it, you gotta keep it under your hat.

jog on
d998


----------



## It's Snake Pliskin (15 March 2006)

> If you can find a quote directly attributable to myself, asserting, entrepreneurs use it all the time then I shall do so.
> Of course, that will never happen, as I never said any such thing.
> I was illustrating a statistically significant methodology something that obviously holds no interest for yourself.
> 
> In regards to its limitation to securities, you are correct, there are a number of arbitrage techniques. The opportunity lasts as long as the price inefficiency, how long is a piece of string?




It was a rhetorical question, one you were obviously not able to detect.:screwy:

Good luck with your grail and watch that window of opportunity!


----------



## tech/a (15 March 2006)

> The quantitative methods must be consistent, comparable, show causation, correlation and be based or founded in the simplest calculations possible. The qualitative methods must support the quantitative on a common sense basis, and again be based on simple logical arguments, that are easily understood by the layman.




While open to arguement,I think thats is exactly what and where the likes of myself and Stevo are at.


----------



## stevo (16 March 2006)

> What you are looking for is statistical significance, the law of large numbers rules within statistical methodologies.




Continuing down the path of disaggregating data (love that word - disaggregating)  more robust systems would be developed using daily data over agreggated weekly data. In practice this does not seem to hold up, although I am sure many would disagree with me.

The concept that the longer term the system the more data required has steered me away from designing systems based on a monthly or quarterly time frames. The large number of trades in a backtest can give some comfort, but it is possible to generate more trades without increasing statistical significance. 

stevo


----------

