Backtested Systems, Lies, Damned Lies, & Statistics

[happytrader]

Well researched, well backtested and then tested on a small scale in the real market with real money is a low risk way to start. Think of this as your tuition fee as you gain insight into how you behave.

I am always amazed at how many people blatantly don't read or follow directions and instructions unless they are penalised in some say. Its like some passive aggressive behaviour where they are compelled to do the opposite. Just because most market participants are adults doesn't mean they have out grown the behaviour. Chances are if you react this way on being 'told what what to do' by an authority figure which looks to me how most people interpret instructions you may have difficulty following through on your own authority which is what you are doing when you conceive a plan 'to tell you what to do and when to do it' I'm just guessing by the way.

Watched 'The Naked Chef' Jamie Oliver selecting would be apprentices. Point to remember here is they would be given the opportunity to rub shoulders with and learn from a very famous and successful chef. All he was really looking for is someone to do what they were told.They were given very specific instructions on how to prepare, cook and present a particular dish. Most had memory lapses about the sequence. Another was impatient did their own thing with the timing and temperature. Another completely overrode all instructions. Another didn't trust the cooking time and second guessed it. Another did okay till it came to presentation however he decided to push his own creative bent. Out of half a dozen hopefuls, only one made a concerted effort to do 'just so'

Cheers
Happytrader

 

[stevo]

Just read the bits in bold to get the short version of this post

Duc
You stated that "Forward testing has one major flaw, and that is it is time consuming."

A forward test can be conducted just as quickly as a back test. It is not necessary to trade live to forward test.

There are probably a lot of ways to forward test. One method is to develop a strategy based on 5 years of data (and lets assume that we have accurate complete data for the market in question) . The strategy is then tested going forward. So we could test from 1998 to 2003, then forward test from 2003 to 2006.

Obviously their are lots of permutations on forward testing. You could also develop the strategy on data from 2002 to 2006, then backtest it from 1999 to 2004. The important thing is that the data that the "forward" test is carried out on is different to the data that the system is developed with.

Because large databases can be used for system testing and the database can be split up in all sorts of ways it is possible to develop robust trading systems going forward. If a medical trial (and I know little of these things) is big enough then it is possible to gain some level of confidence in the results.

I am sure that the large drug companies would love to be able to simulate medical tests on a computer but the human body is quite complex to simulate. Forward testing by drug companies is littered with unfortunate failures, but that doesn't mean we shouldn't test drugs.

Fortunately we don't have to simulate trading data - we can use actual price and volume data and draw on thousands of stocks for our databases.

It would be nice to have a definitive text on system testing but any I have seen are sadly lacking. Most people don't have the skills, knowledge and perseverance to carry out comprehensive testing of any sort of trading strategies. Many strategies touted are untestable - something the seller of the method probably appreciates.

You cannot prove that system testing doesn't work and is totally useless, you can just state that you haven't found any evidence of it working from the limited research that you have done.

regards

stevo

ALSO - don't even mention optimisation! The debate will never end.

 

[It's Snake Pliskin]

Hi ducati

I think it is very important to paper trade a plan for however long it takes (and that will vary according to each individual) until the person is satisfied the plan is generating the objectives/returns specified in the plan before committing funds.

cheers

bullmarket

Read this article on paper trading
http://www.actionforex.com/articles...r_trading_can_pump_you_up,_but_it's_not_real/

:alien2:

 

[tech/a]

Most people have no alternative.

They develope a plan and wish to see how it goes before committing funds to it.
This is where Duc and his accuracy in sample test has credence.You can paper trade all day everyday and record every detail but wont have anything meaningful for years.
But if you could in the end youd have a set of Numbers which tell you if your profitable of not. 90%+ of the time after years the answer will be not profitable.

Fundamentalists tend to succeed simply by buying and holding.
Try trading short term Fundamentally!
Technical methods developed around longterm trading where % won far exceeds %'s lost are also simpler to develope.

While short term trading methods can be designed and can return good profits I personally havent tested one "Plan" which results in consistant profit.I'm yet to see one based on a portfolio style of stock trading.

My personal feeling is that Futures methods and singular index and or stock trading short term have more chance of success than Attemping to portfolio trade short term.Developing methods with greater % wins and reasonable Reward to Risk ratios are harder in my veiw.Than less wins higher Reward to Risk.

Most plans are based upon short term methods as result can be seen earlier,in terms of days or weeks rather than Months and Years.
Spectacular gains are confused with consistancy.
Impatience at a possible 200% gain far outweighs the patience of attaining consistancy in return.

With excellent software like Amibroker available for under $500 I cant understand why people dont take the time to become proficient so that they can test their plans and find one that works and simply trade it within its "Numbers" from the "Blueprint" generated.

Forget the paper step into the 21st century!!

 

[bullmarket]

Morning everyone or to whoever is reading this

I don't believe it has to take years before you can have any meaningful data from paper trading, though for some depending on their strategy I suppose it could.

I know a hand full of people who started out paper trading quite a few years ago before commiting real funds and who are still profitable now. At the time they paper traded for anywhere from 6 to 12 months and from memory made on average a few 100 hundred trades before they were convinced the results from their paper trading were good and reliable enough for them to start trading using real funds.......but as I said in earlier posts, it's horses for courses and what suits and works for each individual.

The obvious downside of paper trading is that you are not under the same emotional stress (fear and greed) as you might be when trading with real funds and so the hard bit is to continue to allow the criteria and parameters in your trading plan to drive buy/sell decisions and not let emotions interfere after starting to trade with real funds.

Now some people will obviously be able to resist letting emotions influence buy/sell decisions during real trading better than others - that is just naturally occuring differences in peoples' personalities, temperament and general makeup etc.

Personally, based on my experiences over the years, I still believe paper trading is one good way to test and fine tune a trading plan before commiting real funds

cheers

bullmarket

 

[tech/a]

Bulldust from someone who has never traded and who has never developed a method which is used for trading,who has never been involved in systems testing or developement,your parrot like "Each to their own" "Horses for courses" "What suits the individual" is getting as boring as hell.

If thats the case might as well not comment Bulldust as really everything is
"Horses for courses,each to their own,what suits the individual."

Get off the fence

Put some meaningful content into your posts if your going to comment.
A few hundered trades as a basis for meaningful results is like saying I can fly a kite so now I'll hop in a Jumbo and fly that just as well as the kite.

T/T had 50,000 (About 30 different times) portfolios tested all with 120 trades in each,Thats 6 million trades and according to Duc even thats hardly meaningful.
Its also been tested by 100s of others over 4 yrs so there would be millions and Millions of trades. (I'm only using T/T as an example as its the only one I know of made Public),It was developed purely for discussions just like this.
And I welcome people pulling it to bits.
After all I trade it with real $$s and if there are 100s of people having critical input ,Thats far better than 1 person--me constantly looking at it.

If you dont know Bulldust you dont have to comment its not compulsory.

And before you wipe this Wayne my opinion is just as valid as Bulldusts,when it comes to input.

 

[bullmarket]

no problem tech/a

You're clearly upset and frustrated

I am just calling things as I see them with my supporting reason and will continue to do so here or anywhere else I see fit with no consideration whatsoever for what you think of my posts

If you disagree with anything I say that is fine....I don't have a problem with that because I don't see how what you think of my posts can be of any interest let alone of any consequence to me.

If my posts annoy you so much, then why not put me on your ignore list......seems like a quick and easy solution to release your frustrations

see you in the soup

bullmarket

 

[stevo]

Why paper trade if you can trade for real?

If someone isn't sure of their strategies they could paper trade. I would need to paper trade for about 3 years before I could say that a method looks like it might work - an impractical approach to testing for many. Shorter term traders would need to get at least 100 trades under the belt - 100 trades would take me 4 years.

So paper trading is impractical for longer term methods and also quite frustrating. Imagine paper trading a successful system for 4 years and knowing how much money you have forgone. "If only I had put real money down!"

stevo

 

[bullmarket]

Hi steve

Why paper trade if you can trade for real?

If someone isn't sure of their strategies they could paper trade. I would need to paper trade for about 3 years before I could say that a method looks like it might work - an impractical approach to testing for many. Shorter term traders would need to get at least 100 trades under the belt - 100 trades would take me 4 years.

So paper trading is impractical for longer term methods and also quite frustrating. Imagine paper trading a successful system for 4 years and knowing how much money you have forgone. "If only I had put real money down!"

stevo

yes I agree in general with you, but I am not suggesting paper trading is suitable or will be beneficial to all.

The friends I was referring to made at least a few hundred paper trades in 6 - 12 months roughly before they were satisfied their strategy was likely to be profitable. For them, paper trading was beneficial.

Imo paper trading is just one option a trader can use when starting out to test their strategies before commiting real funds. I'm not suggesting at all that everyone will gain useful information from paper trading - obviously some will, some won't.

cheers

bullmarket

 

[ducati916]

I see that the discussion is fluctuating a little, and that we are starting to discuss some rather diffuse variables.

Just to bring it back into focus a little.
The object is to really ascertain whether the software that is available on the market today is really utilizing statistical methodology to generate the results, and if they are, is the design quality of a high enough standard that you can actually claim a genuine statistical significance to the result.

If, however the answer is a negative in regards to the statistical significance, is there any benefit to the utilization of the testing methodology that is undertaken.

To my mind the answer to question one, is that the statistical quality is too low to qualify as a statistically significant result and therefore this would be potentially misleading to the average consumer purchasing this software.

But as stevo has joined the discussion, and I believe that he trades a successful mechanical system, I shall address the following questions to both tech/a & stevo

Because large databases can be used for system testing and the database can be split up in all sorts of ways it is possible to develop robust trading systems going forward. If a medical trial (and I know little of these things) is big enough then it is possible to gain some level of confidence in the results.

Introduced is the concept of confidence
This has a great deal of statistical significance. Confidence is correlated to probability

One of the first values a student learns to calculate is the P value - that is the probability that any particular outcome would have arisen by chance. Standard scientific practice which is entirely arbitrary, usually deems a P value of less than 1 in 20 (expessed as P<0.05) as statistically significant and a P value of less than 1 in 100 (P <0.01) as statistically highly significant.

Therefore, if multiple outcomes are to be analyzed from the data set, you must make a correction to try and allow for this (usually achieved by the Bonferoni method)

A result in the statistically significant range (P<0.05, P<0.01) suggests that the null hypothesis should be rejected viz. the methodology has revealed a statistically relevant, and tradeable strategy.

However, a P value in the non-significant range tells you either that, there is no difference, or, there were too few subjects to demonstrate the difference if in point of fact it existed.

I have yet to see any P values generated by the software in regards to this statistical value. An absence, reinforces the poor design quality of the statistical methodology.

Which then brings us to the point that stevo has raised, viz. "Confidence levels"

A confidence interval (that can be run on t tests, r values, sensitivity, specificity etc) allows you to estimate for both positive trials (those that show a statistically significant difference between the two arms of the trial) and "negative" ones (those that seem to show no difference)

Whether the the strength of the evidence is strong or weak, and whether the study is definitive

If you were to repeat the same trial hundreds of times, you would not get the exact same result each time, but on aggregate, you would establish a particular level of difference, or lack thereof.

This calculation is the confidence interval and should read something along the lines of; The 95% confidence interval around this difference is -1.2% to +12%

In this example, as zero level is overlapped, therefore we have a dichotomy, and would classify the trial as a negative, however, in reality, there "probably is a real difference, and it probably lies closer to 5% than either -1.2%, or +12%

In all the systems results that have been generated and posted, I have never seen P values nor Confidence Intervals posted as part of the computer software generated results.

Therefore, again, the quality of the design methodology is just not of a high enough quality to derive a statistically significant result.

Combine that with all the additional flaws, data mining, optimization, hidden biases, poor selection of testing criteria, you are really invalidating any semblence of credibility if putting forward a "statistical" argument for the testing that is performed.

What we have left I believe is option #2, which is a glorified supercharged way of testing trading plans

The sexiness of the Automated trading plan tester really isn't as marketable as the current, mathematically jazzed up version that is marketed to traders.

A forward test can be conducted just as quickly as a back test. It is not necessary to trade live to forward test.

This is a classic.
Before I lump it in with the hated expectancy calculation, would you care to expand on this, in regards to the logical basis on which a future prediction is predicated?

You cannot prove that system testing doesn't work and is totally useless, you can just state that you haven't found any evidence of it working from the limited research that you have done.

No indeed, quite the opposite, and all the more dangerous for this wrinkle;
that it does work, either, some of the time, or all of the time, is the danger, you will never know which one will pop up.

tech/a has stated that his TT system is currently "outperforming"
That in my book qualifies as a "failure of the system" as, should it be "underperforming" it would be pulled, due to the exceeding of drawdown, and results falling outside of "expectation"

That the results are positive, seemingly do not invalidate the methodology.
No need to ask why.
Psychologically, which should be the primary area to benefit from going mechanical, the guidelines have been scrapped. Would the same guidelines in a negative scenario, also be thrown out?

There are just so many variables raised, but I shall limit this post here and see how these points are addressed.

jog on
d998

 

[tech/a]

Duc.
I'm now out of the discussion.
You simply wont look at what I have said.You'll ignore it and plough head long into argument regardless of opinion held by those who are happy with their results.
On T/T results,"Outperforming"

When tested using Montecarlo analysis of the 50000 portfolios tested
there were NO portfolios that were not profitable.
Of the mean there were some below the mean and some above the mean.
Outperformance of T/T that I am trading is at the upper end of those above the mean. So its within the numbers and wont be pulled. Guidelines are well in place and the method is well within them.

I'll say it again.
If someone has a set of numbers from 5 tests which are giving a positive expectancy and they trade within the numbers generated then they can expect and will get a profit.If it goes outside the tested numbers then failure is possible.
Now turn that to 10 Million Tested Trades same thing.--Knock yourself out with arguement---I conceed that you can and will be able to prove statistical "Insignificance" you could do that with any example.
Doesnt matter While your arguing I'm trading until my numbers are proven to be incorrect.

Bulldust.
Its not what you say its what you dont say!! Read your posts there is NO content. Nothing presented, argued, given up as example.
Wouldnt mind if you had something to add.
You'd frustrate a room full of Buddhist Monks.

Chances are you "Used to be indecisive but now you're not so sure."

Have a lovely afternoon.

 

[bullmarket]

no problem tech/a

Bulldust.
Its not what you say its what you dont say!! Read your posts there is NO content. Nothing presented, argued, given up as example.
Wouldnt mind if you had something to add.
You'd frustrate a room full of Buddhist Monks.

Chances are you "Used to be indecisive but now you're not so sure."

Have a lovely afternoon.

As I said earlier, you are clearly upset and frustrated and your resorting to petty name calling against other chatters when they disagree with you confirms, for me at least, your immaturity and frustration and reflects poorly on your character and integrity.

But each to their own.......I'll just continue to post what I like when I like, as everyone is entitled to do, and if my posts annoy you simply put me on your ignore list as I suggested before

cheers

bullmarket

 

[lesm]

Duc,

Always enjoy your posts, including the variation from informative to argumentative and sometimes baiting of TAs.

At times, I wonder if you are attempting to over analyse/rationalise the approach to the development or the application of trading systems.

You seem to have difficulty with the term of 'positive expectancy' would the probability of a 'positive outcome' or 'positive return on investment' be more palatable?

 

[happytrader]

Duc,

Always enjoy your posts, including the variation from informative to argumentative and sometimes baiting of TAs.

At times, I wonder if you are attempting to over analyse/rationalise the approach to the development or the application of trading systems.

You seem to have difficulty with the term of 'positive expectancy' would the probability of a 'positive outcome' or 'positive return on investment' be more palatable?

Hi lesm totally agree with your post.

Even a 99% positive expectancy trading system would be a wasted effort on anyone who lacked the courage to do the Nike thing and 'Just do it'

I notice no one ever specifically mentions those most necessary of all trading traits called courage and obedience.

Cheers
Happytrader

 

[It's Snake Pliskin]

Hi steve
yes I agree in general with you, but I am not suggesting paper trading is suitable or will be beneficial to all.

Imo paper trading is just one option a trader can use when starting out to test their strategies before commiting real funds. I'm not suggesting at all that everyone will gain useful information from paper trading - obviously some will, some won't.

cheers

bullmarket

...a bit of fence leaning here. :dunno:

 

[It's Snake Pliskin]

ALSO - don't even mention optimisation! The debate will never end.

"optimisation"...

 

[tech/a]

Glad you asked.

Optimisation has its place for singular entities,be it a stock,index,or future.
Provided that the optimised variable is varified at and after each trade.

To optimise a set of variables and then apply that optimisation to a portfolio or group is plain crazy as the variables within the group basically render the optimisation impractical.

Again of course running an optimised methodology over an entity will give a set of Numbers---your Blueprint---so while trading is within the blueprint all OK when outside STOP---.

Thats my veiw but of course but its up to the individual,what suits some may not suit others,its subject to various points of veiw and interpretation,it could be right and could be wrong,I knew a taxi driver who once optimised his rides,he's still a taxi driver today so it must have worked for him.
Can I phone a friend??

Thats should get them going.

 

[It's Snake Pliskin]

...a bit of fence leaning here. :dunno:

...generalising or committed thought?

And finally, all of what I mention above imo should be included in someone's trading plan as part of their portfolio/trading performance monitoring process....and hence the importance imo of paper trading a plan for however long it takes prior to commiting funds so that your trading performance monitoring process can show you if your trading strategies are generating the returns you expected.

 

[It's Snake Pliskin]

Bullmarket,

How is all of this determined?

Imo a sound plan should include but not be limited to things like:

1) Your objectives - long and/or short term

2) What markets and types of securities you will trade/invest in - shares, bonds, options, hybrids, fixed interest etc etc

3) How you will search/scan for investment/trading candidates - technical/fundamental or other criteria.

4) How you will determine entry/exit points - technical/fundamental criteria or a combination of both

5) How you will determine the position size for each investment/trade.

6) How you will manage risk - diversification, stop losses etc etc

Now how complicated or simple someone makes their plan is solely up to them - but imo a key purpose of a plan is to remove emotion as much as possible from decision making by making the plan as comprehensive as required to suit ones objectives and circumstances.

 

[bullmarket]

Hi Snake

...generalising or committed thought?

It's a committed thought

cheers

bullmarket