Backtest your way to financial security

After reading a post in another thread, I thought I would start this one. The title is not sarcasm by the way.

MichaelD said:

I'm afraid I'll have to steadfastly disagree with this.

I agree on one point - that it is very hard or impossible to backtest or prospectively test discretionary strategies with software.

I vehemently disagree, however, that it is impossible to test these strategies. Indeed, I'd argue even more strongly that such testing is ESSENTIAL for shorter term strategies.

Right now, your strategy is extremely profitable. Fantastic! (No sarcasm or envy intended). But...is this because of skill or because of current market conditions? How will you know the difference? How will it perform in different market conditions?

You know these answers for long term trend following systems, and spent a long time working them out, so why all of a sudden do you not need to know these same answers for your short term system? It's ironic, don't you think, that 3 months ago we were on opposite sides of this argument. What changed?

I'm doing things a bit differently. I essentially have 3 short term plan methodologies, all of which attempt to achieve the same thing. The entry signals are the same. I'm keeping records of how all 3 trade going forwards (systems 2 and 3 have real money - system 1 has been superceded but I'm still tracking it). I can look back at market conditions and how all 3 systems traded these conditions. It's very, very time consuming and tedious, but the knowledge available from this data is well worth the effort, particularly in the ability to go back and try a new idea out on existing data. It also means it is going to be more and more obvious as time goes by when a system is no longer performing as expected.

Is it simply a different road to the same end point? Maybe.

Click to expand...

Michael,

I have some questions:

Why do people backtest?
What constitutes a successful backtest leading to a blueprint?
What biases are there to be aware of?
What issues are there with equipment and software deficiencies?
Why can every aspect of human nature be backtested?
What is one thing that corrupts all backtesting, and what same thing is relevant for trading without backtesting?

Answers to these questions would be greatly appreciated. Others please feel free to comment.

Thanks
Snake

Re: Backtest your way to financial security.

It's Snake Pliskin said:

After reading a post in another thread, I thought I would start this one. The title is not sarcasm by the way.



Michael,

I have some questions:

Why do people backtest?
What constitutes a successful backtest leading to a blueprint?
What biases are there to be aware of?
What issues are there with equipment and software deficiencies?
Why can every aspect of human nature be backtested?
What is one thing that corrupts all backtesting, and what same thing is relevant for trading without backtesting?

Answers to these questions would be greatly appreciated. Others please feel free to comment.

Thanks
Snake

Click to expand...

Greetings Snake,

Backtesting is just that! Testing the past ~ Many have done the testing to form their trading plans for the future.

I DON'T.
Will be interesting to see what answers you get.

Regards Bob.

Re: Backtest your way to financial security.

A very interesting topic indeed.

It's Snake Pliskin said:

Why do people backtest?

Click to expand...

To work out whether a particular strategy has ever worked in the past. If it hasn't ever worked in the past, it is less likely to work now.

It's Snake Pliskin said:

What constitutes a successful backtest leading to a blueprint?

Click to expand...

The answer for me is acceptable profitability, acceptable drawdown and acceptable trading characteristics in all test sequences and test timeframes (bear, bull, choppy market and a few others).

It's Snake Pliskin said:

What biases are there to be aware of?

Click to expand...

You could write a book about these, but in the spirit of Socratic discussion, here are some of the more obvious ones;
1. Inclusion bias - testing the strategy on stocks which are now part of the test universe but wouldn't have been at the time the backtest trade is taken.
2. Survivor bias - not including delisted stocks in the test thus positively influencing the results.
3. Hindsight bias - you can see the right side of the chart if manually backtesting which may bias your decision to exit.

It's Snake Pliskin said:

What issues are there with equipment and software deficiencies?

Click to expand...

It's TIME CONSUMING to write and run multiple backtests. There's also quite a steep learning curve involved in learning to drive the software for mechanical backtesting. It's even more time consuming to backtest discretionary trading.

It's Snake Pliskin said:

Why can every aspect of human nature be backtested?

Click to expand...

I'll presume you mean can't, not can. Backtesting is mechanical. It's human nature to want to "second guess" the machine when actually trading if choosing between entry signals. It's human nature to not want to take a stop when your system says exit because you "know better".

It's Snake Pliskin said:

What is one thing that corrupts all backtesting, and what same thing is relevant for trading without backtesting?

Click to expand...

You've got me on this one. I'm going to go for human emotion as the answer.

Bobby said:

I DON'T .

Click to expand...

Why not? (I'm interested as it contrasts to my current approach to the market which is to backtest everything before I trade it.)

I need to spend time answering this,Just dont have that time right now.

However even though on the other thread (my post below.) appears to be contradictory to Michaels veiw it certainly isnt.

Michael.

I certainly used to have your view.
That everything should be tested and results tabulated to a "blueprint" from which to work.
If there was one piece of advice I would give to a new comer it would be to learn how to trade long term profitably first before attempting to trade shorter timeframes.
Further I would certainly tell them to learn how to develop profitable trading methodologies through backtesting.

BUT.

Trading shorter term is a completely different ball game. Discretionary variables cannot be successfullly tested with the software I know of.
But I can without reservation tell you that my experience is that when trading short term you'll satisfy all those "Numbers" if you get on momentum and ride it as long as you can and then get off it!

Sitting in a stagnant stock not only means no profit till it moves BUT ALSO opportunity cost.

So I'll have to disagree with both you and Snake on this one.


Quote:
Pronouncements one way or the other without solid data behind them are meaningless.



To you!
They used to also be to me as well.

Click to expand...

I'll re phrase part in that I dont have the expertise to backtest what I do trading short term.If I could I would.I trade in many timeframes in the one move.I dont have software that can test this.
Experience does play a huge part and that experience has been gained from many hrs of backtesting and systems developement.

More when I have time.

Snake.

Survivorship.

Todays ASX 300 isnt that of 10 yrs ago.So we are backtesting todays as how it would have performed 10 yrs ago.Practically chances are it would not have even been in the list.

tech/a said:

Snake.

Survivorship.

Todays ASX 300 isnt that of 10 yrs ago.So we are backtesting todays as how it would have performed 10 yrs ago.Practically chances are it would not have even been in the list.

Click to expand...

I was talking about time before which is what you are talking about too.

Regards
Snake

Re: Backtest your way to financial security.

MichaelD said:

You've got me on this one. I'm going to go for human emotion as the answer.

Click to expand...

I was hoping you could tell me the answer because I don't think I know. It's definately not emotion, though.

Re: Backtest your way to financial security.

Bobby said:

Greetings Snake,

Backtesting is just that! Testing the past ~ Many have done the testing to form their trading plans for the future.

I DON'T.
Will be interesting to see what answers you get.

Regards Bob.

Click to expand...

Yes Bobby, I have always had trouble linking the past with the future. Markets change, companies change or fail to survive, new products change trading, so how can the past be representative of the future? How can past bull markets be representative of bullmarkets of late?

I am not criticising backtesting here but would love to have some experts convince me that it is valid and worthy of doing.

Bob have a go at the questions. I like to read what you write!
Regards
Snake :run:

WayneL,

Could you please answer the questions too. Bring that avatar along when you do it.

I don’t understand trading – why anyone would want to do this for a living? As an investor with a buy and hold philosophy, I find it strange that there are some investors who take such extraordinary risks with their wealth.

We should realise that for every buyer there is a seller. Every time you buy the stock that you think is going to go gangbusters, the seller thinks it is going to go south. So In order to be a successful trader you have to be consistently a better judge of the stock than everyone else. You have to be able to outguess others most of the time.

And remember that not only do you have to do it better than others, but significantly better since you have to cover your brokerage and your capital gains tax when you sell, in addition to a risk premium. And if you are leveraged then add interest and a further risk premium. OK, you do get to write off losses against gains, but are you honest in assessing your success rate? Do your successes lead you to discount or ignore your failures and do you become over-confident? As they say: “success has a thousand fathers; failure is an orphan”.

The market overall for the last 17 or so years has been very forgiving of investors generally. It has been the longest bull run in history, except for some minor upsets, and for that reason many people think that this is the status quo – the normal state of affairs in investing. It ain’t so. A large number of younger investors have never known a major bear market (a drop of 40% or more) and are in for a very unpleasant surprise when it happens; particularly if they are leveraged to the hilt with margin loans. A lot of people in that position jumped out of their office windows in 1929.

To those traders who can point to a history of successful trading I ask: how much of that has been due to your skill and how much to the good fortune of simply being in a sustained bull market? I suspect that it has been the market that has been carrying most traders and if they were honest they would admit this and realise the risks they are taking.

Traders should realise that they are in a negative sum game. The universe of traders has to lose money in total – the losses go in brokerage, taxes and interest. Most of the time traders are merely swapping stocks among themselves (like managed funds).

Admit that trading is an expensive hobby and go and get a real job!!!

Any comments?

battiwallah said:

Admit that trading is an expensive hobby and go and get a real job!!!

Any comments?

Click to expand...

Oh brother, not another one.

My comment is to get qualified on what you are commenting on before commenting on something you are obviously not qualified to comment on.

Re: Backtest your way to financial security.

It's Snake Pliskin said:

WayneL,

Could you please answer the questions too. Bring that avatar along when you do it.

Click to expand...

I'm by no means an expert on backtesting, but will put some thought into this.

later.....

battiwallah said:

I don’t understand trading – why anyone would want to do this for a living? As an investor with a buy and hold philosophy, I find it strange that there are some investors who take such extraordinary risks with their wealth.

We should realise that for every buyer there is a seller. Every time you buy the stock that you think is going to go gangbusters, the seller thinks it is going to go south. So In order to be a successful trader you have to be consistently a better judge of the stock than everyone else. You have to be able to outguess others most of the time.

And remember that not only do you have to do it better than others, but significantly better since you have to cover your brokerage and your capital gains tax when you sell, in addition to a risk premium. And if you are leveraged then add interest and a further risk premium. OK, you do get to write off losses against gains, but are you honest in assessing your success rate? Do your successes lead you to discount or ignore your failures and do you become over-confident? As they say: “success has a thousand fathers; failure is an orphan”.

The market overall for the last 17 or so years has been very forgiving of investors generally. It has been the longest bull run in history, except for some minor upsets, and for that reason many people think that this is the status quo – the normal state of affairs in investing. It ain’t so. A large number of younger investors have never known a major bear market (a drop of 40% or more) and are in for a very unpleasant surprise when it happens; particularly if they are leveraged to the hilt with margin loans. A lot of people in that position jumped out of their office windows in 1929.

To those traders who can point to a history of successful trading I ask: how much of that has been due to your skill and how much to the good fortune of simply being in a sustained bull market? I suspect that it has been the market that has been carrying most traders and if they were honest they would admit this and realise the risks they are taking.

Traders should realise that they are in a negative sum game. The universe of traders has to lose money in total – the losses go in brokerage, taxes and interest. Most of the time traders are merely swapping stocks among themselves (like managed funds).

Admit that trading is an expensive hobby and go and get a real job!!!

Any comments?

Click to expand...

Holy, angry investor Batman!
It's Realist's long lost twin!
Quick, to the batcave, to get our investor repellant spray.
It's the only way to keep us traders safe

professor_frink said:

Holy, angry investor Batman!
It's Realist's long lost twin!
Quick, to the batcave, to get our investor repellant spray.
It's the only way to keep us traders safe

Click to expand...

heheheh

I might have a bit of fun with this later... unless someone gets in before me

Right now I have to go and do a deal with a wage slave. back later

battiwallah

I don’t understand trading – why anyone would want to do this for a living? As an investor with a buy and hold philosophy,

Click to expand...

Doing such is RISKY!

We should realise that for every buyer there is a seller.

Click to expand...

Actually there are multiples of sellers and buyers depending on the $ value of transactions. To suggest there is one buyer for every seller, or vice versa, is ridiculous.

Every time you buy the stock that you think is going to go gangbusters, the seller thinks it is going to go south. So In order to be a successful trader you have to be consistently a better judge of the stock than everyone else. You have to be able to outguess others most of the time.

Click to expand...

It isn't about guessing or judging. Businesses are the same.

And remember that not only do you have to do it better than others, but significantly better since you have to cover your brokerage and your capital gains tax when you sell, in addition to a risk premium. And if you are leveraged then add interest and a further risk premium. OK, you do get to write off losses against gains, but are you honest in assessing your success rate? Do your successes lead you to discount or ignore your failures and do you become over-confident? As they say: “success has a thousand fathers; failure is an orphan”.

Click to expand...

Great advice.

The market overall for the last 17 or so years has been very forgiving of investors generally. It has been the longest bull run in history, except for some minor upsets, and for that reason many people think that this is the status quo – the normal state of affairs in investing. It ain’t so. A large number of younger investors have never known a major bear market (a drop of 40% or more) and are in for a very unpleasant surprise when it happens; particularly if they are leveraged to the hilt with margin loans. A lot of people in that position jumped out of their office windows in 1929.

Click to expand...

Yes I remember that back in 1929.

To those traders who can point to a history of successful trading I ask: how much of that has been due to your skill and how much to the good fortune of simply being in a sustained bull market? I suspect that it has been the market that has been carrying most traders and if they were honest they would admit this and realise the risks they are taking.

Click to expand...

From an investor.

Traders should realise that they are in a negative sum game. The universe of traders has to lose money in total – the losses go in brokerage, taxes and interest. Most of the time traders are merely swapping stocks among themselves (like managed funds).

Click to expand...

I believe it is a minus some game.

Admit that trading is an expensive hobby and go and get a real job!

Click to expand...

I captured one exclamation mark. But why the vicious tone?

Any comments?

Click to expand...

Have a nice day.

Not sure if it constitutes back testing or not, but I looked at my losers and most of them had one thing in common, lack of volume. So I consider I learnt from having a good look back.

battiwallah said:

So In order to be a successful trader you have to be consistently a better judge of the stock than everyone else.

Click to expand...

Totally incorrect, and that's why you can only understand buy/hold. The best traders are the best risk managers. The "best" judges are making money writing newsletters, not trading.

Re: Backtest your way to financial security.

It's Snake Pliskin said:

Yes Bobby, I have always had trouble linking the past with the future. Markets change, companies change or fail to survive, new products change trading, so how can the past be representative of the future? How can past bull markets be representative of bullmarkets of late?

I am not criticising backtesting here but would love to have some experts convince me that it is valid and worthy of doing.

Bob have a go at the questions. I like to read what you write!
Regards
Snake :run:

.

Click to expand...

Hello Snake,

Hope this answer helps, its for Michael as well ( regarding I DON'T ).

When one backtests they see past performance that was based on EOD trading plans, now today we have realtime online technology that allows us to see , plan , & trade in an instant.
The key levels used by the EOD traders can be used to advantage by the realtime trader.
Guess what i'm trying to say is ~ Times have changed, new conditions now apply.
Backtesting past performance = Just that .
I may be wrong but think about it

I like this, what stage are you at ?

1. Unconscious incompetence = you don't even know you don't know what to do.
2.Conscious incompetence = you now know you don't know what to do.
3.Conscious competence = you know what to do. if only you can do it.

4.Unconsious competence = { have a guess at this one }

I'm almost at number 3 , I hope *.

Cheers Bob.

Wow there is some stuff in this thread to cover.

battiwallah said:

I don’t understand trading – why anyone would want to do this for a living? As an investor with a buy and hold philosophy, I find it strange that there are some investors who take such extraordinary risks with their wealth.

Click to expand...

Investors without trading experience have no place "Risking" their wealth attempting to trade.Indeed they will "Risk" their wealth.

We should realise that for every buyer there is a seller. Every time you buy the stock that you think is going to go gangbusters, the seller thinks it is going to go south.

Click to expand...

Well the seller could also be taking a profit after holding well before you bought,he may also be selling at a breakeven after getting his original trade wrong.

So In order to be a successful trader you have to be consistently a better judge of the stock than everyone else. You have to be able to outguess others most of the time.

Click to expand...

Not so you need to understand Reward to Risk how it works and how to be consistantly profitable over the long term even if trading very short timeframes.Your statement is common with those who dont or cannot trade consistently profitably.If you believe its about guessing then you should never attempt to trade.

And remember that not only do you have to do it better than others, but significantly better since you have to cover your brokerage and your capital gains tax when you sell, in addition to a risk premium. And if you are leveraged then add interest and a further risk premium.

Click to expand...

Better tha others only in the Business of trading

OK, you do get to write off losses against gains, but are you honest in assessing your success rate? Do your successes lead you to discount or ignore your failures and do you become over-confident? As they say: “success has a thousand fathers; failure is an orphan”.

Click to expand...

My success rate is around 2-3/10 my profit not only would make your eyes water but it has mine going as well.Frankly the returns that can be made in the maket are OBSCENE Without sounding like a blow hard but using the following as an example---On Fridays trade WMT-- I made the equivalent
to 3 mths pay for my Production Manager in 5 hrs. This morning in a split second on open I made 3 weeks of his wage---that is OBSCENE

The market overall for the last 17 or so years has been very forgiving of investors generally. It has been the longest bull run in history, except for some minor upsets, and for that reason many people think that this is the status quo – the normal state of affairs in investing. It ain’t so. A large number of younger investors have never known a major bear market (a drop of 40% or more) and are in for a very unpleasant surprise when it happens; particularly if they are leveraged to the hilt with margin loans. A lot of people in that position jumped out of their office windows in 1929.

Click to expand...

I love doom and gloom ---very profitable. One of my investment portfolio's is available for public veiw and has been live traded for 4 yrs.Its designed and traded on Margin. If it did drop 50% in a day or so the return over that 5 yrs period would still be 400% on initial capital. IE starting Capital was $30k----- $70K on Margin loan.Balance as at Friday $386,000---50% = $193,000 les the $70k to lender. Balance $126,000.

As for the longest Bullmarket in history you havent looked at a chart of the ASX lately.

To those traders who can point to a history of successful trading I ask: how much of that has been due to your skill and how much to the good fortune of simply being in a sustained bull market? I suspect that it has been the market that has been carrying most traders and if they were honest they would admit this and realise the risks they are taking.

Click to expand...

Both the market and skill. It takes skill to outperform the market. Every business has risk--however in all busines including trading I take quantified risks--if you dont take quantified risks---even investors take risk--- then you'll never excell.

Traders should realise that they are in a negative sum game. The universe of traders has to lose money in total – the losses go in brokerage, taxes and interest. Most of the time traders are merely swapping stocks among themselves (like managed funds).

Click to expand...

5% of us can and do feed then from the constant supply of new blood who venture into the business of trading.
My Company dominates in its field here and is no different to any business which succeeds---it has a greater market share and some fail.Many competitors have been and gone over the 30 yrs I've been in business. If they cannot do it as well as I or as profitably then they deserve what they sow.

Admit that trading is an expensive hobby and go and get a real job!!!

Any comments?

Click to expand...

Its not a hobby to me yet I dont trade fulltime,it is simply part of my investment strategy.

Your comments are very general and you take the veiw that all here are in one box. You may well be suprised at how inept your own investment planning may well be in comparison to some here. Agreed many are struggling.
But put in the apprenticeship and the profit will come as will the Business acumen.

tech/a said:

Every business has risk--however in all busines including trading I take quantified risks--if you dont take quantified risks---even investors take risk--- then you'll never excell.

Click to expand...

Just quoting for emphasis. There is no difference between trading and any other type of investing, be it business, property, etc.

swingstar said:

Just quoting for emphasis. There is no difference between trading and any other type of investing, be it business, property, etc.

Click to expand...

Exactly Swing.

Snake
Wow - good questions. Certainly got me thinking. Trading mechanical systems completely transformed my trading to very average to results that I am very happy with. Developing trading systems is a very rewarding approach to the markets. Back-testing allows me to develop systems.

There have been whole books written on this topic. Beyond Technical Analysis by Tushar Chande springs to hand as a starting point. Although some on these forums are well past this sort of stuff the main points of the book are still reasonable.

My responses are obviously my view of trading and what has worked for me;
Why do people back-test?
People back-test in an attempt to develop a winning trading system. For the average trader, like myself, trading a mechanical trading system maximises my chances of success. Without a well thought out, extensively "pressure" tested, and documented set of rules to trade by I am indeed a worse than average trader. With a documented system I can teach nearly anybody to trade successfully. And I am a lousy teacher!

What constitutes a successful back-test leading to a blueprint?
One that, over a statistically significant number of trades is profitable.

What biases are there to be aware of?
Lots! Back-testing can give insight into these biases. Van Tharpe goes into biases in some detail.

What issues are there with equipment and software deficiencies?
Data quality is an issue that needs to be considered. Software is very good these days and is more limited on our abilities to code our ideas and weaknesses than anything inherently wrong with the software. The power of software is quite amazing these days - we can test fundamental data as well as price and volume with AmiBroker and we can use artificial intelligence to optimise. We have the power of Monte Carlo tests at our fingertips. The main limiting factor is ourselves.

Why can every aspect of human nature be back-tested?
I doubt that it can - but it is possible to test procrastination and poor execution so it probably possible to test for most issues that a rational person would confront.

What is one thing that corrupts all back-testing, and what same thing is relevant for trading without back-testing?
Obviously back-testing can tend to curve fit - which is why you can also forward test. Holding some data outside of the initial sample is a well known strategy. It is possible to trade without a tested system. But why would someone consciously trade a system if they don't know what it's chances of success are?

I am glad that you didn't ask about optimisation!

Stevo
http://drawdown.blogspot.com/