Backtest your way to financial security

MichaelD · 27 November 2006

Re: Backtest your way to financial security.

Bobby said:
When one backtests they see past performance that was based on EOD trading plans, now today we have realtime online technology that allows us to see , plan , & trade in an instant.
The key levels used by the EOD traders can be used to advantage by the realtime trader.

This presumes that you want or need to engage the market intraday. I don't. That would for me defeat the whole point of what I am trying to achieve lifestyle wise by trading.

Backtesting intraday strategies is certainly possible, but is very time consuming (I have access to tick data if I want to utilize it for backtesting, but so far have chosen not to use it as part of a system design).

EOD trader levels do indeed appear to be used by intraday traders to their advantage - 'tis one of the reasons I don't put my stops in market.

Bobby said:
Guess what i'm trying to say is ~ Times have changed, new conditions now apply.

"This time it's different."

Bobby said:
Backtesting past performance = Just that .

Yep. In the absence of a crystal ball, it's the next best thing because it can clearly alert you in advance to the following;
1. You're trading a system which has never been profitable so is unlikely to be profitable now.
2. Your system is no longer trading within previously tested limits - why?

Bobby said:
what stage are you at ?

Position trading - 3, just following the plan without fear or favour
Short term trading - 2 and steadily working towards 3

nizar · 27 November 2006

stevo said:
Why can every aspect of human nature be back-tested?
I doubt that it can - but it is possible to test procrastination and poor execution so it probably possible to test for most issues that a rational person would confront.

Stevo

How can back-testing, test procrastination and poor execution?

It's Snake Pliskin · 27 November 2006

Re: Backtest your way to financial security.

Bobby said:
Hello Snake,

Hope this answer helps, its for Michael as well ( regarding I DON'T ).

When one backtests they see past performance that was based on EOD trading plans, now today we have realtime online technology that allows us to see , plan , & trade in an instant.
The key levels used by the EOD traders can be used to advantage by the realtime trader.
Guess what i'm trying to say is ~ Times have changed, new conditions now apply.
Backtesting past performance = Just that .
I may be wrong but think about it

I like this, what stage are you at ?

1. Unconscious incompetence = you don't even know you don't know what to do.
2.Conscious incompetence = you now know you don't know what to do.
3.Conscious competence = you know what to do. if only you can do it.

4.Unconsious competence = { have a guess at this one }

I'm almost at number 3 , I hope *.

Cheers Bob.

Bob,

4. Intuitively trading?

It's Snake Pliskin · 27 November 2006

stevo said:
Snake
Wow - good questions. Certainly got me thinking. Trading mechanical systems completely transformed my trading to very average to results that I am very happy with. Developing trading systems is a very rewarding approach to the markets. Back-testing allows me to develop systems.

There have been whole books written on this topic. Beyond Technical Analysis by Tushar Chande springs to hand as a starting point. Although some on these forums are well past this sort of stuff the main points of the book are still reasonable.

My responses are obviously my view of trading and what has worked for me;
Why do people back-test?
People back-test in an attempt to develop a winning trading system. For the average trader, like myself, trading a mechanical trading system maximises my chances of success. Without a well thought out, extensively "pressure" tested, and documented set of rules to trade by I am indeed a worse than average trader. With a documented system I can teach nearly anybody to trade successfully. And I am a lousy teacher!

What constitutes a successful back-test leading to a blueprint?
One that, over a statistically significant number of trades is profitable.

What biases are there to be aware of?
Lots! Back-testing can give insight into these biases. Van Tharpe goes into biases in some detail.

What issues are there with equipment and software deficiencies?
Data quality is an issue that needs to be considered. Software is very good these days and is more limited on our abilities to code our ideas and weaknesses than anything inherently wrong with the software. The power of software is quite amazing these days - we can test fundamental data as well as price and volume with AmiBroker and we can use artificial intelligence to optimise. We have the power of Monte Carlo tests at our fingertips. The main limiting factor is ourselves.

Why can every aspect of human nature be back-tested?
I doubt that it can - but it is possible to test procrastination and poor execution so it probably possible to test for most issues that a rational person would confront.

What is one thing that corrupts all back-testing, and what same thing is relevant for trading without back-testing?
Obviously back-testing can tend to curve fit - which is why you can also forward test. Holding some data outside of the initial sample is a well known strategy. It is possible to trade without a tested system. But why would someone consciously trade a system if they don't know what it's chances of success are?

I am glad that you didn't ask about optimisation!

Stevo
http://drawdown.blogspot.com/

Thanks Stevo.

Yes optimisation could open up a can of worms.

Snake

It's Snake Pliskin · 27 November 2006

carmo said:
Not sure if it constitutes back testing or not, but I looked at my losers and most of them had one thing in common, lack of volume. So I consider I learnt from having a good look back.

And you didn't need software to realise it! It's marvelous how the human brain can learn through experience. It is rather uncomfortable taking the plunge though without some safety blanket so to speak - human nature.

tech/a · 28 November 2006

nizar said:
Stevo

How can back-testing, test procrastination and poor execution?

It should go a long way to eliminating it.Once you have the trading "Blueprint" .
If you follow exactly entry/exit and stops you'll return similar numbers to those found in backtesting.
Certainly been the case in the 3 I use.
I know Stevo's had the same result with his and Jose' Silva.

The most consistent returns I have found are those returned by traders who have a well tested simple Trading System. WELL TESTED

stevo · 28 November 2006

"How can back-testing, test procrastination and poor execution?"

Procrastination could be tested by taking only a % of signals that are triggered. Or random entry factor could be added to the buy signal, such as "buy only if I toss heads or roll a 6 on the dice" type approach.

Poor execution could be along the lines of buy and sell at the worst possible price, or buy at a random price above the midpoint for the week.

It really comes back to thinking creatively. Try to make the system fail. If you are trading a daily timeframe try delaying the exits for a couple of days and see what happens. I am sure that the time delay could be random - "sell sometime in the week after the exit is given. I can't trade a daily timeframe due to work commitments but it took me some time to work this out. It's much better to learn through back-testing than to learn through experience and real dollars. Experience can be the worst teacher!

I usually test with a random entry / exit in the week following the signal since I trade on a weekly timeframe. If a system returns a positive result even when the trades are executed at the worst possible price then it is more robust than a system that doesn't. Try all the possible combinations.

I wouldn't trade a system that relies on me executing the trades at the open or the close price or a very specific trigger price but fails at a random price or worst possible price for the day / week. But I would like to know that the system tends to work better if I get out at the open price.

Stevo

tech/a · 28 November 2006

stevo said:
"How can back-testing, test procrastination and poor execution?"

Good points Steve.

Procrastination could be tested by taking only a % of signals that are triggered. Or random entry factor could be added to the buy signal, such as "buy only if I toss heads or roll a 6 on the dice" type approach.

Poor execution could be along the lines of buy and sell at the worst possible price, or buy at a random price above the midpoint for the week.

It really comes back to thinking creatively. Try to make the system fail. If you are trading a daily timeframe try delaying the exits for a couple of days and see what happens. I am sure that the time delay could be random - "sell sometime in the week after the exit is given. I can't trade a daily timeframe due to work commitments but it took me some time to work this out. It's much better to learn through back-testing than to learn through experience and real dollars. Experience can be the worst teacher!

I usually test with a random entry / exit in the week following the signal since I trade on a weekly timeframe. If a system returns a positive result even when the trades are executed at the worst possible price then it is more robust than a system that doesn't. Try all the possible combinations.

I wouldn't trade a system that relies on me executing the trades at the open or the close price or a very specific trigger price but fails at a random price or worst possible price for the day / week. But I would like to know that the system tends to work better if I get out at the open price.

Stevo

Montecarlo testing will turn up a great deal. For those un familiar.

Lets say you test 20000 portfolios.

It's like giving 20000 people your system and your starting capital and telling them to follow it for the test period,then coming back and reporting the results.

Some one off results will return a profit but when Montecarlo tested 80% of portfolio's may return a profit and 20% fail. I and I'm sure steve look for 100% returning profit.I then look at the deviation from the Average to the worst and best performing portfolio. I personally look for around 20% max from best to worst.

Steve's other ways of testing worst cases are very good.

As Steve points out do everything you can to FAIL your system after all some will place $100s of thousands into trading methodologies in the form of Super or hard earned wealth created over many years.
Better to find failure now than when fully invested.

rub92me · 28 November 2006

I personally would not be comfortable just back testing on recent (say past 24 months) historical data on one exchange.
I wonder whether it would be better to back test on a variety of data sets you can create with certain characteristics. (Without any pre-conceived bias of what the outcome should be)

1) A number of Bull datasets – a basket of 200 stocks with certain minimum price / volume characteristics with the following criteria:
# 60-80% went up by more than x % over a given time
#10- 30% stayed within x% of the start price over a given time
# 5-20 % went up more than x % over a given time

2) A number of Sideways datasets - a basket of 200 stocks with certain minimum price / volume characteristics with the following criteria:
# 10-30% went up by more than x % over a given time
# 30-50% stayed within x% of the start price over a given time
# 10-30 % went up more than x % over a given time

3) A number of Bear datasets - a basket of 200 stocks with certain minimum price / volume characteristics with the following criteria:
# 5-20 % went up by more than x % over a given time
#10-30% stayed within x% of the start price over a given time
# 60-80% went up more than x % over a given time

I wonder if anyone has attempted to do this with their trading systems, to see how robust they are in certain types of markets, or am I barking up the wrong tree here?

Bobby · 28 November 2006

Re: Backtest your way to financial security.

It's Snake Pliskin said:
Bob,

4. Intuitively trading?

Yes Snake
You subconsciously sense when to make that trade .

Michael- Thanks for your reply, you said that you don't place a stop in the market, I understand why, but could you expain in your own words for others.

Like to know more of your thoughts on backtesting " discretionary " as markets follow volume this may be a challenge.

Cheers Bob.

It's Snake Pliskin · 28 November 2006

Re: Backtest your way to financial security.

Bobby said:
Yes Snake
You subconsciously sense when to make that trade .Cheers Bob.

The marvelous human brain.

rub92me · 28 November 2006

rub92me said:
1) A number of Bull datasets – a basket of 200 stocks with certain minimum price / volume characteristics with the following criteria:
# 60-80% went up by more than x % over a given time
#10- 30% stayed within x% of the start price over a given time
# 5-20 % went up more than x % over a given time

2) A number of Sideways datasets - a basket of 200 stocks with certain minimum price / volume characteristics with the following criteria:
# 10-30% went up by more than x % over a given time
# 30-50% stayed within x% of the start price over a given time
# 10-30 % went up more than x % over a given time

3) A number of Bear datasets - a basket of 200 stocks with certain minimum price / volume characteristics with the following criteria:
# 5-20 % went up by more than x % over a given time
#10-30% stayed within x% of the start price over a given time
# 60-80% went up more than x % over a given time

Sorry didn't proofread this properly; should be a-b% went down more than x% for the third category in these scenarios

stevo · 28 November 2006

"I wonder whether it would be better to back test on a variety of data sets you can create with certain characteristics."

This is a very good idea. You could also great random data sets or ones just based on the alphabet. You can also test different time periods and do walk forward testing using something like Fred's IO (Intelligent Optimizer). Monte Carlo testing that Tech mentions above also give insights into performance with different stock selections.

I like to have at least 5 years worth of data. It's interesting to watch the market change over time - companies come and go. The dataset is not static and I cannot guarantee that is is 100% correct. It's great to have the 1999 to 2003 period in the tests.

I guess the obvious thing to say about back testing and trading systems are that there are no guarantees. I am trading systems that I developed back in 2002 and 2003 using the market data from the period prior to 2003. These same systems test quite well now and have performed to expectations given the market conditions. So they forward tested well! I feel reasonably confident that they will perform going forward, although I have been fortunate, as many have, favourable market conditions have prevailed since 2003.

Tech and some others that use long term mechanical trading systems have shown that systems do work. Some are very surprised at the money they have made using a system (my very trusting brother is one!). At the same time other less well thought out strategies / traders have done a lot of damage. I have seen people post that they put all there money into Telstra and have lost 40% of their equity! A "no strategy" (have a punt, it can't go down anymore) style buy and hold strategy is not for me. But each to their own, what works for one doesn't work for another.

Stevo

tech/a · 28 November 2006

I guess the obvious thing to say about back testing and trading systems are that there are no guarantees.

Ah yes.

But what you do have a clearly defined parameters that will tell you if the system is no longer working al la turtle system.
OR
If it trades outside the know parameters you have early warning that something is seriosly wrong. IE market conditions havent been seen like those being traded in the test period.
OR
The system is failing.
OR
Your system testing was/is flawed.

Like Steve I have been trading 3 systems designed using data from 1996-2002 and all have not traded outside those parameters delived during testing.

MichaelD · 28 November 2006

Re: Backtest your way to financial security.

Bobby said:
Michael- Thanks for your reply, you said that you don't place a stop in the market, I understand why, but could you expain in your own words for others.

Like to know more of your thoughts on backtesting " discretionary " as markets follow volume this may be a challenge.

I don't place a stop in market because backtesting shows me that systems using such a stop have more drawdown and less profit than the exact same systems with an EOD stop not placed in the market. An intraday stop will get hit by extremes of panic selling.

Backtesting discretionary is exactly the same as backtesting mechanical except you have to backtest each trade by hand. Write down the plan. Pick a time to start the backtest. Find your entries. Walk each trade through to its exit. Note the results. Repeat and repeat.

professor_frink · 28 November 2006

For anyone interested- Amibroker has discretionary back testing capabilities. When you run it, it simply allows you to put in buy,sell,short,cover entries in manually, and will punch out all the same stats as a mechanical test. Quite handy for anyone interested.

nizar · 28 November 2006

professor_frink said:
For anyone interested- Amibroker has discretionary back testing capabilities. When you run it, it simply allows you to put in buy,sell,short,cover entries in manually, and will punch out all the same stats as a mechanical test. Quite handy for anyone interested.

Very nice!

stevo · 29 November 2006

Re: Backtest your way to financial security.

MichaelD said:
I don't place a stop in market because backtesting shows me that systems using such a stop have more drawdown and less profit than the exact same systems with an EOD stop not placed in the market. An intraday stop will get hit by extremes of panic selling.

This is the same approach I take - the results test better without a stop placed in the market. Only I use a weekly timeframe because I haven't found an EOD strategy that is worth the effort. This is a much more relaxed approach with lower transaction costs due to less activity (129 trades over nearly 4 years). An ideal week for me is when I don't have to do anything - it makes taking a break much easier.

Obviously not for everyone but it suits me.

Stevo

swingstar · 29 November 2006

professor_frink said:
For anyone interested- Amibroker has discretionary back testing capabilities. When you run it, it simply allows you to put in buy,sell,short,cover entries in manually, and will punch out all the same stats as a mechanical test. Quite handy for anyone interested.

Hi PF, how do you do this (can give me some terms to search the help file)?

nizar · 30 November 2006

Re: Backtest your way to financial security.

MichaelD said:
Backtesting discretionary is exactly the same as backtesting mechanical except you have to backtest each trade by hand. Write down the plan. Pick a time to start the backtest. Find your entries. Walk each trade through to its exit. Note the results. Repeat and repeat.

Michael,

Isnt discretionary backtesting the same thing as paper-trading in the sense of what you will get out of it in terms of results ?

Backtest your way to financial security

MichaelD

Not fooled by randomness

nizar

It's Snake Pliskin

It's Snake Pliskin

It's Snake Pliskin

tech/a

No Ordinary Duck

stevo

tech/a

No Ordinary Duck

rub92me

Don't look back

Bobby

It's Snake Pliskin

rub92me

Don't look back

stevo

tech/a

No Ordinary Duck

MichaelD

Not fooled by randomness

professor_frink

Moderator

nizar

stevo

swingstar

nizar

Similar threads

Backtest your way to financial security

Not fooled by randomness

No Ordinary Duck

No Ordinary Duck

Don't look back

Don't look back

No Ordinary Duck

Not fooled by randomness

Moderator

Similar threads

Privacy & Transparency

Privacy & Transparency