Mechanical System Testing Results

[AMSH]

Hey,
I was hoping one of the system gurus on here could give me some feedback about analysing test results, specifically, the impact of very low win rates.

Our hypothetical system has the following stats:
- No of trades - 620 (1 trade every 3 days say)
- Win Rate - 20%
- Average Profit - 982
- Average Loss - 75
- Reward/risk - 13
- Expectancy - 1.86
- Average days in trade - 8

Questions as follows:

1.
What is the lowest win rate that you, personally would trade (have traded profitably)?

2.
Are the main concerns with a low win rate that (1) max dd will be greater and (2) inability to leverage as a result? Should I be concerned about anything else?

3.
What expectancy would you consider a profitable system to produce - I know there are many other factors to consider but there must be some general benchmarks?

4.
Would you trade the system with the stats above? Why/why not

Any help greatly appreciated,

Cheers,

AMSH

 

[Nick Radge]

AMSH,
A few comments for you.

An answer about trading a system is its comfort level for the user. In simple terms we can measure comfort level by way of Profit Factor, which is net profit / net loss. Yours is over 3.0 which is extremely acceptable. Anything below 2.0 is probably going to be difficult for an amateur to trade. Personally I wouldn't trade a system below 1.5. 3.0 and above is a great system, albeit this is a simplistic measure.

We can gauge the maxDD using probability theory, or at least get an accurate indication, using:

LS = (LN(TS)/-LN((1-PW/100)))

Where,
LS = Losing Streak
LN = Natural Logarithm
TS = Trade Sample
PW = Probability of winning

A system with a 20% win rate will at some stage suffer a string of 48 losses in a row. Using 1% risk per trade means a maxDD of some 50% which would be very difficult for most to travel through. Even at 0.5% risk per trade a maxDD of 25% is to be expected, which again is usually beyond the threshold of most.

With this maxDD also comes lengthy periods of sideways equity growth which many will find extremely frustrating, so much so that most would throw it in.

I'd also say, without knowing anything about the system specifics, is that a win/loss ratio as high as yours is extremely rare, if not odd. As such I'd see it as a red flag, possibly suggesting a postdictive error of some type. Just guessing, but that high w/l ratio would make me investigate the mechanics to ensure that you can actually trade it in the real world.

Nick

This post may contain advice that has been prepared by Reef Capital Coaching ABN 24 092 309 978 (“RCC”) and is general advice and does not take account of your objectives, financial situation or needs. Before acting on this general advice you should therefore consider the appropriateness of the advice having regard to your situation. We recommend you obtain financial, legal and taxation advice before making any financial investment decision.

 

[Cartman]

Hey,
I was hoping one of the system gurus on here could give me some feedback about analysing test results, specifically, the impact of very low win rates.

Our hypothetical system has the following stats:
- No of trades - 620 (1 trade every 3 days say)
- Win Rate - 20%
- Average Profit - 982
- Average Loss - 75
- Reward/risk - 13
- Expectancy - 1.86
- Average days in trade - 8

Questions as follows:

1.
What is the lowest win rate that you, personally would trade (have traded profitably)?

2.
Are the main concerns with a low win rate that (1) max dd will be greater and (2) inability to leverage as a result? Should I be concerned about anything else?

3.
What expectancy would you consider a profitable system to produce - I know there are many other factors to consider but there must be some general benchmarks?

4.
Would you trade the system with the stats above? Why/why not

Any help greatly appreciated,

Cheers,

AMSH

AMSH i know nothing about coding and backtesting but from your data you r obviously running real tight stops and relying on the odd runner to profit --- have u considered taking a half position size on the initial entry and setting a wider stop loss --- if the position behaves ok u can add the second half to the position --- might give your system a bit more breathing space and improve the strike rate --- if it doesnt improve it id say back to the drawing board

 

[Stormin_Norman]

Hey,
I was hoping one of the system gurus on here could give me some feedback about analysing test results, specifically, the impact of very low win rates.

Our hypothetical system has the following stats:
- No of trades - 620 (1 trade every 3 days say)
- Win Rate - 20%
- Average Profit - 982
- Average Loss - 75
- Reward/risk - 13
- Expectancy - 1.86
- Average days in trade - 8

Questions as follows:

Would you trade the system with the stats above? Why/why not

i wouldnt trade it, because like you pointed out, and was mentioned above the drawdown probability is huge.

is it possible to add another limiting factor to the trade to increase the win rate? when it wins it wins nicely. is there something which will keep it out of some of the losing trades?

the other thing ill say is backtesting notoriously give vastly different results to forward testing in MT4 (if that is what youre using).

 

[MRC & Co]

I think Howard Bandy (looks the best systems book out there from what I have quickly seen) talks about the last 8 weeks of data as being the only useful sample to backtest.

I would agree. If your system executed 600+ trades at an average of 3 days, it's way over this limit.

Environment would just be too different to the current environment and your system will probably be demolished in these times. I would guess.

Who knows for sure, but taking a 50% DD to find out, is not really worth the pain...........at least for me it wouldn't be.............good answers by Nick though, gives some very useful things to think about. Cartman also points out a good point, you look like your trying to ride longer trends, so partial exits could help minimise risk and would be viable for a trend following system (MichaelD from around here uses such entries I believe).

 

[tech/a]

I think Howard Bandy (looks the best systems book out there from what I have quickly seen) talks about the last 8 weeks of data as being the only useful sample to backtest.

Surely he is talking short term systems?

Developing a long term system around 8 weeks of data isn't ideal!
Even a shorter term system wont see the majority of possible market conditions that can be experienced in just 8 weeks.

 

[AMSH]

Thanks for the replies gents. Your comments basically reinforced what is was thinking re the DD, but was unable to calc.

Couple of comments:
Nick, thanks for the post. I'll look into the DD calc which I haven't seen before and get back to you. If you have a moment, could you expand on the LN? I'll research it myself, so only if you're bored You're partially right re the high win/loss. The system stats i quoted had some duff code which referenced something that wouldn't be known at the entry. I knew this but thought the stats would be good for analysing very low win rate and high win/loss. The adjusted version (with some fiddling) returns close to the same expectancy and a more acceptable win rate. I'll come back with the stats for that when it's finalised.

MRC and SM, I'm fairly new to system trading but with regards to back testing I take the approach that the greater the data set that tests are based on the better, then systems with good stats can be tested in shorter timeframes. For example, the previous system was tested from Jan 00 through Dec 08. If results are good, i'll then test against every six month period within the initial timeframe and analyse for volatility in the stats. Thoughts on this approach?

Cartman, because of the limitations of MS, and the fact that I'm following a purely mechanical system, it's hard to apply partial position sizes, averaging up etc. When i get some better software, I'll start playing about, but for now I'm simply looking for something that returns strong stats on a fairly straight-forward approach. Then i can start messing with money management.

Tech, I'm currently using MS Pro 9 and exporting my results to a spreadsheet to calc total system stats. I know this approach will distort my stats so I'm looking for a proper set of testing software. I've heard that you use Tradesim with MS and was wondering if you'd recommend it as a platform - or any other thoughts you might have.

Cheers,

AMSH

 

[Cartman]

Cartman, because of the limitations of MS, and the fact that I'm following a purely mechanical system, it's hard to apply partial position sizes, averaging up etc. When i get some better software, I'll start playing about, but for now I'm simply looking for something that returns strong stats on a fairly straight-forward approach. Then i can start messing with money management.

the fact that u r thinking this way b4 seriously entering the market indicates u will prob do better than most --- wish i had done a bit more research a cupla years back b4 going in boots n all !! ---- good luck with it.

 

[howardbandy]

Greetings all --

I haven't suggested eight weeks as the answer to any general question.

The point that I make most often is the importance of dividing the data into two sets. Use the earlier set, called the in-sample period and in-sample data, to develop your trading system. Fool around with it all you want to. When you are done fooling with it, the results are always good. But those results have little or no value in estimating what the performance of the system will be when you start to trade the system. So test the system (ideally one time only) using the second set of data, called the out-of-sample data and the out-of-sample period, that immediately follows the in-sample data. If the results on the out-of-sample data look good, there is a higher probability that the system will be profitable when you actually trade it.

The out-of-sample results can be subjected to statistical tests. The more closed trades there are in the out-of-sample period, the more confidence you can have that the real trading will be approximately like the out-of-sample results.

Be careful. All of us have the tendency to examine the out-of-sample results and change the model so that the results will improve. It only takes one peek followed by a model change to convert what was previously out-of-sample data into in-sample data. If you do this, have another set of data reserved for the real out-of-sample testing.

Depending on the characteristics of both the model and the market it trades (together these make up the trading system), your model may need periodic readjustment to stay in sync with the market.

The question we all have is: "What will happen when my system goes from development to trading?" My answer is "Practice". Every transition from in-sample to out-of-sample in testing is similar to the transition from development to trading. If you use a set of in-sample and out-of-sample periods, you can run a walk-forward test. Each of the walk-forward steps gives one more data point in the practice.

Run a search on my name to bring up the threads where I discussed my feeling about system development, testing, and validation.

Thanks for listening,
Howard

 

[MS+Tradesim]

Tech, I'm currently using MS Pro 9 and exporting my results to a spreadsheet to calc total system stats. I know this approach will distort my stats so I'm looking for a proper set of testing software. I've heard that you use Tradesim with MS and was wondering if you'd recommend it as a platform - or any other thoughts you might have.

I know you asked tech but I use MS Pro 9.1 and Tradesim (hence my name!) and it is an extremely powerful combination. I do have Amibroker but I've never bothered learning to use its testing abilities as the MS/TS combo is sufficient - more so now with intraday testing soon to be added. I bought the Enterprise edition which seems exy but it has paid for itself not only in allowing me to discard ideas which 'seemed' good but actually fail in broad portfolio testing, but in allowing me to develop very good robust trading systems.

Backtesting in Tradesim proves itself over and over when I see trades I take in real life showing up as they should in the backtested results.

And listen to Howard when it comes to system development.

 

[Stormin_Norman]

thanks for the thoughts howard.

 

[nizar]

Tech, I'm currently using MS Pro 9 and exporting my results to a spreadsheet to calc total system stats. I know this approach will distort my stats so I'm looking for a proper set of testing software. I've heard that you use Tradesim with MS and was wondering if you'd recommend it as a platform - or any other thoughts you might have.

AMSH.

Re:TradeSim, I couldnt recommend it enough. I use it coupled with MS v9 as well.

 

[AMSH]

Thanks for the thoughts gents. Got a copy of TradeSim and have been playing with it about 5 hours a day since. Great software.

Not exactly on topic, but I've noticed that a very effective way to reduce DD is to increase your capital and reduce the % risked per trade. Presumably a very short term system that had an average trade time under a day (using EOD data, not intraday) could take advantage of this by using the +3 days till clearance - i.e. using the banks money when trades are closed before the broker has debited your account. I've been playing with this type of thing and some systems are VERY profitable given the level of capital is raised enough. Thoughts on this?

 

[tech/a]

an average trade time under a day (using EOD data, not intraday) could take advantage of this by using the +3 days till clearance

How on earth can you test a system using EOD data that trades under a day?

A fatal mistake most fall for when systems testing is they fail to factor into their trading that an entry can only be taken on the NEXT bar.
If a signal is given for an entry on an EOD bar you wont see it until you analyse that bar. You can set an exit to be intraday .

Make sure you have this line of code in your Tradesim formula

ExtFml( "Tradesim.EnableDelayOfEntryByOneBar") ;

 

[MS+Tradesim]

A fatal mistake most fall for when systems testing is they fail to factor into their trading that an entry can only be taken on the NEXT bar.

Unless of course you use the closing auction to position yourself thus taking advantage of the day's close. Works beautifully for me ....but is not an option for those who can't be there at that time of course, and will need to use the "delay entry till next bar" code.

 

[AMSH]

Tech, 9 out of 10 of my entry conditions are based on yesterdays (and prior) price info. The final is an open today above yesterdays close. I create an explorer to find the setup then manually watch each possibility on the market open (normally only about three or four possibilities per day). If it opens above the close, then I buy. The less than one day on EOD is the above with a buy on the open and a sell on the close. I suppose this essentially means time in trade is one day.
For backtesting it's easy - you use ref(X,-1) for the 9 conditions and O>Ref(C,-1). I haven't traded in this way yet, but I think it's entirely possible.

 

[Plan_Trader]

amsf,

Yes, but if you are backtesting for the open to be greater than yesterday's close, what bar are you entering on? If it is the current bar, you can't use the open price as trade price. Maybe average price or closing price? Pretty hard to get the open price very time.

 

[AMSH]

Hey Plan_trader,

The open of the current bar is the buy signal. I've assumed missing two price ticks (i.e. the entry price is the open plus a cent). Hence the system assumes that you miss the open by 2 ticks. I would have thought you could get close to the open - you just have to be sitting in front of your PC every morning ready to make the trade..... Is this unrealistic do you think?

AMSH

 

[tech/a]

How do you account for gaps?

Really think you need software that can test intraday.
Tradesims next update will be able to do this evidently.

 

[AMSH]

Do you mean price gapping up intraday tech (would that involve price jumping a number of ticks at once)? If it's a normal gap up of the open then it's treated the same as if it had opened within yesterday's range. Like i said, i haven't traded this type of system, but the results look pretty promising even if the imlementation is a little bit onerous - due in large part to almost daily compounding which really starts to crank up after a year or so. Think i might splash out and get myself a live data feed