Would you trade this system?

[tech/a]

Howard
Thank you for your rsponse and I like the analogy.

Can I pose the following to add perhaps something which is possibly not considered in the above.

The database which you are applying your conditions,arguements and variables to---your system.
We choose a database as this is from where we will extract our model to best take advantage of the characteristic we know is within that dataset.
Lets say that this is our dataset and since the beginning of time it has looked like this.


We also have our own objectives which we require in the results which we see when applying our system and its constituents to the data.
The out of sample data needs to be long enough to have any meaning full result on the out of sample testing.

To select an out of data sample which looks like this

Will have obviously negative results as the system wasnt designed for these conditions.
You will find similar conditions across most datasets.

So while your system has been designed across a large set of data your out of sample dataset is much less (Generally) if taken in isolation it can be misleading.You could throw away a perfectly viable method which suits your objectives.Hence the question of how many out of sample test results give rise to confidence.

My arguement is that in the end you want to outperform the dataset your designing your system to take part in.Wether it outperforms it at the very highest level,you will never determine as the data will be dynamic.Wether it fits perfectly your criteria ALL the time is also a pointless exercise. At times it will at times it wont. The destination is in my view more important than the journey as each time its applied through time the journey (Catastrophic events avoided as best we can) will be different to the in sample AND the out of sample testing.

Lets say this is my system with all the objectives,variables and conditions applicable to my own requirements and those that work well against the data base its designed for.

Here is my system--I like it!
I've selected all of my objectives and this is what I have come up with.



Continued in next post due to number of images.

 

[tech/a]

Continued

Ive tested it over many thousands of tests tweeking and improving over my database which is.Its dynamic everytime I go out on it it alters.I never know exactly what it will look like.I can and have tested it all and on just part sections (out of sample).


And

And


In the end I found I get to my destination,sometimes quicker and safer than other times.
At times in my testing I come across this and could select this as an out of sample period even randomly.


My system isnt designed to perform at all well in this,although many have tried and failed.


For me the bottomline is that while we can NEVER be certain we have the very best system which will cope with EVERY condition the market throws at us,we can have confidence through even in sample testing that out system can and will perform WELL ENOUGH to out perfom most in those conditions it was/is designed for.

I would argue that even better results given the basic system will come from outside improvements like leverage and or re investments of profit back into the system,rather than endlessly looking for the perfectly straight road--or more to the point a system that makes the road endlessly straight.

 

[rnr]

Nizar,

rnr -- position size limit using 100% hasnt made the simulations any worse (I have tried 20,25,30).

I was thinking more along the lines of 10%, 12.5% & 15%.

Changes made:
*Position sizing model changed to $1,600 risk per trade
*Initial stop changed to 20% of purchase price
*Exit changed to 250d EMA.
*Chose to favour pyramid.

Did you make all these changes in 1 hit or did you stage and test the changes 1 at a time in an attempt to optimise each element of the trade?

Did you change the InitialStop from 3*ATR(10) to say 6*ATR(10) and what affect did it have on the Protective Stop Exits.

I don't quite understand why the results of the second simulation have run on to 2005?

Cheers,
rnr

 

[nizar]

Nizar,



I was thinking more along the lines of 10%, 12.5% & 15%.



Did you make all these changes in 1 hit or did you stage and test the changes 1 at a time in an attempt to optimise each element of the trade?

Did you change the InitialStop from 3*ATR(10) to say 6*ATR(10) and what affect did it have on the Protective Stop Exits.

I don't quite understand why the results of the second simulation have run on to 2005?

Cheers,
rnr

rnr.

No, not all at one go. I did it bit by bit so i could find out which element was adding the edge.

6ATR i did try it, as with 4 and 5ATR as well. None of them worked as well as the 20% stop. The wider stops took on a less number of trades (which is low enough as it is!!) and i couldnt get the # of stocks hitting the protective stop exits better than it is now with the 20% stop.

As to why the results have run into 2005. It was never my intention. See my post #30.
In both cases the metastock exploration was set to:

ExtFml( "TradeSim.SetStartRecordDate",01,06,1992);
ExtFml( "TradeSim.SetStopRecordDate",30,09,2002);

With position size limit set to anywhere below 30% the results get worse, in terms of profit. Im not keen on this because since my entry criteria is quite selective, when the market is going a bit sideways, not many stocks are going to be making all time highs. But the ones that do, i want to buy them again and again and again. And due to this pyramiding, i feel that its okay to have a bit of flexibility and leave the position size open.

But i think one drawback is that open profit drawdowns are likely to be increased if im too concentrated in a few stocks, and im gonna have to give back alot of profit at the end of the trend.

If only i could get my open equity chart on tradeSim to start working, then ill have a proper look at it, and see if its worth the pain or not.

Shane.

The whole market is my universe, with liquidity filter $500k per day.

Thanks for the feedback guys, keep it coming

 

[tech/a]

Tradesim Open Equity.

Go to trade database log.
Right click - click plot open equity-Daily
Then load data base from drop down.
Run it --tick plot closed equity when looking at the open equity chart.

 

[howardbandy]

Hi Tech/a --

Nice car.

The post I made earlier did not describe the walk-forward process that goes with the out-of-sample testing. But I thought I posted that already?

Thanks,
Howard

 

[nizar]

Tradesim Open Equity.

Go to trade database log.
Right click - click plot open equity-Daily
Then load data base from drop down.
Run it --tick plot closed equity when looking at the open equity chart.

Thanks.

Tech do you get a warning/error when plotting open equity for weekly systems?

Im getting this warning:

"There are trades with non-daily periodicity.
It is recommended that Open Equity chart be generated only using trades with daily periodicity.
You may receive price invalidation warnings if Entry and Exit price checking has been enabled.
Do you wish to proceed to plot the Open Equity?"

Any ideas?

(Iv got it working fine for EOD systems -- and Iv got to say, the style of the open equity chart looks awesome!!!)

 

[tech/a]

Hi Tech/a --

Nice car.

The post I made earlier did not describe the walk-forward process that goes with the out-of-sample testing. But I thought I posted that already?

Thanks,
Howard

And your point is?

At some point you'll trade your system and it to will be walk forward only realtime.
Upon completion of a period that period of out of sample trading would have become in sample.

All your doing is testing snippets of data in isolation.
How long do you test out of sample data?
When is enough enough?
You tweek and twiddle from that out of sample test/s then test another---ad infinitum.

When is a system ready to trade?
At what point do you say yes I'm happy that my objectives have been met?
How do you know you have the optimum conditions and parameters?
When do you stop Tweeking and Twiddling.?

Your books on the way---Very interested in its contents.--Seriously!

 

[kaleon]

Hi Nizar,

I have attached a metastock file with the actual constituents of the ASX300 back in 1/1/2001.

Run tradesim on this data for the year 2001 only. Make sure all your exits occur on the last trading day of 2001. That way we can see what the actual profit or loss there was for the system.

To make tradesim exit on the last trading day, you have to find what the last trading day was on the daily chart or the weekly chart depending what chart your system uses.

Use this 'date filter'

StDay:=Input(" Start day",1,31,28);
StMnth:=Input("Start month",1,12,12);
StYear:=Input("Start year",1980,2020,2001);
EnDay:=Input(" End day",1,31,28);
EnMnth:=Input(" End month",1,12,12);
EnYear:=Input(" End year",1980,2020,2001);
If((Year() > StYear OR (Year()=StYear AND ((Month() >
StMnth) OR (Month() = StMnth AND DayOfMonth() >=
StDay)))) AND (Year() < EnYear OR (Year()=EnYear AND
((Month() < EnMnth) OR (Month() = EnMnth AND
DayOfMonth() <= EnDay)))) ,1,0);


and then in the tradesim formulae do this to your exit trigger


ExtFml("TradeSim.SetStartRecordDate",1 ,1 ,2001);
ExtFml("TradeSim.SetStopRecordDate",31 ,12 , 2001);

ExitTrigger:=If(Fml("date filter"),Fml("date filter"),Fml("your exit trigger"));

ExitPrice:=C;

Then post the results here so we can see how in performed in this bear period. Gives you a good idea how robust the system is.


Tradesim parameters should include

Transaction Cost rate:0.75%
Fraction risked capital 1.5%
max positions 10
position size limit:10%
portfolio heat limit 20%
portfolio limit 100%
margin requirement 100%
entry order control: stop order,unconditionally enter trade
exit order control: default
limit position size to a maximum of : 10%
include breakeven trades with losing trades

Now that should really test your system

 

[nizar]

Hi Nizar,

I have attached a metastock file with the actual constituents of the ASX300 back in 1/1/2001.

Run tradesim on this data for the year 2001 only. Make sure all your exits occur on the last trading day of 2001. That way we can see what the actual profit or loss there was for the system.

To make tradesim exit on the last trading day, you have to find what the last trading day was on the daily chart or the weekly chart depending what chart your system uses.

Use this 'date filter'

StDay:=Input(" Start day",1,31,28);
StMnth:=Input("Start month",1,12,12);
StYear:=Input("Start year",1980,2020,2001);
EnDay:=Input(" End day",1,31,28);
EnMnth:=Input(" End month",1,12,12);
EnYear:=Input(" End year",1980,2020,2001);
If((Year() > StYear OR (Year()=StYear AND ((Month() >
StMnth) OR (Month() = StMnth AND DayOfMonth() >=
StDay)))) AND (Year() < EnYear OR (Year()=EnYear AND
((Month() < EnMnth) OR (Month() = EnMnth AND
DayOfMonth() <= EnDay)))) ,1,0);


and then in the tradesim formulae do this to your exit trigger


ExtFml("TradeSim.SetStartRecordDate",1 ,1 ,2001);
ExtFml("TradeSim.SetStopRecordDate",31 ,12 , 2001);

ExitTrigger:=If(Fml("date filter"),Fml("date filter"),Fml("your exit trigger"));

ExitPrice:=C;

Then post the results here so we can see how in performed in this bear period. Gives you a good idea how robust the system is.


Tradesim parameters should include

Transaction Cost rate:0.75%
Fraction risked capital 1.5%
max positions 10
position size limit:10%
portfolio heat limit 20%
portfolio limit 100%
margin requirement 100%
entry order control: stop order,unconditionally enter trade
exit order control: default
limit position size to a maximum of : 10%
include breakeven trades with losing trades

Now that should really test your system

Hi kaleon.

Thanks for the date filter!
Good work. And thanks for the feedback.

I still havent got the (2001) ASX300 constituent list form you.

Im more than happy to run the test, and as a matter of fact, I'll go one better.

How about from 01-07-2001 until 01-03-2003?
Thats just about peak to trough of the "bear market". XAO lost 18% in this period.

2001 the market actually ended positive +6.98% for the year.

Also another question, where do you put the "0.75%" as a transaction cost percentage in TradeSim?

Thanks.

 

[rnr]

Hi Nizar,

I don't quite understand why the results of the second simulation have run on to 2005?

I must be losing the plot as it was only when I read kaleon's post that it twigged and I remembered reading a post about this on the TradeSim forum (TradeSim > Ideas and Suggestions > "Closing" prices of open trades) which may well negate the "date filter" to which kaleon makes reference.

You could try this perhaps - it should close out all OPEN TRADES at the StopRecordDate (or the day after) at the CLOSING price of the day!

ExitPrice:= If(ExitTrigger,OPEN,CLOSE);

Cheers,
rnr

 

[Sir Burr]

And your point is?

Tech, a question that I have been pondering:

Why is Monte Carlo Analysis better than Optimising/Walk Forward?

We've been lead down the Monte Carlo path with Tradesim but is it the better way to go?

What's the difference?

Cheers SB

 

[bingk6]

Why is Monte Carlo Analysis better than Optimising/Walk Forward?

We've been lead down the Monte Carlo path with Tradesim but is it the better way to go?

What's the difference?

Cheers SB

Hi SB,

Monte Carlo Analysis and Optimising/Walk Forward are not mutually exclusive. I see Optimising/Walk Forward as a single facet process whereas Optimising/Monte Carlo is a multi-facet process and is therefore more thorough.

 

[nizar]

Tech, a question that I have been pondering:

Why is Monte Carlo Analysis better than Optimising/Walk Forward?

We've been lead down the Monte Carlo path with Tradesim but is it the better way to go?

What's the difference?

Cheers SB

Hi SB.

Monte Carlo simulations are an exhaustive type of analysis.
I would do a monte carlo analysis (20,000-100,000 simulations) on both in-sample and out-sample data (ie. when doing a walk-forward analysis).

I probably havent understood your question properly

 

[nizar]

rnr.
Thanks for the code.

kaleon.
Results below.

System didnt do too flash -- in fact i have left this system and have gone for a weekly approach.

As I suspect both the EOD and Weekly systems did poorly in this period.

Also note that 1-2 years is not enough time to trade systems in which average holding time for a winner is 300-600 days.

The systems -- or in fact any system im working on -- is not designed to do well in bearmarkets. But still it should beat the overall market.

For Year 2001 (XAO +6.98%)


Monte Carlo Report

Trade Database Filename
C:\TradeSimData\EODSystemv2Kaleon.trb

Simulation Summary
Simulation Date: 15/09/2007
Simulation Time: 7:42:23 PM
Simulation Duration: 16.09 seconds

Trade Parameters
Initial Capital: $30,000.00
Portfolio Limit: 100.00%
Maximum number of open positions: 10
Position Size Model: Fixed Percent Risk
Percentage of capital risked per trade: 1.50%
Position size limit: 10.00%
Portfolio Heat: 20.00%
Pyramid profits: Yes
Transaction cost (Trade Entry): $44.00
Transaction cost (Trade Exit): $44.00
Margin Requirement: 100.00%
Magnify Position Size(& Risk) according to Margin Req: No
Margin Requirement Daily Interest Rate (Long Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Long Trades): 0.0000%
Margin Requirement Daily Interest Rate (Short Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Short Trades): 0.0000%

Trade Preferences
Trading Instrument: Stocks
Break Even Trades: Include with losing trades
Trade Position Type: Process all trades
Exit Order Type: Default Order
Minimum Trade Size: $500.00
Accept Partial Trades: No
Volume Filter: Reject Trades if Position Size is greater than
10.00% of the maximum traded volume
Pyramid Trades: Yes
Favour Trade Pyramid: Yes
Start Pyramid at any level up to level: N/A
Maximum Pyramid Level Limited to: N/A
Maximum Pyramid Count Limited to: N/A

Simulation Stats
Number of trade simulations: 5000
Trades processed per simulation: 741
Maximum Number of Trades Executed: 24
Average Number of Trades Executed: 18
Minimum Number of Trades Executed: 15
Standard Deviation: 1.21

Profit Stats
Maximum Profit: $3,371.95 (11.24%)
Average Profit: $1,064.02 (3.55%)
Minimum Profit: -$3,379.61 (-11.27%)
Standard Deviation: $1,003.91 (3.35%)
Probability of Profit: 84.28%
Probability of Loss: 15.72%

Percent Winning Trade Stats
Maximum percentage of winning trades: 68.75%
Average percentage of winning trades: 50.99%
Minimum percentage of winning trades: 26.09%
Standard Deviation: 6.23%

Percent Losing Trade Stats
Maximum percentage of losing trades: 73.91%
Average percentage of losing Trades: 49.01%
Minimum percentage of losing trades: 31.25%
Standard Deviation: 6.23%

Average Relative Dollar Drawdown Stats
Maximum of the Average Relative Dollar Drawdown: $3,183.46
Average of the Average Relative Dollar Drawdown: $516.52
Minimum of the Average Relative Dollar Drawdown: $87.16
Standard Deviation: $380.12

Average Relative Percent Drawdown Stats
Maximum of the Average Relative Percent Drawdown: 10.6115%
Average of the Average Relative Percent Drawdown: 1.7572%
Minimum of the Average Relative Percent Drawdown: 0.2924%
Standard Deviation: 1.2707%

Maximum Peak-to-Valley Dollar Drawdown Stats
Maximum Absolute Dollar Drawdown: $6,532.00
Average Absolute Dollar Drawdown: $2,781.43
Minimum Absolute Dollar Drawdown: $934.02
Standard Deviation: $846.15

Maximum Peak-to-Valley Percent Drawdown Stats
Maximum Absolute Percent Drawdown: 21.7733%
Average Absolute Percent Drawdown: 9.2714%
Minimum Absolute Percent Drawdown: 3.1134%
Standard Deviation: 2.8205%

 

[nizar]

For 01-07-2001 to 01-03-07 (XAO -18%)




Monte Carlo Report

Trade Database Filename
C:\TradeSimData\EODSystemv2KaleonWorst.trb

Simulation Summary
Simulation Date: 15/09/2007
Simulation Time: 10:33:06 PM
Simulation Duration: 21.63 seconds

Trade Parameters
Initial Capital: $30,000.00
Portfolio Limit: 100.00%
Maximum number of open positions: 10
Position Size Model: Fixed Percent Risk
Percentage of capital risked per trade: 1.50%
Position size limit: 10.00%
Portfolio Heat: 20.00%
Pyramid profits: Yes
Transaction cost (Trade Entry): $44.00
Transaction cost (Trade Exit): $44.00
Margin Requirement: 100.00%
Magnify Position Size(& Risk) according to Margin Req: No
Margin Requirement Daily Interest Rate (Long Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Long Trades): 0.0000%
Margin Requirement Daily Interest Rate (Short Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Short Trades): 0.0000%

Trade Preferences
Trading Instrument: Stocks
Break Even Trades: Include with losing trades
Trade Position Type: Process all trades
Exit Order Type: Default Order
Minimum Trade Size: $500.00
Accept Partial Trades: No
Volume Filter: Reject Trades if Position Size is greater than
10.00% of the maximum traded volume
Pyramid Trades: Yes
Favour Trade Pyramid: Yes
Start Pyramid at any level up to level: N/A
Maximum Pyramid Level Limited to: N/A
Maximum Pyramid Count Limited to: N/A

Simulation Stats
Number of trade simulations: 5000
Trades processed per simulation: 987
Maximum Number of Trades Executed: 43
Average Number of Trades Executed: 31
Minimum Number of Trades Executed: 21
Standard Deviation: 3.08

Profit Stats
Maximum Profit: $15,886.86 (52.96%)
Average Profit: $2,462.05 (8.21%)
Minimum Profit: -$8,992.86 (-29.98%)
Standard Deviation: $3,556.86 (11.86%)
Probability of Profit: 75.54%
Probability of Loss: 24.46%

Percent Winning Trade Stats
Maximum percentage of winning trades: 50.00%
Average percentage of winning trades: 21.19%
Minimum percentage of winning trades: 3.23%
Standard Deviation: 6.14%

Percent Losing Trade Stats
Maximum percentage of losing trades: 96.77%
Average percentage of losing Trades: 78.81%
Minimum percentage of losing trades: 50.00%
Standard Deviation: 6.14%

Average Relative Dollar Drawdown Stats
Maximum of the Average Relative Dollar Drawdown: $2,869.04
Average of the Average Relative Dollar Drawdown: $1,259.00
Minimum of the Average Relative Dollar Drawdown: $370.87
Standard Deviation: $395.45

Average Relative Percent Drawdown Stats
Maximum of the Average Relative Percent Drawdown: 10.5273%
Average of the Average Relative Percent Drawdown: 4.4158%
Minimum of the Average Relative Percent Drawdown: 1.2599%
Standard Deviation: 1.4398%

Maximum Peak-to-Valley Dollar Drawdown Stats
Maximum Absolute Dollar Drawdown: $10,297.40
Average Absolute Dollar Drawdown: $5,451.14
Minimum Absolute Dollar Drawdown: $1,766.96
Standard Deviation: $1,385.42

Maximum Peak-to-Valley Percent Drawdown Stats
Maximum Absolute Percent Drawdown: 34.3247%
Average Absolute Percent Drawdown: 18.1696%
Minimum Absolute Percent Drawdown: 5.8899%
Standard Deviation: 4.6200%

 

[howardbandy]

Greetings --

Walk forward is the process of selecting a series of time periods, each a combination of an in-sample period and a following out-of-sample period, performing a system building operation on the in-sample period, recording the results on the out-of-sample period, and analyzing the combined out-of-sample performance.

Monte Carlo is a general term describing an analysis method where pseudo-random changes are made to inputs to a system and the resulting outputs are analyzed. Monte Carlo can be used to study trading systems in several ways, including:
Perturbing the values of the arguments to a system to test for sensitivity.
Perturbing the data processed to test for sensitivity.
Reordering the sequence of trades to test for equity curve behavior.
Drawing simulated trades from a distribution to test for equity curve behavior.

It is possible to combine walk forward and Monte Carlo.

Thanks,
Howard

 

[howardbandy]

Hi Tech/a --

And your point is?

At some point you'll trade your system and it to will be walk forward only realtime.
Upon completion of a period that period of out of sample trading would have become in sample.

All your doing is testing snippets of data in isolation.
How long do you test out of sample data?
When is enough enough?
You tweek and twiddle from that out of sample test/s then test another---ad infinitum.

When is a system ready to trade?
At what point do you say yes I'm happy that my objectives have been met?
How do you know you have the optimum conditions and parameters?
When do you stop Tweeking and Twiddling.?

Your books on the way---Very interested in its contents.--Seriously!

Sorry for my flip answer -- I misunderstood your post.

I made two posts in the thread on System Robustness -- numbers 120 and 126, both on August 23. Those describe the process of validating a trading system.

There is no certainty that any given trading system will be profitable when traded with real money. All we can do is perform the validation steps, with the goal of increasing our confidence that the system will be profitable.

To me, that means in-sample and out-of-sample testing, walk forward processing, sensitivity analysis, and so forth.

Thanks,
Howard

 

[julius]

RE: Out of sample testing

Are the nature of the signals we select for a trading system subject to a natural bias toward performance? ie. a non-systematic process of optimization by the trader who develops the system.

If so, does this support the use of out of sample testing to validate the objectiveness of the design?

 

[nizar]

Hi Nizar,



I must be losing the plot as it was only when I read kaleon's post that it twigged and I remembered reading a post about this on the TradeSim forum (TradeSim > Ideas and Suggestions > "Closing" prices of open trades) which may well negate the "date filter" to which kaleon makes reference.

You could try this perhaps - it should close out all OPEN TRADES at the StopRecordDate (or the day after) at the CLOSING price of the day!

ExitPrice:= If(ExitTrigger,OPEN,CLOSE);

Cheers,
rnr

Actually Kaleons code and yours both didnt seem to work.
My latest exit is still in 2005 sometime.

I'll email David and see what he says.