Would you trade this system?

nizar said:

Thanks. Its next on my list, after Iv finished the book by your best friend Bob Pardo LOL.

Click to expand...

Hi Nizar,

Now that you have read it would you recommend Pardo's book on System Development / Testing??

Howard.

In your experience would you suggest
Fixed origin evaluation
OR
Rolling origin evaluation and why?
How do you evelute the period ahead that you should use in the evaluation?
If optimising at what point do you re calibrate?
How many out of series test periods are in your veiw enough to give confidence and at what point do you gain that confidence?

How do you do your out of sample testing,what software do you use?
Or are you simply walking forward through a data set?
How do you extrapolate single special events that fall within the out of sample test period/s.

I would argue that out of sample testing of models offers no guarentee that the in sample test results will perform any better or worse than the results returned within the parameters of those results obtained through multiple or Montecarlo testing going forward in realtime.

Howard.
I agree with Nick.

From my understanding, if the system had not been optimised and curve-fitted, then in-sample is just the same as out-sample data.

What i mean by the same is that you can draw the same conclusions.

For example, if i have an idea to test.
And i test it ONCE over a certain data set -- and i dont change anything thereafter, then the results from this initial run can be considered as an out-of-sample data.
Correct?

Chorlton -- I been busy with school (and testing). I havent finished Pardo's book yet. I will PM you when i have,

OK so the improvements that were required to be made to the system were:

*Increasing the number of trades
(I personally would like 20-30 per year)
*Loosening the stop -- 50% being exited by hitting the protective stop is far too high. We aim for 10-15%.
*Increased trade frequency as there were long periods of time without trading -- Opportunity cost issue and goes hand-in-hand with the first point.

Drawdowns at 15%ish were okay for me, as this system will be traded unleveraged.

Stevo -- OK i will try fixing the maximum number of open positions to 5,6,7,10, see if that improves things.

Shane -- i thought that scatter chart was standard in tradesim, apparently not, im not really that good at using Excel.

Open equity im having some problems getting it up on TradeSim. Iv been in communication with David (the developer) through email and he's going to get back to me.

I'll get back tonight with the results after changes were made.

Nizar,

rnr -- the trade log is coming.

Click to expand...

No mention of this in your latest post - any chance of a look?

Maximum number of open positions: 100

Click to expand...

Whilst this may be causing you a minor problem perhaps you need to look at the problems that may be caused by such a high percentage here.

Position size limit: 100.00%

Click to expand...

According to one of your charts the maximium number of trades at any point in time is 11 and quite often it's sitting around 6.

Cheers,
rnr

rnr.

I have made significant changes to the system.

The trade log of the original is attached.

Position size limit in the past when used it hasnt helped the results.

Changes made:
*Position sizing model changed to $1,600 risk per trade
*Initial stop changed to 20% of purchase price
*Exit changed to 250d EMA.
*Chose to favour pyramid.


Monte Carlo Report

Trade Database Filename
C:\TradeSimData\BlueprintMachinewithOoooomph20pcstop250.trb

Simulation Summary
Simulation Date: 14/09/2007
Simulation Time: 5:57:29 PM
Simulation Duration: 2567.08 seconds

Trade Parameters
Initial Capital: $30,000.00
Portfolio Limit: 100.00%
Maximum number of open positions: 100
Position Size Model: Fixed Dollar Risk
Capital risked per trade: $1,600.00
Position size limit: 100.00%
Portfolio Heat: 100.00%
Pyramid profits: Yes
Transaction cost (Trade Entry): $44.00
Transaction cost (Trade Exit): $44.00
Margin Requirement: 100.00%
Magnify Position Size(& Risk) according to Margin Req: No
Margin Requirement Daily Interest Rate (Long Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Long Trades): 0.0000%
Margin Requirement Daily Interest Rate (Short Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Short Trades): 0.0000%

Trade Preferences
Trading Instrument: Stocks
Break Even Trades: Process separately
Trade Position Type: Process all trades
Entry Order Type: Default Order
Exit Order Type: Default Order
Minimum Trade Size: $500.00
Accept Partial Trades: No
Volume Filter: Reject Trades if Position Size is greater than
10.00% of the maximum traded volume
Pyramid Trades: Yes
Favour Trade Pyramid: Yes
Start Pyramid at any level up to level: N/A
Maximum Pyramid Level Limited to: N/A
Maximum Pyramid Count Limited to: N/A

Simulation Stats
Number of trade simulations: 20000
Trades processed per simulation: 4111
Maximum Number of Trades Executed: 228
Average Number of Trades Executed: 215
Minimum Number of Trades Executed: 195
Standard Deviation: 10.45

Profit Stats
Maximum Profit: $689,298.81 (2297.66%)
Average Profit: $593,946.77 (1979.82%)
Minimum Profit: $450,261.55 (1500.87%)
Standard Deviation: $82,742.05 (275.81%)
Probability of Profit: 100.00%
Probability of Loss: 0.00%

Percent Winning Trade Stats
Maximum percentage of winning trades: 46.26%
Average percentage of winning trades: 44.60%
Minimum percentage of winning trades: 41.36%
Standard Deviation: 1.68%

Percent Losing Trade Stats
Maximum percentage of losing trades: 58.64%
Average percentage of losing Trades: 55.40%
Minimum percentage of losing trades: 53.74%
Standard Deviation: 1.68%

Average Relative Dollar Drawdown Stats
Maximum of the Average Relative Dollar Drawdown: $3,026.46
Average of the Average Relative Dollar Drawdown: $2,457.74
Minimum of the Average Relative Dollar Drawdown: $2,106.96
Standard Deviation: $315.17

Average Relative Percent Drawdown Stats
Maximum of the Average Relative Percent Drawdown: 1.3266%
Average of the Average Relative Percent Drawdown: 1.0801%
Minimum of the Average Relative Percent Drawdown: 0.8874%
Standard Deviation: 0.1309%

Maximum Peak-to-Valley Dollar Drawdown Stats
Maximum Absolute Dollar Drawdown: $28,562.26
Average Absolute Dollar Drawdown: $24,614.07
Minimum Absolute Dollar Drawdown: $18,405.32
Standard Deviation: $3,295.07

Maximum Peak-to-Valley Percent Drawdown Stats
Maximum Absolute Percent Drawdown: 15.3609%
Average Absolute Percent Drawdown: 15.3609%
Minimum Absolute Percent Drawdown: 15.3609%
Standard Deviation: 0.0000%

Profit Summary
Profit Status: PROFITABLE
Starting Capital: $30,000.00
Finishing Capital: $677,739.52
Maximum Equity/(Date): $647,739.52 (12/10/2005)
Minimum Equity/(Date): -$4,608.27 (8/10/1992)
Gross Trade Profit: $750,738.03 (2502.46%)
Gross Trade Loss: -$102,998.51 (-343.33%)
Total Net Profit: $647,739.52 (2159.13%)
Average Profit per Trade: $3,114.13
Profit Factor: 7.2888
Profit Index: 86.28%
Total Transaction Cost: $18,304.00
Total Slippage: $0.00
Total Trade Interest: $0.00
Daily Compound Interest Rate: 0.0643%
Annualized Compound Interest Rate: 26.4368%

Trade Statistics
Trades Processed: 4111
Trades Taken: 208
Partial Trades Taken: 0
Trades Rejected: 986
Winning Trades: 96 (46.15%)
Losing Trades: 112 (53.85%)
Breakeven Trades: 0 (0.00%)

Largest Winning Trade/(Date): $32,575.40 (12/10/2005)
Largest Losing Trade/(Date): -$2,734.90 (11/08/1999)
Average Winning Trade: $7,820.19
Average Losing Trade: -$919.63
Average Win/Average Loss: 8.5036

Trade Breakdown Long and Short Trades Long Trades Short Trades
Normal Exit: 176 (84.62%) 176 (84.62%) 0 (0.00%)
Protective Stop: 21 (10.10%) 21 (10.10%) 0 (0.00%)
Open Trade: 11 (5.29%) 11 (5.29%) 0 (0.00%)

Total Trades: 208 (100.00%) 208 (100.00%) 0 (0.00%)

Trade Duration Statistics Winning and Losing Trades Winning Trades Losing Trades
Maximum Trade Duration: 1784 (days) 1784 (days) 492 (days)
Minimum Trade Duration: 2 (days) 26 (days) 2 (days)
Average Trade Duration: 365.39 (days) 626.31 (days) 141.74 (days)

Consecutive Trade Statistics
Maximum consecutive winning trades: 24
Maximum consecutive losing trades: 18
Average consecutive winning trades: 4.17
Average consecutive losing trades: 4.87

Trade Expectation Statistics
Normalized Expectation per dollar risked: $1.89
Maximum Reward/Risk ratio: 19.82
Minimum Reward/Risk ratio: -1.67
Average Positive Reward/Risk ratio: $4.76
Average Negative Reward/Risk ratio: -$0.56

rnr -- position size limit using 100% hasnt made the simulations any worse (I have tried 20,25,30).

I figured if theres only a few stocks breaking out no harm in holding only these stocks and pyramiding into them like a champion.

Is the large consecutive losers a concern for anybody?
$1,600 risk per trade i realise is VERY HIGH initially.

A long string of losses to begin with and its goodbye.
Though this didnt happen in 20,000 simulations.

Maximum number of open positions, because of the money management model chosen, at the peak reached more than 40.

Is this a concern for anybody?

But we are on the right track so good news i guess as we:
*Managed to increase the number of trades
*Decrease number of exits from the maximum stop loss (only 10%)
*Managed to still report good figures for profit and max.DD.

Some more feedback would be appreciated.

Some charts.

Note: The exploration was run from 01-07-1992 until 30-09-2002.

MS/Tradesim seems to take this as STOP taking entries after 30-09-2002.
BUT the trades that are open, are left to run their course until the exit is hit.

So thats why we have upto 2005 on the yearly profit charts and on the closed equity chart, even though no new entries were taken after the stop date in 2002.

Even if you cut off at 2002, thats 400k, so 30%pa.

Looking at the Trade Database Manager in TradeSim, it seems in this period UTB, BLD, ORG, GNS were the standouts. The system pyramided into them several times and each parcel made >100% profit. Small parcels though. But thats what happens when you fix your risk to a dollar amount.

Hi Nizar

I wouldn't suggest using a fixed $ risk. Basically you need a money management algorithm that increases the dollars risked as your equity grows and decreases it as you lose. This basically leaves you with fixed fractional percentage risk based (ie risk 2% of equity per trade) or fixed percentage allocation money management models (ie allocate 10% of capital per trade).

I might suggest trying something like 15 % portfolio heat, 0.5% risk for a pyramiding system and see how you go. I'd also consider for the asx 200/300suggesting a closer initial stop such as 15% and then pyramid every 15% rather than using a trigger as a pyramid point. If aggressively pyramiding doesn't work then probably the system has a poor dependency and pyramiding should be dispensed with.

Hope this helps

Cheers

Shane

nizar said:

Is this a concern for anybody?

Click to expand...

Re: 40 concurrent open positions...

I think the standard reponse here is that beyond a certain point you start to move toward having too great of a representative sample of the index, and your results will become anchored to the index rather than outperforming it.

I have personally found via some Amibroker optimisations that 8-15 is an effective range. Having fewer seemed to improve CAGR but tended to increase Max DD. Having more improved Max DD, without disproportionately affecting CAGR...this is what I was after.

Would you trade this system?

No. 5.5 trades/per year doesn't seem enough trades (backtested) to make me feel confident with this system. Don't ask me how many would though, maybe 5-10x that!

Greetings --

Let me try a slightly different explanation of why I think it is necessary to perform an out-of-sample test of any trading system.

Imagine that all the possible trading systems have been described and there is a piece of paper with the computer code and description for each one.

One system might be "buy when two moving averages cross" and another might be "buy a specific number of days before options expiration" and so forth -- there are thousands.

Every system has formal parameters associated with it. The moving average crossover system has several that we can identify easily. Which type of moving average? Simple, exponential, weighted, adaptive, ... How many bars in the first moving average? 1, 2, 3, ... How many bars in the second moving average? 1, 2, 3, ... And so forth. Every characteristic of the system that can be changed is a parameter. When the design and testing of the system has been completed, there will be one value chosen for each of the parameters. Those values represent the actual arguments that will be used when the system is run.

Each system description is put into a separate drawer in a large room. In the drawer, along with the description of each system are a large number of marbles. Each marble represents one of the possible sets of arguments that go with that trading system. Pretend this is the drawer for the moving average crossover system. One marble has "simple, 2, 20" another "exponential, 3, 45" another "weighted, 7, 2" and so forth. Every possible combination has its own marble. The drawer has Every possible combination -- those that encapsulate some characteristic of the market that will trade well, along with those that are curve-fit. There may be many that trade well, or there may be none. There are certainly many that are curve-fit. We cannot tell which are which yet, because they must be applied to a specific time series and evaluated.

In order to trade any of these systems, I need the description and code for the system, and the values for the arguments. I will be applying that code and that set of arguments to a specific ticker and a specific time period.

As a system designer, I walk into the room and go to the drawer marked "Moving Average Crossover." I pull the drawer open, read the description of the system and pick out one marble. There is nothing hidden -- I can read what is on the marble before I pick it. If I wish, I can specify which marble I want -- "Exponential, 5, 40" -- and pick it.

With my code and arguments in hand, I turn to my computer, load the set of data I want to use, and run the system. This is the in-sample test. If I like the results, I can either run another computer test, or begin trading live. Either way, this is an out-of-sample test.

If I think there might be another combination that will work better, I can go back to the drawer and select another marble -- either by asking for a specific one or by picking one at random -- and run another in-sample test. I can choose which of the two results I like better, according to my objective function. I have an objective function, whether I have formally defined it or not. Whenever I make a choice from among alternatives, I am applying my objective function and ranking the alternatives.

Whether I pick one marble, or two, or four, or take every marble in the drawer -- I have no way of knowing whether I have picked a curve-fit marble or a good marble. And if it is a good marble, I have no way of know whether it is the best marble until I run a test using each marble on the ticker and data I want to trade. So I run all those tests and choose the best, according to my objective function. If I have run the tests over all the data I have, I have test results, but no way of judging whether the marble is a curve-fit marble or a good marble. The out-of-sample test gives me a way to evaluate that.

Whether the out-of-sample test is run on data that I have reserved for that purpose, or whether the out-of-sample test is run on live data using real money, there IS an out-of-sample test.

My point is that I can perform the out-of-sample test myself for the cost of a little electricity, or the market can perform it for me using real money.

You are each free to use whichever method of performing your out-of-sample testing you wish. But do not be fooled into thinking that since you picked just one marble out of the drawer that that marble is any more likely to be a good marble than if you had searched through all of them.

Thanks for listening,
Howard

Shane Baker said:

Hi Nizar

I wouldn't suggest using a fixed $ risk. Basically you need a money management algorithm that increases the dollars risked as your equity grows and decreases it as you lose. This basically leaves you with fixed fractional percentage risk based (ie risk 2% of equity per trade) or fixed percentage allocation money management models (ie allocate 10% of capital per trade).

I might suggest trying something like 15 % portfolio heat, 0.5% risk for a pyramiding system and see how you go. I'd also consider for the asx 200/300suggesting a closer initial stop such as 15% and then pyramid every 15% rather than using a trigger as a pyramid point. If aggressively pyramiding doesn't work then probably the system has a poor dependency and pyramiding should be dispensed with.

Hope this helps

Cheers

Shane

Click to expand...

Thats what i thought to.
So much so that i totally ignored the other options.

I never thought the results would be enhanced with a fixed $ stop.

Sir Burr said:

Would you trade this system?

No. 5.5 trades/per year doesn't seem enough trades (backtested) to make me feel confident with this system. Don't ask me how many would though, maybe 5-10x that!

Click to expand...

How about 2nd set of results with 208 trades?

Hi Nizar

Can I ask what is your universe of shares that you are testing over? If it is a group of shares such as asx 200 then there may only be a small number of those shares that were trading in 1992 :-(

Cheers

Shane

Hi ASX G

Re: 40 concurrent open positions...

I think the standard reponse here is that beyond a certain point you start to move toward having too great of a representative sample of the index, and your results will become anchored to the index rather than outperforming it.

I have personally found via some Amibroker optimisations that 8-15 is an effective range. Having fewer seemed to improve CAGR but tended to increase Max DD. Having more improved Max DD, without disproportionately affecting CAGR...this is what I was after.

Click to expand...

This depends how the system entry selection is constructed. If you have a single entry and exit than the index may be reflected unless you have a filter eg price that restricts the entries to a particular group of shares. Then it would probably be indicative of that sub set of shares movements.

If you have a system that pyramids aggressively into a trend then you may have forty open positions but in only five shares as an example.

Cheers

Shane

Shane Baker said:

This depends how the system entry selection is constructed. If you have a single entry and exit than the index may be reflected unless you have a filter eg price that restricts the entries to a particular group of shares. Then it would probably be indicative of that sub set of shares movements.

If you have a system that pyramids aggressively into a trend then you may have forty open positions but in only five shares as an example.

Click to expand...

I knew someone was going to say that! Agree 110%.

Interesting re: the pyramiding, I never thought of it that way. Seems then that it comes down to whether Nizar is pyramiding in this instance.

ASX.G

I might be mistaken but I thought Nizar was pyramiding.

I figured if theres only a few stocks breaking out no harm in holding only these stocks and pyramiding into them like a champion.

Click to expand...