Trading Model Metrics

[rb250660]

With regards to trading model development, what metrics are the most important to you? The discussion assumes proper model development procedures have been followed.

Lately I have been spending a lot of time on the development of new models. I look at the following things:
- Net profit
- CAGR distribution
- MaxDD distribution
- CAGR @ n%ile / MaxDD @ n%ile (percentiles for CAGR and MaxDD are not necessarily the same)
- Number of trades
- Percent winners and losers
- Bars held
- Payoff ratio
- Trade return distribution, average and standard deviation
- Trade average return / standard deviation
- Profit/MFE, Loss/MAE
- Average MAE/Average MFE
- Daily return distribution and average
- Rolling annual returns (monthly)
- Equity curve and draw down characteristics
- Exposure curve
- Any outliers in trades, drawdowns, etc. These get exposed pretty quickly from the timeline plots and distributions

It really depends on what type of model I am making but these are on the top of y list of things to focus on with every model.

 

[Gringotts Bank]

The curve itself tells you a lot - first thing I look at. Then Sharpe, Win% (>66% pref for a reversion system) and outliers.

 

[howardbandy]

The curve itself tells you a lot - first thing I look at. Then Sharpe, Win% (>66% pref for a reversion system) and outliers.

Hi GB, and all --

When we develop trading systems, we are fitting a model to a set of historical data, hoping that the patterns our rules find are signals and not noise, and that they continue from the in-sample period and its data to the out-of-sample period and its data. If the development platform is a traditional one, such as AmiBroker or trade station, the result of the fitting is a single set of trades in sequence. These form an equity curve and a set of trade metrics -- such as terminal wealth, maximum drawdown, etc.

If our system (model plus data) is a good one, and the signal persists into the future, the best we can hope for is that the distribution of trades in the OOS is the same as the distribution of trades in the IS. That is, that the distribution is stationary. We cannot expect the order of the trades in the future to be the same as the order of trades in the past. Just that the distribution is the same.

Each equity curve is one data point -- one instance of a set of trades drawn from that distribution. The trades come from the distribution are random and their order is arbitrary. We see one individual data point drawn from a distribution of thousands of such data points -- each one of which is an equally likely equity curve.

When we are testing many alternative models, each with adjustable rules and parameters, each test results in one specific equity curve data point. Each is one individual equity curve that is the only representative of the distribution of equity curves for that system that we can see. It is one individual drawn from a population of many. The good equity curves draw our attention and we investigate further. The specific equity curve that caused us to stop and look in more detail is always above average for the distribution from which it was drawn. Metrics drawn from that set of trades without the specific order are more indicative of future performance than metrics that are order dependent. (Be certain trades are equally sized, and that no position sizing was used at this point.)

Metrics that are independent of order include percentage of trades won versus lost, number of bars held, percentage gained or lost per constant unit of trade, MFE, MAR, etc. These are generally good metrics to look at.

Metrics that are dependent on order include maximum drawdown and any that would change if the trades changed sequence. These metrics are misleading.

-----------------

Begin by defining your personal risk tolerance. Say it is desire to hold losses greater than 20% to a chance of 5% or less. You only need to do this once, then refer back to it as you evaluate alternatives.

For each alternative -- each trading system -- create a distribution of those trades that are the best estimate of future trades. The in-sample trades that were produced during testing and created the equity curve that got our attention are always overly optimistic. Depending on the type of model and the degree to which the distribution of trades is stationary, the in-sample results range from "a small chance of being useful" to "completely worthless and totally misleading." Assume worthless and ignore in-sample results. Rather, use out-of-sample results from unbiased validation period.

Using the distribution of the best estimate set of trades, draw many sets of trades, retaining their sequence, each of which represent the forecast horizon you want to test -- say two years. Each series of trades produces an equity curve. From these thousand or so equally likely equity curves, estimate the maximum drawdown. Do this by forming a distribution of maximum drawdowns. Compare that distribution with your personal risk tolerance.

Adjust the position size (percentage of account used for each trade [for stocks]) and rerun creation of the distribution of drawdown. Repeat until the probability of a drawdown that exceeds your tolerance matches your willingness to have a drawdown that large. That number is safe-f -- the maximum safe position size.

Rerun again with position size set to safe-f. This time gather the final equity of the sequences of trades and form the distribution of that. Translate from final equity to compound annual rate of return (CAR). Look at a consistent point on that distribution -- say the 25th percentile. The value at that point is called CAR25.

CAR25 is a universal metric.

Given two or more alternative uses of funds, the one with the highest CAR25 should be traded. The position size for the next trade should be safe-f.

After each trade, add the trade just closed to the best estimate set of trades. Rerun the simulations to compute a new safe-f and CAR25. (Sliding window and associated techniques are used to keep the best estimate set current.)

Do the same for whichever alternative trading systems might be considered, using the "shadow trades" that they would have produced to create their own best estimate sets and distributions.

Trade-by-trade -- day-by-day -- trade the single best system. The alternative that has the highest CAR25 that day.

--------------

To be clear. You can compute a single universal metric -- CAR25 -- that measures the risk-normalized profit potential of any set of trades. There is no better metric and no need to look further.

Best, Howard

 

[howardbandy]

The previous long post described a procedure that took many test runs and adjustments to arrive at safe-f and compute CAR25.

If a traditional development platform is being used, the OOS run produces the first best estimate set. That is exported and imported into a statistical platform, such as Excel, where some of the adjustments are made. Then back and forth between the trading system platform and the statistical platform until CAR25 has been computed. I have published a complete step-by-step procedure. It is a little tedious.

If Python is used for trading system development, the statistical analysis can be included in the development program. CAR25, computed using the OOS best estimate set of trades, comes out as the metric from a single run. I have published this procedure as well.

Best, Howard

 

[howardbandy]

My posts caused one reader to send me a private message. Edited so as to be family-friendly, it asked whether I was claiming safe-f and CAR25 to be better than ...

Yes. Risk-normalized safe-f and CAR25 are better metrics, indicators of system health, estimators of position size, and system selection indicators than Kelly formula, optimal-f, Sharpe ratio, and modern portfolio theory.

Do the math.

Best, Howard

 

[howardbandy]

That same message referred to some publications and asked my opinion. When I have published a book that is very similar to some other book, in most cases it is unethical for me to criticize that other book. And, even when I do not have a book in direct competition with another, I prefer to say nothing about poor books rather than publish derogatory reviews. There have been a few exceptions for extremely bad books.

That said, several topics I address and points I regularly make are important. Without being critical of any particular book or article I'd like to review some of them here.

When an article, book, or presentation omits discussion of these topics, that does not mean they can be ignored. It does mean the author is making implicit assumptions and allowing the defaults rather than making explicit assertions and stating parameters and implications. Some cases in point.

... Ignoring out-of-sample validation and basing forecasts on in-sample results assumes that the future will resemble the past. We hope it does. We need it to. But the only way to have confidence that it might is through use of the scientific method -- observe, think, test, validate.

... Decision tree models (the kind implemented by traditional trading system development platforms) are very susceptible to over-fitting to in-sample noise, and poor generalization and out-of-sample forecasting. Validation is very important.

... Using a period earlier than the in-sample period as a validation period might give some confidence that the past resembles the present, but tells us nothing about the future.

... Using the results of a single test run ignore the fact that it is one visible data point that was drawn from a very large population of unseen data points.

... Discussing individual tests without acknowledging they are representatives of distributions loses the opportunity to analyze the distribution.

... Assuming the results of a single test run, including sequence-dependent results, can be used to estimate future performance assumes that the trades that occurred in the test will occur again in the future, and in the same order.

... Using long time periods and long lookbacks for indicators assumes stationarity.

... Computing position size in the trading model assumes there is a strong serial correlation among signal patterns.

... Ignoring the trading management system removes the opportunity to have rule-based deterrmination of system health and position size

... Ignoring intra-trade drawdown invites discretionary exits from rule-based systems.

... Ignoring risk assumes systems always recover from drawdowns.

... The characteristics of the data series are ignored. A trading system is a combination of a model and some data. Too often, we begin with an arbitrary data series and focus on developing a model that trades it well. Even before beginning to model, we can assess the likely profit potential of the data series itself. Very few data series have the characteristics that are suited to trading them safely and profitably.

... Creation of portfolios through extensive search of issues is optimizing over the symbol space.

... Use of broad metrics, such as advance-decline, without accounting for membership changes assumes the membership always had the constituents it currently has.

... As you will have gathered from my material, most of rule-based trading is math. Whatever is being presented should be precise enough for the reader to generate repeatable result and to be testable. If the presenter does not show the math, do it yourself.

And so forth.

End of rant,
Thanks for listening,
Best, Howard

 

[Willzy]

Hi All,

Having purchased, studied and applied many of his teachings, I completely agree with Howard's approach and in particular the use of CAR25 at safe-f as a universal metric.

However, the primary issue for many system developers including myself is that CAR25 is the last in a long line of steps in the development process. The overall objective of the system development process is to find a system which produces the best distribution set of trades, whilst adhering to proper development procedures (inSample, outOfSample, walkForward etc...) Once we have that distribution of outOfSample trades only then can we find CAR25 @ safeF.

Unfortunately we cant just click Optimize and have Amibroker rank each alternative parameter set according to CAR25 @ safeF.

The answer to finding the system which produces the best distribution, which I'm sure Howard will agree with is to create a Custom Metric using Amibroker's custom backTest procedure. For further details I refer everyone to Howard's publications as he explains it far better than I can.

During my system development process, I allow Amibroker to take every position possible over the data set and allow each trade to be of the same size (eg $10000 per trade). The metrics I then look at to place into my custom metric are as follows.

1. Accuracy
2. averageTrade / averageLoss
3. number of Trades per Year
4. MDD95 (the 95th percentile of Maximum Drawdown)

Some empirical calculations are required to penalize or reward the system based on the above metrics and I will adjust the penalties and rewards to suit my own individual trading requirements.

I then use the customMetric to rank all alternative systems in a walkForward analysis to find my distribution of trades. I then conduct my separate monteCarlo analysis in MATLAB to find CAR25 @ safeF.
Excel or Python work well also.

----- side note -----

Amibroker can now conduct monteCarlo analysis and return values such as CAR25 and MDD95. These values are based on the position sizing method set in the backTest formula itself.
For example
1. fixedFraction = 0.2 ; (thus 20% on each position) or
2. positionValue = 10000 (thus each trade taken is worth $10,000)

One must be careful to ensure that Amibroker settings are correct with regards to the monteCarlo simulation....

Whilst this simulation cannot calculate CAR25 @ SafeF where MDD95 is limited to a set value. I have found that provided a fixedFraction position sizing method is used then CAR25 divided by MDD95 can be a useful metric.

perhaps call it CAR25MDD95

---- side note #2 ----

MonteCarlo analysis can be a very useful tool in the system developers arsenal, however it must be used with care!

If you are developing a trading system which allows multiple positions to be open at any one time, then a trade by trade monteCarlo analysis will under estimate risk due to serial correlation ( the opposite of "a high tide floats all boats" )
If this is the case you should use daily changes in equity for the monteCarlo simulation. Amibroker also has a setting for this type of simulation.

I hope this helps.

Regards
Matt

 

[Willzy]

---- side note # 3 -----

If you think you have created a robust system for trading then I've attached a useful piece of code to help you check!

bootStrap = 2;
if( bootStrap > 0 )
{
PositionScore = MTRandom();

if( bootStrap > 1)
{
MC = Optimize( "MC", 1, 1, 30, 1 );
}
}

this is essentially a bootstrap cycle which will shuffle your trades if more than one signal is available on a given day.

 

[Boggo]

Excellent bit of code Matt, creates some interesting outcomes and is an essential part of any backtest imo.

 

[howardbandy]

I agree that custom objective functions can be constructed that guide development toward systems that are more likely to perform well in the future. The metrics in Willzy's post are good -- except for MaxDD95. DD95 is the drawdown at the 95th percentile of the distribution of equity based on analysis of the best estimate set of trades. At the time the signal generation model is running, there is only one equity value and that is of an in-sample test. The distribution, based on out-of-sample trades, is not yet known.

Be certain you are working with the trades that are the best estimate of future performance. Those are the out-of-sample trades from one time test of data that is more recent than in-sample data. In-sample trades always over estimate profit and under estimate risk. Multiple passes through the test/adjust cycle move previously out-of-sample data into in-sample data.

As Matt says -- Monte Carlo analysis is an important, powerful, and valuable tool in validation and estimation of future performance. Use it wisely.

-------------

I see a reference to fixed fraction in the signal generation model. Fixed fraction is position sizing. I recommend against doing that. Rather, use constant-sized trades, marked to market daily, managed daily, and let the trading management system determine safe-f (which is a fractional position sizing technique) by analysis of trades.

Please recognize that there are Two systems required for trading an issue.
... One, the trading model or signal generation model, has prices as its input and signals to be long or flat for the next evaluation period as its output. The only thing the signal generation model does is look for patterns that precede profitable trades. It has no data to indicator what the maximum safe position size should be.
... Two, the trading management model, has trades as its input and safe-f and CAR25 as its output. The trading management model has sole responsibility for analyzing risk and determining position size. Risk and position sizing cannot be done in the trading / signal generation model. Moving position sizing into the signal generation model removes it from the trading management model. If that is done, the control over adjusting position size to reflect recent trading is lost.

Depending on the trading system development platforms being used, it may be necessary to export the out-of-sample trades from a walk forward run to a disk file, then use a different model for trading management. One way to do that uses Excel. The code published in the "Modeling Trading System Performance" book does that.

If you are using Python, or the equivalent, everything can be done within a single program in a single run. The code published in "Quantitative Technical Analysis" does that.

In either case, be aware of my caution above. There is always a possibility that results are good by luck rather than by identifying a valid signal among the noise. Multiple passes through the two step process contaminates the originally out-of-sample data and makes it part of the in-sample data. Running multiple walk forward runs, modifying the rules and parameters between runs, will eventually find a trading model that appears to give good result when processed by the trading management model -- but it may be just a two step overfitting. It is loading the dice that will eventually suggest that a good system result has been found, but it is more likely that it is a spurious fit.

-------------

Posts often mention multiple signals on a given day. If the system is long/flat for a single issue, multiple signals from multiple rules should be consolidated within the afl and resolved to a single signal.

If multiple signals relate to choices of which issue of many should be traded, the analysis becomes very complex. Probably beyond the analysis capabilities of ordinary mortals, even working with Monte Carlo tools. Please reread my many comments in various videos, posts, and books about single issues versus portfolios. Portfolios are overrated. I strongly recommend developing individual systems, each of which trades a single issue long/flat. Use CAR25 during the daily mark-to-market and trading management analysis to choose among them, then trade the one that ranks highest. One.

For example, assume you have two systems. One trades XLV long/flat and as of today's close, based on trades its signal generation model has given, has a CAR25 of 15.0. The other trades XLP long/flat and as of today's close, based on its signal generation model, has a CAR of 8.0. Why use any funds to trade the lower ranked XLP? Trade XLV using its safe-f position size, then reevaluate tomorrow.

CAR25 is risk-normalized. Normalized to have the same probability of a drawdown of a given percentage. It is designed so that choices can be made solely using profit potential because alternatives have identical risk -- apples to apples.

Best, Howard

 

[Willzy]

Hi all,

My apologies for the reference to MDD95, if you are attempting to design a system to trade a single issue or stock then indeed, Howard is correct.

However I should have specified that my research has been into the development of trading systems which do indeed trade several issues within a portfolio.

With regards to Howard's statement:
" If multiple signals relate to choices of which issue of many should be traded, the analysis becomes very complex. Probably beyond the analysis capabilities of ordinary mortals "

He is most likely correct, however it is the field in which I choose to conduct my research attempting to trade profitably and it is indeed most difficult! That said I have found limited success in the area.

With respect to how I chose to apply MDD95 to the system development specifically during optimizations in the walk forward process:

1) All trades available to the system are taken to ensure the optimization run is looking at the distribution of trades as opposed to a single backtest run.
2) All trades are valued the same (eg $10000 on each trade)
3)

 

[Willzy]

accidentally submitted this before it was finished and it appears that there is a 20minute cut off period for editing posts! .... Grrrrr

.........

3) monteCarlo analysis is conducted within the amibroker custom backTest procedure with each trade of equal value (ie $10,000) I like to run at least 1000 monteCarlo runs on each set.

If we were to evaluate the monteCarlo distribution of final equity of each optimization run, we would find a bias towards the system which produces the most trades. This occurs because amibroker defaults the number of trades in each run to the total number of trades within the distribution (or backTest). Thus Final Equity is meaningless.

However; the distribution of Maximum Drawdown from the monteCarlo simulation is not affected in the same way. Indeed the variance of MDD95 appears to narrow as the number of trades increases. Whilst it doesn't appear to approach a limit, it may still be useful.

In this case I am using MDD95 at constant trade value to penalize systems which produce less desirable trade distributions. NOT for position sizing calculations!

When the walk forward optimization is complete. I then export the trades into another program (MATLAB, EXCEL, PYTHON) for further analysis and indeed position sizing using CAR25 at safeF.

Again my apologies for the initial confusion, and perhaps the sore heads from reading this post!

Cheers
Matt

 

[howardbandy]

Hi Matt --

Thanks for the explanation.

Please take my cautions into account:

... Only out-of-sample results give estimates of future performance.
... Repeated optimization and walk forwards reduce the "out-of-sample-ness" of the validation data and results.
... The issues available for inclusion into a portfolio need to be chosen in advance by a rule-base technique.
... Portfolios are less efficient -- less profitable for a given risk -- than systems that trade individual issues.

All of that said, if you have a technique that works for you, please continue with it.

Best regards, Howard

 

[rb250660]

Howard,

How would I dynamically position size a model that may hold multiple positions at any one time?

I have been experimenting with the following for my mean reversion strategies (~3-5 day hold, 70-75% win rate, 120 trades a year):
1. Record daily changes in equity,
2. Feed into bootstrap allowing it to only sample with replacement the last X days that cover the daily changes in equity for the last N trades (I use 30 trades, so about 63 trading days),
3. Plot the CDF's for MaxDD and CAR. Iterate the simulation by changing safe-f until you converge to your personal risk tolerance statement,
4. Say you arrive at a safe-f of 0.73 and your base system uses 10 positions, adjust your position numbers to 10/0.73 = 14 positions and still trade 100% of your funds.

Point 4 is a bit 'loose' in my opinion. However I have done some tests in AmiBroker where (using the example above) I change my number of positions to 14 based on a safe-f of 0.73 (still trading 100% of funds) and the resulting MaxDD at my personal risk tolerance arrives almost precisely at the desired values. Note that it is very rare that I am fully invested so I essentially don't take additional positions by increasing the number to 14. This means I would be trading at fraction 0.73 anyway. So I do take some confidence from that all be it in hindsight. (If the model was usually fully invested most of the time I would assume the method of increasing position numbers to be totally invalidated.)

Based on the example with a safe-f of 0.73 how do you cut that 73% of your total capital up and trade into the future? Is this as close as it gets for a valid approach for this type of model? I'd appreciate your feedback and thoughts on this issue.

If multiple signals relate to choices of which issue of many should be traded, the analysis becomes very complex. Probably beyond the analysis capabilities of ordinary mortals, even working with Monte Carlo tools.

Maybe this is your answer. However this is how I currently choose to trade, multiple ASX stocks at any one time. My future research is looking towards Commodity Futures and Index ETF's in the US to trade as single issues using their own models and management systems.

As a side note for those interested, the results AmiBroker's Monte-Carlo Simulation gives based on daily equity changes compared to those I get in MATLAB (Statistics and Machine Learning Toolbox) and are within 1% across all percentiles which gives me confidence in my code, but not necessarily in my method.

Cheers
rb

 

[rb250660]

Getting back to the metrics, during development I find it somewhat helpful to look at signal efficiency from the perspective of MFE retained as profit and MAE retained as loss. Although any changes made to your model must consider you objectives such as overall profitability. For example, if you change your entry and exit to maximize these metrics you might be doing so at the peril of trade frequency resulting in less opportunities to exploit your edge in turn giving lower overall profitability and longer duration in draw down. And therein lies a well constructed objective function, for me that is balancing profitability with a model that is 'comfortable' for me to trade.

 

[howardbandy]

Hi RB --

Analyzing portfolios is complicated for several reasons.

... One is composition of the mix of issues. What list they are chosen from, by what rules, how sensitive the portfolio is to slight changes in issue price history, model parameter, etc.

... Another is correlation between day-by-day individual returns that make up day-by-day portfolio returns. Monte Carlo analysis depends on individual data drawn be independent and identically distributed (iid). Portfolio returns are not iid.

So, if you want to continue to trade a portfolio, do the best you can with the tools that have worked for you to date. As always, be careful to avoid over-estimates of profit potential due to use of in-sample results and selection bias.

When you have a little extra time, you might try constructing a portfolio of systems as an alternative to a portfolio of issues. The portfolio of systems is many individual systems, each of which trades one issue long/flat or short flat. Validate each individually. Every day, compute CAR25 for every system. Create a portfolio by trading the one, two, or several of the systems ranked highest that day.

Best regards,
Howard

 

[Willzy]

Hi Howard and RB,

Howard, as always thanks for the cautions and advice, they are most certainly taken under advisement!

RB, Having looked at your question regarding dynamic position sizing of the portfolio, there are a few ways you can go about this, each depend on your own personal choice.

If you are going to use the steps you outlined in your post (steps 1,2 and 3) to come up with a safeF value of say 0.73, then with that safeF value (0.73). I would run a backTest through periods of the poorest performance such as 2008 and 2011 to see what drawDown you get on both a closed trade by trade drawDown and on an equity drawDown. I suspect it will be more than you could tolerate! Always keep in mind that the worst drawDown of the system is yet to come!!!

As for point 4 in your post... By changing the number of positions that your system is taking you are in fact changing the system itself and I would caution against this having learn't the hard way $$$
(more positions is not necessary a good thing even though numTrades/Year will increase)

With mean reversions systems that trade a portfolio of issues you will find that most of the time during profitable runs, your system will only take only 3 or 4 positions at a time. However when the overall market takes a turn, serial correlation between stocks will kick in and you'll find your entire basket filled with sharp knives falling at tremendous velocity.

This is the reason that the monteCarlo process fails when more than 1 issue is being traded. As Howard stated above the data is not "independent" and identically distributed (iid).

You will find I'm sure that if you were to use a dynamic positions sizing technique such as the one you have mentioned above that it will tell you to increase your position size at the worst possible time.

Perhaps a solution is to find safeF during the worst drawDown periods in history such as 2008 or 2011 or even late 2015 and use that safeF figure as a ceiling on your dynamic position sizing process.

Personally I choose to use an anchored walkForward approach to the dynamic position sizing process. I include as many outOfSample trades as possible in the position sizing process. Over time safeF becomes quite stable and even stagnant but such an approach has saved me in the past and served me well to date.

Another approach which is possibly worth looking at is to examine the relationship between a (trade by trade) monteCarlo analysis against actual backTest history. With the bootstrap code I provided above you can run several backTests recording the MDD and finalEquity and compare those distributions with that provided by the monteCarlo analysis.

You will probably find that the more positions you allow to be open at any one time, the more useless the monteCarlo analysis becomes. With 10 positions open at a time I would use a reserveFactor of no less than 2. (ie if my risk statement was max(MDD95 @ safeF) = 20%) I would find safeF with MDD95 == 10%.

A final option is to use a more restrictive percentile of MDD ie (try the 99th percentile).

It is very important to note that these techniques are not using monteCarlo anaylsis in the way it is meant to be applied and as soon as you allow more than one issue to be traded at any one time then the original definition of MDD95 (being the 95th percentile of maximumDrawDown over the forcastHorizon) no longer holds true. That said I believe it is still better than eyeballing the equity curve of a single backTest.

My advice to you is to vary the number of positions you allow you system to trade attempting to find a sweet spot between the numTrades/Year and your exposure to serial correlation and its effects. For my systems I've found that to be between 3 and 7 trades at a time.

Just out of interest RB, how are you trading your system on the ASX, are you manually placing and managing the trades or are you using an automated process somehow?

Kind Regards Everyone
Matt

 

[rb250660]

Hi Matt and Howard. Thanks for the detailed responses.

As for point 4 in your post... By changing the number of positions that your system is taking you are in fact changing the system itself and I would caution against this having learn't the hard way $$$
(more positions is not necessary a good thing even though numTrades/Year will increase)

The signal model isn't changing but rather the position sizing model. What happens is I find that safe-f is 0.73 and fundamentally I allocate 13% to cash, 73% to trading. Then I divide that 73% into my standard 10 positions. So effectively I reduce position size after recent bad performance. This has worked quite well for me in the past.

I just checked my stats (back test, not real trades) and over the last 15 years I would have been fully invested 19 times. At these times the model was running red hot and making good profit.

However when the overall market takes a turn, serial correlation between stocks will kick in and you'll find your entire basket filled with sharp knives falling at tremendous velocity.

True but I filter for downside volatility which is quick to pick up times like these. However I do let the occasional falling knife through, you can't avoid that. Most of my bigger draw downs begin with a falling knife but I am quick to get out of them due to the high number of trades.

You will find I'm sure that if you were to use a dynamic positions sizing technique such as the one you have mentioned above that it will tell you to increase your position size at the worst possible time.

I'm not sure how you come to this conclusion. I position size based on recent past performance and if the market is turning south and I log a few bad trades, my position sizing algorithm is quick to react and reduce size. Conversely, if performance picks up it is quick to stack on size.

Personally I choose to use an anchored walkForward approach to the dynamic position sizing process. I include as many outOfSample trades as possible in the position sizing process. Over time safeF becomes quite stable and even stagnant but such an approach has saved me in the past and served me well to date.

I'd be very cautious about anchoring. 20 years down the track I wouldn't want old returns from way back influencing today's decisions. My opinion is that anchoring isn't really in the spirit of dynamic position sizing. As you say safeF becomes stable, is that what you really want? I would want something that is more responsive to recent performance, not increasing stability. Eventually you will converge to an almost static number, or a number that has an ever decreasing rate of change. This in my mind is not dynamic position sizing.

Personally I use the following:
- Portfolios of issues use a sliding window of X recent daily returns covering N trades.
- Single issue models (which I don't have but plan to implement soon) I will use a sliding window of N trade returns with weighted sampling (triangular distribution) to avoid the 'sudden' effects of trades dropping off.

 

[rb250660]

Just out of interest RB, how are you trading your system on the ASX, are you manually placing and managing the trades or are you using an automated process somehow?

My process is somewhat automated however it is fully supervised. Occasionally I need to step in and make a few adjustments.

 

[howardbandy]

In Post Number 18, two older than this one as I write, RB makes some very good points. The paragraph cautioning against anchoring in favor of a sliding window is particularly relevant.

Dynamic position sizing is a Bayesian approach -- learn from the data.

In a stationary environment, more data is better. If a credit organization is trying to learn how to distinguish whether a potential borrower is likely to repay, 1000 examples are better than 100 examples.

When the conditions change over time, weighting recent data more heavily than older data gives better indication of change. When trading a system, we want to know as quickly and accurately as possible that conditions have changed. For that purpose, the sliding window gives better results than the anchored window that uses all data. The length of the window and the weighting technique are hyperparameters to the model. They can be determined using crossvalidation.

Best, Howard