Trembling Hand
Can be found on the bid
- Joined
- 10 June 2007
- Posts
- 8,852
- Reactions
- 205
I would be genuinely interested in your ideas that you back tested?
Howard Bandy "Quantitative Trading Systems" & systems design Q&A
- Thread starter tech/a
- Start date
I would be genuinely interested in your ideas that you back tested?
- Joined
- 30 June 2007
- Posts
- 7,200
- Reactions
- 1,226
There's two ways around survivorship bias, and I think you know what they are, yes?
The systems on that site are more useful for getting into the swing of coding, then you start to do your own thing.
CanOz
Home runs feel good, but base hits pay bills!
- Joined
- 11 July 2006
- Posts
- 11,543
- Reactions
- 519
My data base includes de-listed stocks as well....even one that still has my money...
Wysiwyg
Everyone wants money
- Joined
- 8 August 2006
- Posts
- 8,428
- Reactions
- 284
Yes, test using delisted stocks also for that period or test over recent periods where most stocks are still in the present list.
Some other considerations are
1) stock volume actually available at entry/exit time
2) commission round trip
3) maximum consecutive losers
4) maximum system drawdown
- Joined
- 30 June 2007
- Posts
- 7,200
- Reactions
- 1,226
Using much shorter time frames will reduce the over-estimation of profits that a long-only backtest might provide. And yeh, using survivorship bias-free data in your testing.
Wysiwyg
Everyone wants money
- Joined
- 8 August 2006
- Posts
- 8,428
- Reactions
- 284
There are also stocks that are not delisted but are removed from an Index list for market capitalisation reasons or other.
CanOz
Home runs feel good, but base hits pay bills!
- Joined
- 11 July 2006
- Posts
- 11,543
- Reactions
- 519
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Greetings --
I have posted four hour-length videos from recent presentations on YouTube. I recommend watching them -- particularly the ones on Stationarity and Risk. The two on Development are also worth while. The first uses traditional trading system development platform, as in AmiBroker. The second uses machine learning.
Please note that backtesting is a necessary step in trading system development, but backtesting alone is not sufficient. The only way to obtain a reasonably unbiased estimate of future performance of a system is validation through applying the system to data that was not used in development. The gold standard of validation is walk forward testing, where the previously unseen data is the same data stream(s) as those used in development, but more recent in time. I cannot say this too strongly -- in-sample test results have no value is estimating future profitability. Out-of-sample validation is a requirement. If it is not done in the lab, it will be done by the market.
There are many sources of bias that will interfere with development. There are no favorable biases. If the results are biased, future performance will be worse than expected -- higher risk, lower profit. Here is a short list of some of the major ones:
..Survivorship / membership.
..Assuming backtesting is sufficient validation.
..Assuming independence when there is correlation.
..Metrics that include sequence dependent terms.
..Multiple uses of out-of-sample test data.
..Ignoring intra-trade drawdown.
..Using single metrics rather than distributions.
..Assuming stationarity.
..Future leaks.
Try very hard to avoid all these.
YouTube video links:
The Four Faces of Risk:
https://www.youtube.com/watch?v=Vw7mseQ_Tmc
The Importance of Being Stationary:
https://www.youtube.com/watch?v=iBhrZKErJ6A
Indicator Based Trading System Development:
https://www.youtube.com/watch?v=W-gSsqHORJE
Machine Learning Based Trading System Development:
https://www.youtube.com/watch?v=v729evhMpYk
A Google search of "Howard Bandy YouTube" will show a dozen or so additional videos. Some by me, but most recordings made by others of interviews of me and presentations by me. Among them are several from my presentations at the ATAA Conferences.
Best,
Howard
I have posted four hour-length videos from recent presentations on YouTube. I recommend watching them -- particularly the ones on Stationarity and Risk. The two on Development are also worth while. The first uses traditional trading system development platform, as in AmiBroker. The second uses machine learning.
Please note that backtesting is a necessary step in trading system development, but backtesting alone is not sufficient. The only way to obtain a reasonably unbiased estimate of future performance of a system is validation through applying the system to data that was not used in development. The gold standard of validation is walk forward testing, where the previously unseen data is the same data stream(s) as those used in development, but more recent in time. I cannot say this too strongly -- in-sample test results have no value is estimating future profitability. Out-of-sample validation is a requirement. If it is not done in the lab, it will be done by the market.
There are many sources of bias that will interfere with development. There are no favorable biases. If the results are biased, future performance will be worse than expected -- higher risk, lower profit. Here is a short list of some of the major ones:
..Survivorship / membership.
..Assuming backtesting is sufficient validation.
..Assuming independence when there is correlation.
..Metrics that include sequence dependent terms.
..Multiple uses of out-of-sample test data.
..Ignoring intra-trade drawdown.
..Using single metrics rather than distributions.
..Assuming stationarity.
..Future leaks.
Try very hard to avoid all these.
YouTube video links:
The Four Faces of Risk:
https://www.youtube.com/watch?v=Vw7mseQ_Tmc
The Importance of Being Stationary:
https://www.youtube.com/watch?v=iBhrZKErJ6A
Indicator Based Trading System Development:
https://www.youtube.com/watch?v=W-gSsqHORJE
Machine Learning Based Trading System Development:
https://www.youtube.com/watch?v=v729evhMpYk
A Google search of "Howard Bandy YouTube" will show a dozen or so additional videos. Some by me, but most recordings made by others of interviews of me and presentations by me. Among them are several from my presentations at the ATAA Conferences.
Best,
Howard
chops_a_must
Printing My Own Money
- Joined
- 1 November 2006
- Posts
- 4,636
- Reactions
- 3
Hi all.
Long time no see. I'm going back through and retesting some code and whatnot.
I seem to be having particular trouble with Howard Bandy's seasonality codes.
For instance, this code:
// SeasonalityDayOfMonth_Testing.afl
//
// Test for seasonality based on the day of the month.
// This version does the day-by-day testing.
//
SetTradeDelays(0,0,0,0);
BuyPrice = Close;
SellPrice = Close;
DayToBuy = Optimize("DTB",1,1,31,1);
Buy = DayToBuy == Day();
Sell = BarsSince(Buy) >=1;
//Figure 11.1 Seasonality Day of Month - Testing
only generates a buy on the 1st day of the month. I can alter it, so that as a parameter, it can do each single day individually, but not all at once.
I'm sure there is a setting I have bumped that is preventing this, but I can't work it out.
Cheers,
Chops.
Long time no see. I'm going back through and retesting some code and whatnot.
I seem to be having particular trouble with Howard Bandy's seasonality codes.
For instance, this code:
// SeasonalityDayOfMonth_Testing.afl
//
// Test for seasonality based on the day of the month.
// This version does the day-by-day testing.
//
SetTradeDelays(0,0,0,0);
BuyPrice = Close;
SellPrice = Close;
DayToBuy = Optimize("DTB",1,1,31,1);
Buy = DayToBuy == Day();
Sell = BarsSince(Buy) >=1;
//Figure 11.1 Seasonality Day of Month - Testing
only generates a buy on the 1st day of the month. I can alter it, so that as a parameter, it can do each single day individually, but not all at once.
I'm sure there is a setting I have bumped that is preventing this, but I can't work it out.
Cheers,
Chops.
chops_a_must
Printing My Own Money
- Joined
- 1 November 2006
- Posts
- 4,636
- Reactions
- 3
CanOz
Home runs feel good, but base hits pay bills!
- Joined
- 11 July 2006
- Posts
- 11,543
- Reactions
- 519
Wysiwyg
Everyone wants money
- Joined
- 8 August 2006
- Posts
- 8,428
- Reactions
- 284
I remember you. Happy to see ye posting again. The 2006 join ups.
chops_a_must
Printing My Own Money
- Joined
- 1 November 2006
- Posts
- 4,636
- Reactions
- 3
Chops!!!
How are ya man!
I'm good. After a few years of moving around and stuff, I've actually been able to settle and hopefully get back into some active trading.
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Greetings --
Use caution with seasonality-based systems. It is very easy to overfit the model in-sample, then find poor performance out-of-sample.
For US stocks and equity ETFs, there have been roughly two cycles per month. Each with a period of about 3 to 5 days of price rise and 3 to 5 days of price drop. One revolves around month-end, the other around third-Friday options expiration. Both have been "discovered" and both are losing their edge.
The seasonality based on October being the poorest performing month was badly wrong this year.
IMO, essentially all of the rise in the stock, bond, and real estate markets has been due to the US government manipulating interest rates and forcing people who want income from their funds to take long positions in those asset classes. As either the government or the financial world lifts US interest rates, I expect all those markets to return to their 2008-2009 lows. If it happens, this effect will swamp seasonality, including the traditional "January" effect.
Best,
Howard
Use caution with seasonality-based systems. It is very easy to overfit the model in-sample, then find poor performance out-of-sample.
For US stocks and equity ETFs, there have been roughly two cycles per month. Each with a period of about 3 to 5 days of price rise and 3 to 5 days of price drop. One revolves around month-end, the other around third-Friday options expiration. Both have been "discovered" and both are losing their edge.
The seasonality based on October being the poorest performing month was badly wrong this year.
IMO, essentially all of the rise in the stock, bond, and real estate markets has been due to the US government manipulating interest rates and forcing people who want income from their funds to take long positions in those asset classes. As either the government or the financial world lifts US interest rates, I expect all those markets to return to their 2008-2009 lows. If it happens, this effect will swamp seasonality, including the traditional "January" effect.
Best,
Howard
chops_a_must
Printing My Own Money
- Joined
- 1 November 2006
- Posts
- 4,636
- Reactions
- 3
Thanks Howard.
That's why I'm retesting these ideas.
I've never used the seasonality findings apart from on the ASX. Where the seasonality is a bit different to the book. I suspect this is because of superannuation flows and influence on our market.
I'm sure blogs such as overnight edges and the like, which many years ago were selling daily seasonality odds for positions, was probably the beginning of the end for a lot of seasonality systems.
Cheers,
Chops.
That's why I'm retesting these ideas.
I've never used the seasonality findings apart from on the ASX. Where the seasonality is a bit different to the book. I suspect this is because of superannuation flows and influence on our market.
I'm sure blogs such as overnight edges and the like, which many years ago were selling daily seasonality odds for positions, was probably the beginning of the end for a lot of seasonality systems.
Cheers,
Chops.
- Joined
- 19 November 2015
- Posts
- 6
- Reactions
- 0
Howard,
I have a question that pertains not to Quantitative Trading Systems but rather your latest book Quantitative Technical Analysis. You use each day's mark-to-market equity as a closed trade, and then go on to sample from these "closed trades" as part of your monte carlo analysis. Is it possible to do so without eliminating the effect of serial correlation that comes from being in a position? Or sampling two consecutive points that occurred in different market regimes? Surely there are entry opportunities where multiple profitable days follow? Otherwise the concept makes a lot of sense from the standpoint of trade management.
Best,
Brian
I have a question that pertains not to Quantitative Trading Systems but rather your latest book Quantitative Technical Analysis. You use each day's mark-to-market equity as a closed trade, and then go on to sample from these "closed trades" as part of your monte carlo analysis. Is it possible to do so without eliminating the effect of serial correlation that comes from being in a position? Or sampling two consecutive points that occurred in different market regimes? Surely there are entry opportunities where multiple profitable days follow? Otherwise the concept makes a lot of sense from the standpoint of trade management.
Best,
Brian
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Hi 11bblandin --
Marking to market daily gives an opportunity to both measure and manage daily. My view of trading systems is that the trader must have a period of time during which he or she is willing to wait. The assumption I make in the book and in my presentation is that period is a single day. That is, I depend that my model will cause an exit whenever conditions are such that I would have entered an order to subjectively exit the trade. If there is a sequence of days where the same position (say, long) is held throughout, there is no need to charge commission or slippage for the continuation days, and the result of a multi-day marked-to-market-daily trade where there is a state reported every day is identical with those of a multi-day trade delimited by impulse buy and sell signals.
If there is in fact serial correlation for the days within a trade, you should treat the entire multiday trade as a single data point. Test this as part of the development. The sequence of intra-trade percentage gain per day may not be serially correlated. If there is no serial correlation, you can treat each day as an independent data point.
Analysis of multiple trades, as in a portfolio, is complicated when there are days with multiple holdings. If the issues being traded are equities, there is probably strong correlation between the gains for two or more issues. If so, they should be treated as a single "daily result" data point. Think ahead to the Monte Carlo simulation that will be run. The trades drawn in the MC process are assumed to be independent. If they are not independent, the results will be biased toward over-optimism.
I hope I am answering the question you were asking,
Best,
Howard
Marking to market daily gives an opportunity to both measure and manage daily. My view of trading systems is that the trader must have a period of time during which he or she is willing to wait. The assumption I make in the book and in my presentation is that period is a single day. That is, I depend that my model will cause an exit whenever conditions are such that I would have entered an order to subjectively exit the trade. If there is a sequence of days where the same position (say, long) is held throughout, there is no need to charge commission or slippage for the continuation days, and the result of a multi-day marked-to-market-daily trade where there is a state reported every day is identical with those of a multi-day trade delimited by impulse buy and sell signals.
If there is in fact serial correlation for the days within a trade, you should treat the entire multiday trade as a single data point. Test this as part of the development. The sequence of intra-trade percentage gain per day may not be serially correlated. If there is no serial correlation, you can treat each day as an independent data point.
Analysis of multiple trades, as in a portfolio, is complicated when there are days with multiple holdings. If the issues being traded are equities, there is probably strong correlation between the gains for two or more issues. If so, they should be treated as a single "daily result" data point. Think ahead to the Monte Carlo simulation that will be run. The trades drawn in the MC process are assumed to be independent. If they are not independent, the results will be biased toward over-optimism.
I hope I am answering the question you were asking,
Best,
Howard
- Joined
- 19 November 2015
- Posts
- 6
- Reactions
- 0
I am focused mainly on systems that trade a single issue long/flat and agree with the concept of assessing each day whether to be long or be flat (whether that means entering, exiting, or continuing to be long/flat). However, when I move on to the next step of determining risk through MC simulation, the sequences of trades that generate the largest drawdowns may likely be the result of several consecutive trades (days) when large losses hit. A system that incorporates a trend filter or similar mechanism to stay out of the market when things start to get turbulent would (hopefully) generate a signal to be flat.
As an example, a system may encounter several very negative days while long over the course of its test period, which drops the price below a threshold (say, a 200-day moving average) and the system dictates to be flat until more favorable conditions return. A MC simulation may string these big losers together, creating a large drawdown and indicating a high level of risk. I understand the concept of tail risk and why it's important in development, but I also try to design systems where I wouldn't continue to be long day after day as the market plummets. Hopefully I'm being clear.. I can't think of a better way, just curious if you've run into these issues in your testing and how you dealt with them
As an example, a system may encounter several very negative days while long over the course of its test period, which drops the price below a threshold (say, a 200-day moving average) and the system dictates to be flat until more favorable conditions return. A MC simulation may string these big losers together, creating a large drawdown and indicating a high level of risk. I understand the concept of tail risk and why it's important in development, but I also try to design systems where I wouldn't continue to be long day after day as the market plummets. Hopefully I'm being clear.. I can't think of a better way, just curious if you've run into these issues in your testing and how you dealt with them
- Joined
- 13 June 2007
- Posts
- 838
- Reactions
- 136
Greetings --
A filter such as taking long positions only when the price is above the 200 day moving average is popular in the literature. The price comparison that rule creates is between the average of the 200 days -- the midpoint of which is 100 days ago -- and the current price. Whether that rule helps or hurts depends on how persistent the price trends of that issue are and what the entry and exit rules are. Perform your own tests before accepting it as good practice.
My recommendation of marking to market every day goes along with managing the trade every day. In order to becoming stuck in a bad trade, the rules must include an exit to get out of it. Some rules that can work in some cases include: maximum holding periods, profit targets, maximum loss. Test to determine which, if any, help.
Note that maximum loss exits are usually entered as "stop" orders. There is a change coming to rules regarding which types of orders are permitted for US markets. It is likely that stop orders will be restricted, and maybe prohibited. Good-until-canceled stop orders are almost certain to be prohibited.
Trading systems that trade infrequently, hold a long period of time, and hope to catch big price moves are most susceptible to getting caught in a large losing position. If your personality permits it, consider using an objective function that rewards systems that: trade frequently, trade accurately, hold a short period of time. Systems with these characteristics often have the best performance (for long/flat positions) when the price is below its long term moving average. In those cases, setting a long term moving average filter removes a high proportion of the profitable trades.
You are correct that a Monte Carlo analysis can choose trade sequences where losing trades follow each other closely and create large drawdowns. The purpose of the Monte Carlo analysis is precisely that -- to estimate the risk if / when that happens in real trading. Do not ignore the tail risk.
Best,
Howard
A filter such as taking long positions only when the price is above the 200 day moving average is popular in the literature. The price comparison that rule creates is between the average of the 200 days -- the midpoint of which is 100 days ago -- and the current price. Whether that rule helps or hurts depends on how persistent the price trends of that issue are and what the entry and exit rules are. Perform your own tests before accepting it as good practice.
My recommendation of marking to market every day goes along with managing the trade every day. In order to becoming stuck in a bad trade, the rules must include an exit to get out of it. Some rules that can work in some cases include: maximum holding periods, profit targets, maximum loss. Test to determine which, if any, help.
Note that maximum loss exits are usually entered as "stop" orders. There is a change coming to rules regarding which types of orders are permitted for US markets. It is likely that stop orders will be restricted, and maybe prohibited. Good-until-canceled stop orders are almost certain to be prohibited.
Trading systems that trade infrequently, hold a long period of time, and hope to catch big price moves are most susceptible to getting caught in a large losing position. If your personality permits it, consider using an objective function that rewards systems that: trade frequently, trade accurately, hold a short period of time. Systems with these characteristics often have the best performance (for long/flat positions) when the price is below its long term moving average. In those cases, setting a long term moving average filter removes a high proportion of the profitable trades.
You are correct that a Monte Carlo analysis can choose trade sequences where losing trades follow each other closely and create large drawdowns. The purpose of the Monte Carlo analysis is precisely that -- to estimate the risk if / when that happens in real trading. Do not ignore the tail risk.
Best,
Howard
Similar threads
