Survivorship bias on the FTSE100 (LSE)

[Chorlton]

Hello All,

Has anyone ever carried out any backtesting on the LSE or more specifically the FTSE100?

If so, how did you deal with the problem of survivorship bias, especially if backtesting over many years of data ??

The reason I ask is that I want to backtest some of my strategies using Metastock & Tradesim but I am not sure how I should go about doing this effectively?


As always, any help would be much appreciated,

Thanks in advance,

Chorlton

 

[howardbandy]

Hi Chorlton --

In order to test the survivorship bias from, say, 1995 to now, you will need a list of those stocks that were members of the index in 1995 and price history from 1995 until they disappeared or until now.

Survivorship bias is a significant problem, but testing for it can be difficult and expensive. High quality data for stocks that are no longer traded is expensive.

You are probably already aware, but there are some other issues to consider.

If you use any fundamental data, such as earnings, revisions can be a significant issue. We trade based on the most recent release of data, but when a series is revised, and the data supplier replaces the original data with the revised data, we usually have no way of knowing what the original data was or when the revised data appeared.

Stock splits can also distort the data. Say a stock is now $64, and that has split 2 for 1 three times since 2005. If it actually traded at $32 in 2004, the data series will show it at $4 to reflect the splits. $4 stocks act differently than $32 stocks, so using price levels when backtesting can be misleading.

When companies reorganize, particularly when they merge or spin off subsidiaries, the data streams (both fundamental and price) are seldom adjusted, and the industry category may change. For example, assume an automobile manufacturer that gets a large portion of its profit from its financing division sells off that division. The data stream for that ticker symbol represents what is essentially a finance company before the sale, and a manufacturing company after.

Thanks,
Howard

 

[Chorlton]

Hi Chorlton --

In order to test the survivorship bias from, say, 1995 to now, you will need a list of those stocks that were members of the index in 1995 and price history from 1995 until they disappeared or until now.

Survivorship bias is a significant problem, but testing for it can be difficult and expensive. High quality data for stocks that are no longer traded is expensive.

You are probably already aware, but there are some other issues to consider.

If you use any fundamental data, such as earnings, revisions can be a significant issue. We trade based on the most recent release of data, but when a series is revised, and the data supplier replaces the original data with the revised data, we usually have no way of knowing what the original data was or when the revised data appeared.

Stock splits can also distort the data. Say a stock is now $64, and that has split 2 for 1 three times since 2005. If it actually traded at $32 in 2004, the data series will show it at $4 to reflect the splits. $4 stocks act differently than $32 stocks, so using price levels when backtesting can be misleading.

When companies reorganize, particularly when they merge or spin off subsidiaries, the data streams (both fundamental and price) are seldom adjusted, and the industry category may change. For example, assume an automobile manufacturer that gets a large portion of its profit from its financing division sells off that division. The data stream for that ticker symbol represents what is essentially a finance company before the sale, and a manufacturing company after.

Thanks,
Howard

Hello Howard,

Thanks for taking the time to reply in such detail.

As I am new to system testing, is it possible for you to elaborate on your first paragraph?

"In order to test the survivorship bias from, say, 1995 to now, you will need a list of those stocks that were members of the index in 1995 and price history from 1995 until they disappeared or until now."

Once I am able to find out what companies made up the index in say 1995, and then which ones are still trading today, what would be my next steps?

For example, for those stocks which have been delisted from the index since 1995, should I ignore them when carrying out my backtesting?

For the record, I am not concerned about the fundamentals of the stocks so issues associated with earnings shouldn't hopefully affect me.

Many Thanks in advance,

A confused Chorlton

 

[howardbandy]

Hi Chorlton --

You will need two things:

1. An accurate list of those stocks that were members of the group you wish to test. If you want to test the stocks in the FTSE (assuming you mean they were component members of the FTSE 100 index), intending to trade the stocks in the FTSE, then you need a list of the stocks that were members of the FTSE index at the start of the period you want to use.

Although it sounds like it would be simple to get that information, it is not. There was a similar discussion on one of the AmiBroker forums regarding membership in the NASDAQ 100, S&P 100, and S&P 500. It took quite a lot of research to find those lists.

2. Accurate historical data for all the tickers in the list of stocks you want to test.

Obviously, when a stock stops trading, the end-of-day data for that stock stops, and the data vendor has nothing to distribute. If you have been maintaining your own database, you will have that data. If your database vendor maintains your database for you, they will probably drop those tickers immediately.

So -- you need a database that you maintain yourself, and a source of end-of-day quotes for the tickers and time periods you want.

If you are using AmiBroker, setting up and maintaining that database is easy. For other programs, other forum members will have better up-to-date information than I have.

For the data, try CSI at:
http://www.csidata.com/

or Norgate at:
http://www.premiumdata.net/

If other forum members know of other vendors who can supply historical data for tickers that are no longer trading, please let us know.

Thanks,
Howard

 

[Chorlton]

Obviously, when a stock stops trading, the end-of-day data for that stock stops, and the data vendor has nothing to distribute. If you have been maintaining your own database, you will have that data. If your database vendor maintains your database for you, they will probably drop those tickers immediately.

Hi Howard,

Many Thanks for your help on this subject....

I am beginning to understand why there is so much discussion on this subject !!! Its an topic which I didn't really appreciate until recently....

With reference to your above comment, if I was carrying out backtesting and a particular stock stopped trading then am I right in assuming that the trade would simply remain shown as "open", as there would be no further data and therefore no way for the exit to be triggered (unless a time-stop was being utilised)?

 

[howardbandy]

Hi Chorlton --

It would be up to your trading system and might depend on your data vendor.

If you are maintaining your own database, when a stock stops trading all together, that is indistinguishable from a stock that has no trades for one or more days.

If the data vendor is maintaining the database, when a stock stops trading all together that ticker disappears.

What happens to your systems now when a stock has no quote for one or more days? Probably nothing. Most trading systems look for data bar-by-bar and at the data for each ticker individually. A day without data could be a day with no trades for that stock or a day with no trading for any stock -- a weekend or a holiday or a really long holiday. A day with no data is a day that cannot generate a signal.

Will a holding period exit get you out? Probably not. Say you have a system that holds exactly one day. It buys on Tuesday and exits on Wednesday. If it buys on Friday, does it exit on Saturday? No. What would the exit price be?

Thanks,
Howard

 

[nizar]

Hi Howard,

Many Thanks for your help on this subject....

I am beginning to understand why there is so much discussion on this subject !!! Its an topic which I didn't really appreciate until recently....

With reference to your above comment, if I was carrying out backtesting and a particular stock stopped trading then am I right in assuming that the trade would simply remain shown as "open", as there would be no further data and therefore no way for the exit to be triggered (unless a time-stop was being utilised)?

Hi Chorlton,

I was facing these very same issues only a few days ago when I found out that 3 of my companies were currently under takeover.

So what does TradeSim do once a stock is delisted and stops trading for any reason (stock code/company name change, bankcrupcy, merger, cash takeover, script takeover), it exits the stock on the last bar it traded and marks it in the trade database as an OPEN TRADE.

I checked my trade database for my main trading period, and 58 out of 1234 trades were like this, or just under 5%. So not a major concern.

Still, when this happens obviously you have to keep track of the company more closely, and use your own judgement. Where discretion can be applied in the future, I will try and avoid stocks which are already under takeover.

Nizar.

 

[tech/a]

So not a major concern.

Disagree.
If tradesim has the stock at the current level and still open and its been delisted,then the total open position value in that entity is ZERO.
If its undergone a split then you could have one position open at the old ticker value and trade any number of positions in the new ticker symbol.Same for merger,take over or name change.
Your 5% could have a far greater impact.

The only way I know around it and one Ive adopted (Howard may comment)
is using a smaller timeframe infact I use only 8 yrs.
Longer term timeframe systems will be affected by changes in the dynamics of a market to a greater degree than those trading shorter timeframes.

The reason that the system works may deminish or disappear if those dynamics change to a greater degree overtime relative to the conditions tested. howards optimisation of parameters could send an early warning signal if you start to see a huge difference in performance from closely corresponding parameter values. Its a case of monitoring the system on a regular basis to be sure its still performing with an edge.

Not perfect I know,but has worked well so far.

 

[nizar]

Yeh well tech/a, why don't you see how many % of your trades were open trades in comparison?

I don't think it would be much less than 5%. In fact 5% is probably roughly the market average.

BUT if you do test for this then make sure you use SetTriggerAtDate as part of your exit so actual open trades get closed out at the end and don't get included up in the tally.

Is there any way around this?
Because I've asked a few gun traders and they told me not so.

Why should I be concerned about something where there's no way around it.
I was told that company takeovers/mergers etc, cannot be accurately backtested and that you have to apply your own discretion here.

Is there any other way to do this?

If so -- Im very interested.

(For this system Im now trading the testing was done from 01-01-1998 until 31-12-2004 (6 years). And i left 3.5 years as an out of sample data. Naturally the returns were lower in the out of sample compared to the data from which it was optimised and initially tested on. Full performance stats are available on my blog if you are interested.)

Thanks.

Nizar.

 

[nizar]

Disagree.
If tradesim has the stock at the current level and still open and its been delisted,then the total open position value in that entity is ZERO.

Yeh but see ALL of the time, when the stock is delisted, you will be offered stock or cash pretty much equivalent to the last price it traded at anyway. It's against the law to take your shares and leave you with nothing.

So I guess if you want to trade exactly how tradesim does THEN at this time you could just take the money (or cash in the script) and run.

 

[tech/a]

De listed = Goes broke.

Hell I wish it were illegal to leave me with zilch.
You can lose 100% of your capital on a delisted stock.
Including any open equity you have in it.

 

[nizar]

De listed = Goes broke.

Hell I wish it were illegal to leave me with zilch.
You can lose 100% of your capital on a delisted stock.
Including any open equity you have in it.

Delisted means that the stock is no longer listed.
Due to change in company activities, or change in stock code, or merger, or a takeover, or bankcrupcy.

If it went broke that is one of the reasons a stock can be delisted. In this case usually the sp plummets first and the trailing stop will save you.

For example. SSX was delisted in August. But it got bought by OneSteel. So if you held SSX shares you would have got OneSteel stock.

For backtesting purposes as far as TradeSim is concerned, delisted means that the stock is no longer quoted and no longer traded. Doesnt matter the reason. Another example, UTB was delisted i think last year (one of your champions ), but if you help UTB on the last day it traded you wouldve got cash or tattersalls stock or something, probably equivalent to what UTB last traded at anyway.

So when a stock is delisted and an EXIT is not triggered, Tradesim will exit on the last day the stock traded and mark it as an open trade in the trade database.

 

[tech/a]

Whatever

 

[howardbandy]

If it went broke that is one of the reasons a stock can be delisted. In this case usually the sp plummets first and the trailing stop will save you.

Hi Nizar --

My preference, in order, for exits from trades is:
1. A signal.
2. A profit target.
3. A holding time.
4. A trailing stop.
5. A maximum loss stop.

For some systems, 2 and 3 are exchanged.

The best trailing stops, in my opinion, are A. the parabolic, and B. the chandelier.

No stop guarantees an exit at any price. If the stock price is falling rapidly, from say $20 to $0.01 as the really bad news hits the news wires, my stop set at $18 (regardless of whether it is a trailing stop or a maximum loss stop) may be executed at any price between $18 and $0.01 -- or it may not be executed at all if there is no one willing to buy my shares at any price.

My advice in one line: Stops hurt systems.

I recommend designing exits into the system rather than relying on stops.

Thanks for listening,
Howard

 

[nizar]

Hi Nizar --

My preference, in order, for exits from trades is:
1. A signal.
2. A profit target.
3. A holding time.
4. A trailing stop.
5. A maximum loss stop.

For some systems, 2 and 3 are exchanged.

The best trailing stops, in my opinion, are A. the parabolic, and B. the chandelier.

No stop guarantees an exit at any price. If the stock price is falling rapidly, from say $20 to $0.01 as the really bad news hits the news wires, my stop set at $18 (regardless of whether it is a trailing stop or a maximum loss stop) may be executed at any price between $18 and $0.01 -- or it may not be executed at all if there is no one willing to buy my shares at any price.

My advice in one line: Stops hurt systems.

I recommend designing exits into the system rather than relying on stops.

Thanks for listening,
Howard

Hi Howard,

As usual thanks for your thoughts.
I have read your post on exits previously on this board and Iv also read yout book which mentions this.

The system that Iv just began trading uses only a chandelier trailing exit. I use an initial stop which is equal to the trailing stop for position sizing purposes.

No alternative initial stop which I have applied have improved the objective function.

Nice example by the way, but myself, like you, am not concerned with what happens in any one trade.

Nizar.

 

[Chorlton]

Hello Guys,

Thanks for everyone who has contributed...

----

Howard,

"Will a holding period exit get you out? Probably not. Say you have a system that holds exactly one day. It buys on Tuesday and exits on Wednesday. If it buys on Friday, does it exit on Saturday? No. What would the exit price be?"

Based on your previous comments, its clear now the importance of maintaining one's own database.

However, one point that still confuses me is what should I do with those stocks which have been delisted over the backtest period? Surely, as long at the software which I am using (Metastock & Tradesim) can efficiently deal with such stocks in an appropriate manner, then surely I should still include them in my backtesting. Afterall, there is no way to know in the future which new stocks may become delisted especially those which have previously signaled a buy signal.

Apologies, if I'm missing something very fundamental or obvious here...


------

Nizar / Tech/A,

It seems there is some confusion as to what the term "delisted" actually refers to.

If its just a case that it is no longer listed due to a change within the company which is what Nizar refers to, then am I still right in assuming that Tradesim will still "exit the stock on the last bar it traded and mark it in the trade database as an OPEN TRADE" ?

---------

Tech/A,

"If tradesim has the stock at the current level and still open and its been delisted,then the total open position value in that entity is ZERO.
If its undergone a split then you could have one position open at the old ticker value and trade any number of positions in the new ticker symbol.Same for merger,take over or name change.
Your 5% could have a far greater impact.

The only way I know around it and one Ive adopted (Howard may comment)
is using a smaller timeframe infact I use only 8 yrs.
Longer term timeframe systems will be affected by changes in the dynamics of a market to a greater degree than those trading shorter timeframes."


Can I ask how a shorter timeframe would help? Surely, a stock can become delisted at any time so backtesting on a smaller time period has no added benefit? Sorry, just confused.....


Reading through the various posts, it seems clear that this area of trading is the most difficult to effectively manage & control..... or maybe its just my lack of understanding!!!

 

[Chorlton]

I recommend designing exits into the system

Hi Howard,

Surely, this would depend on the type of system one is developing / trading? In the case of a Trend-Following System, how can one effectively choose an exit in advance without sacrificing possible further profits, as isn't it normally a small minority of trades which produce the majority of the profit?

If this is the case, by coding a pre-defined exit, isn't one seriously impacting on its overall performance?

Regards,

Chorlton

 

[tech/a]

Can I ask how a shorter timeframe would help?

I have 5 yrs of all data including stocks delisted,I just havent deleted them.

Even if I didnt going back 8 yrs rather than 20 will give me less.For example the constituents in the ASX 500 are not even remotely common to the constituents of 5 yrs ago and probably only a couple remain after 20 yrs.
Survivorship can also mean a shifting in an index or universe of stock being traded.

Finally dont care how a stock is delisted if Tradesim treats it as an open position and there is open equity until the test is completed and that open equity is then added to closed equity---then the results can be alarmingly skewed toward profit that wouldnt/shouldnt be there.
If it sells at the last traded price also a problem obviously if the stock goes bankrupt.
You could be trading a system which isnt profitable.

On exits.(Chorlton)
What your missing is that you'll never find the exits of exits or the entry of entries.
What you can find is a system that suits your objectives,one of which will be an acceptable return on investment.
So its the system your after not the singular component.
What Im sure Howard means is that stops actually damage systems (IE impact on common objectives).

Takes a bit to get your head around,its the whole not the component/s.

 

[weird]

Reuters DataLink is another one to look at, which is quite inexpensive, for overseas data.

If you interested in setting it up, this post may be helpful,

http://www.thechartist.com.au/forum/ubbthreads.php?ubb=showflat&Number=43447&page=0&fpart=2

I normally trade from constituent lists, which has a survivorship bias, considering that a stock now in the AllOrds or above may not have been in this list 5 years ago from the commencement of backtesting.

I personally like using a simple price filter, that is above 0.4 cents or whatever as opposed to a liquidity filter, to weed out some of the survivorship bias (perhaps because I also use Tradesim to help here). I find I can more effectively use such a filter in Tradesim, which allows me to filter out trades I would not be willing to take due to the position size vs the actual liquidity.

However this said, I had a few examples this week, where I noticed, that the actual trades put through, with the accompanying position sizes (larges gaps in price ranges) had me thinking.

Apart from using Tradesim, I find exporting the trade database to excel, and looking at trade average performance useful. If trading a sufficient size portfolio, even using trendfollowing techniques, the average trade performance when totaled should have sufficient edge without relying on one or two outliners.

 

[weird]

whoops, bit distracted with other things around the computer, I meant $ 0.4 , anyhow most the above looks intended or correct as the post ... sorry abit busy with other things atm.