Dump it Here

[Richard Dale]

My advice is to thoroughly understand what is omitted from the entry-level data packages

Indeed - we offer multiple levels of packages. See below - I think it's pretty clear-cut from our descriptions but would certainly ask for feedback if this seems to be inaccurate/insufficient for systematic backtesting.

This is our current set of subscription levels (as at Sep 2023) for AU Stocks (and there are additional information web pages available too):

 

[peter2]

@Skate and @Richard Dale Thank-you both for taking the time to emphasise the insidious nature of survivorship bias when back-testing.

Looking through the last few pages, if I was a beginner looking to research a trading system for myself I'd be totally confused and probably off-put by the potential problems with back-testing. I wouldn't know what data I'd need to buy to suit my needs. Most beginners, after buying some software, don't have the capital to pay for the highest quality data. Beginners don't really know what they need when they start out.

I fail to understand the need to use index constituents when back-testing. For me, if the security has enough liquidity then it's OK to be considered for trading signals. Would I avoid survivorship bias doing this?

--------

@Skate I wanted to ask about the buy triggers in this system. Are the top ten trading candidates selected because the buy signal triggered last week (bar) or do some get included because the buy signal was triggered weeks ago and the trade still open? I ask because imho some of the buy signals indicated this week may have triggered earlier but didn't due to the market filter being off. They may qualify now due to the market filter allowing buying, but the initial buy signal triggered weeks ago. My concern is buying late (after the initial buy trigger).

(I hope I've made this last query clear?)

Edit: Haven't read Richard's last post.

 

[Skate]

Indeed - we offer multiple levels of packages. See below - I think it's pretty clear-cut from our descriptions but would certainly ask for feedback if this seems to be inaccurate/insufficient for systematic backtesting.

This is our current level (as at Sep 2023) for AU Stocks (and there are additional information web pages available too):

View attachment 161940

@Richard Dale, I understand the intention behind offering multiple data package options is to provide flexibility for users with diverse needs and budgets. However, as someone new to algorithmic trading, I found it difficult to discern the key differences between entry-level and more advanced packages from the descriptions alone.

When I first started exploring backtesting, I simply did not have enough knowledge to comprehend the significance of aspects like historical constituent data or certain attribute completeness. The terminology itself was overwhelming, let alone trying to determine the impact on my analysis. I made incorrect assumptions about the capabilities of the basic package, which led me down the wrong path early on.

That experience is why I want to provide more context to help others avoid potential pitfalls in selecting the right data package. My advice comes from a place of wishing someone had broken it down for me when I was new to this world. I aim to explain the limitations in plain language, not as a warning but rather as an insight I learned first-hand.

For a beginner, concrete examples are invaluable in illustrating the differences between packages. I could have avoided missteps if I understood from the start how missing attributes would impede certain types of backtesting. While too much technical detail too soon can also overwhelm new users, I believe there is a way to strike a balance and deliver the right information at the right time to guide their learning.

My suggestion is to develop robust educational resources that match the user's current level of understanding. This allows users to build knowledge at their own pace while giving them a roadmap. By providing enough detail to make an informed decision, yet avoiding information overload, we can ensure users select the offering that best fits their needs as they advance in their journey.

Skate.

 

[Skate]

@Skate and @Richard Dale Thank-you both for taking the time to emphasise the insidious nature of survivorship bias when back-testing.
Looking through the last few pages, if I was a beginner looking to research a trading system for myself I'd be totally confused and probably off-put by the potential problems with back-testing. I wouldn't know what data I'd need to buy to suit my needs. Most beginners, after buying some software, don't have the capital to pay for the highest quality data. Beginners don't really know what they need when they start out.
I fail to understand the need to use index constituents when back-testing. For me, if the security has enough liquidity then it's OK to be considered for trading signals. Would I avoid survivorship bias doing this?

@peter2, I trust @Richard Dale will read your response. Peter summed it up well - newcomers simply do not have the knowledge foundation yet to grasp all the intricacies you described. We first need to comprehend the basic principles before diving into more advanced implementation details.

For instance, I did not inherently understand why backtesting only liquid stocks could still result in survivorship bias. The theory behind needing historical index constituents was lost on me early on. Frankly, most of the terminology sounded like a foreign language.

I don't aim to dismiss your advice - I understand you are trying to impart best practices. But without concrete examples illustrating how inadequate data can ruin a backtest, I missed the bigger picture. I heard the "what" but not the "why".

Perhaps an analogy could help convey the risks more relatable. Coming from a place of limited capital myself, beginners need help prioritising spending on quality data. We learn by doing, so demonstrating potential pitfalls through sample backtests could enable us to learn the lesson first-hand.

I hope this provides some perspective on why your technical details, though well-intentioned, missed the mark for me when I was a beginner. I appreciate you taking the time to respond to my thread and concerns as the information is very educational.

Skate.

 

[Richard Dale]

Looking through the last few pages, if I was a beginner looking to research a trading system for myself I'd be totally confused and probably off-put by the potential problems with back-testing. I wouldn't know what data I'd need to buy to suit my needs. Most beginners, after buying some software, don't have the capital to pay for the highest quality data. Beginners don't really know what they need when they start out.

I fail to understand the need to use index constituents when back-testing. For me, if the security has enough liquidity then it's OK to be considered for trading signals. Would I avoid survivorship bias doing this?

I agree on this: Most beginners in the systematic trading space have no idea how much of an impact that survivorship bias or pre-inclusion bias has on the validity of backtesting.

On your second point: If you only have a liquidity requirement in your system, then index constituents are irrelevant. However, you also need to include stocks that have been delisted (for whatever reason). An example of a stock I actively traded on the long side in the DotCom times (liqudity was fine - this was doing 1+M volume a day)

 

[Skate]

@Skate I wanted to ask about the buy triggers in this system. Are the top ten trading candidates selected because the buy signal triggered last week (bar) or do some get included because the buy signal was triggered weeks ago and the trade still open? I ask because imho some of the buy signals indicated this week may have triggered earlier but didn't due to the market filter being off. They may qualify now due to the market filter allowing buying, but the initial buy signal triggered weeks ago. My concern is buying late (after the initial buy trigger).

Great question about the timing of buy signals and how the top ten candidates are selected each week.

Let me explain how the system handles ongoing buy signals
The initial buy signals are triggered when the breakout occurs, based on the weekly price action. However, those signals remain active even if the market filter restricts buying at that moment. So a stock that broke out 3 weeks ago but did not qualify for purchase then due to the "Percentage Up" filter would still retain that active buy signal. It does not have to break out again to be considered.

Each week when selecting the top ten candidates, the system looks at all active buy signals, not just the new ones triggered that week. So some candidates on the list may have an older active buy signal from previous weeks that is just now able to be acted upon.

Skate.

 

[peter2]

@Skate - Thank-you for clearing that up as I do worry that some of your buys are "late".

Have you researched the difference? Back-test with only new buy signals vs new and open signals.

 

[Skate]

I agree on this: Most beginners in the systematic trading space have no idea how much of an impact that survivorship bias or pre-inclusion bias has on the validity of backtesting.

@Richard Dale, after reviewing my recent research and strategy development, I realise there were some limitations that may have compromised the validity of the signals I had hoped to promote. Specifically, the lower Silver subscription level of the Norgate data I was using imposed some restrictions on the backtesting and analysis I could perform.

While my intention was to share signals that I believed could be profitable, it's clear now that my research was not as rigorous or thorough as it needed to be to support making any definitive claims about the strategy's efficacy. The data limitations opened the door to flaws and biases that I did not properly account for.

Going forward, I will be more transparent about disclosing any limitations and uncertainties in my research process. Only through rigorous testing over significant time periods can we prove that a strategy may have merit.

Skate.

 

[peter2]

Pre-inclusion bias - would certainly be a concern for trend traders if a security is delisted or suspended when the price is going up.

Two current examples are AVZ and LLL in the ASX.



Not many back-testers include code to price these anomalies to zero in their back-tests.

 

[Richard Dale]

FWIW: I've added "Suitable for backtesting" to the subscription option descriptions for Norgate Data fr the novice traders that don't really understand the implications (and hopefully might research it further).

@peter2 Whether a trading system is weekly or daily, there should be no ambiguity here. A weekly trading system (i.e. calculated on week-end close) generates a trading signal that can only be enacted on the next bar.

 

[Skate]

I do worry that some of your buys are "late". Have you researched the difference? Back-test with only new buy signals vs new and open signals.

@peter2, I understand your concern about potential delays in buy signals and the impact of the "Percentage Up" buy filter on the performance of the strategy. It was important for me to consider the differences the buy filter made in my trading results before implementation.

I have indeed researched the disparity between using certain filters to time the entry. Through extensive analysis, I have found value in sticking with the "Percentage Up" buy timing filter, even if it means sacrificing the precise timing of breakouts. By utilising the "Percentage Up" buy timing filter, I prioritize consistency and reduce the risk of false or premature buy signals. While this approach may result in some signals being delayed, it can also help filter out noise and increase the reliability of the signals generated.

It's crucial to note that every trading strategy has its strengths and weaknesses. By focusing on the buy timing filter, I prioritise the overall trend and aim to capture sustained price movements rather than attempting to time the exact breakout point. This approach can provide more stable and reliable results over the long term. However, I'm continually refining and improving all my strategies based on ongoing research and analysis.

Skate.

 

[Richard Dale]

Pre-inclusion bias - would certainly be a concern for trend traders if a security is delisted or suspended when the price is going up.

Two current examples are AVZ and LLL in the ASX.

View attachment 161942

Not many back-testers include code to price these anomalies to zero in their back-tests.

Ignoring the (HUGE) pre-inclusion bias, these are examples of systemic risk in speculative stocks.

Whilst not completely avoidable, it's far less likely in higher-cap stocks within in the larger indices.

For these two:
AVZ entered the All Ords in Mar 2018 and the S&P/ASX 200 in Mar 2023.
LLL isn't even in the All Ords (and never has been)

 

[Skate]

FWIW: I've added "Suitable for backtesting" to the subscription option descriptions for Norgate Data fr the novice traders that don't really understand the implications (and hopefully might research it further).

@Richard Dale, thank you for your response. I appreciate your proactive approach in adding the clarification "Suitable for backtesting" to the subscription option descriptions for Norgate Data. This addition will be helpful for novice traders who may not fully grasp the implications of different subscription levels and hopefully encourage them to delve deeper into the subject.

I'm glad our exchange has been productive and conducted in a respectful manner. It's essential to engage in discussions and share information in a respectful and collaborative manner, as it fosters a positive learning environment and encourages further exploration of trading concepts.

Skate.

 

[peter2]

AVZ entered the All Ords in Mar 2018 and the S&P/ASX 200 in Mar 2023.

I'm disgusted that speculative stocks like AVZ and BRN get into the ASX200 in the first place.

Note: BRN is finally being removed from the ASX200 soon (Sept 18th).

But another speculative company WBT is getting into the ASX200. Come on, S&P get serious.

 

[Richard Dale]

@peter2 The alternative is to choose your own criteria, or a higher index level (S&P/ASX 100) or an alternative market (eg. US stocks).

 

[Richard Dale]

With suitable software/programming you could also incorporate your own "meta" metholdogies too - (eg. must be in the index for more than at least 250+ trading days prior etc.)

The (current) S&P/ASX methodology is here:

https://www.spglobal.com/spdji/en/documents/methodologies/methodology-sp-asx-australian-indices.pdf

(but note that such methodology has changed over time)

 

[qldfrog]

Pre-inclusion bias - would certainly be a concern for trend traders if a security is delisted or suspended when the price is going up.

Two current examples are AVZ and LLL in the ASX.

View attachment 161942

Not many back-testers include code to price these anomalies to zero in their back-tests.

Some of the extra code you add with experience:
It is one thing to look at the past, another which costed me dearly in real system trading is the need to update /refresh the custom watchlists constituents you may use in your code.
otherwise,you slowly miss the new entrants. Raising stars often while keeping using the losers.
I moved to the premium data ...after that blunder and refresh my lists at every ab run...
This lost me thousands...
I will system trade again..

 

[Skate]

Most beginners in the systematic trading space have no idea how much of an impact that survivorship bias or pre-inclusion bias has on the validity of backtesting. FWIW: I've added "Suitable for backtesting" to the subscription option descriptions for Norgate Data fr the novice traders that don't really understand the implications (and hopefully might research it further).

The Norgate subscription page has been updated
@Richard Dale, I want to express my gratitude for including the clarification "Suitable for backtesting" in the subscription option descriptions for Norgate Data. This addition will greatly benefit those who may not have a full understanding of the impact of survivorship bias or pre-inclusion bias on the validity of backtesting. By highlighting this aspect, you are encouraging traders to conduct further research and gain a deeper understanding before committing to a subscription.

We all start somewhere, and it is crucial to make informed decisions by accounting for data biases in our explorations. Even with starter packages, there is still an opportunity to gain valuable experience and exercise creativity within the given constraints. However, it is essential to avoid over-extrapolating the results. By recognizing the limitations and biases present in the data, traders can approach their trading strategies with a more informed and realistic perspective.


Before




After



Thank you once again for taking this step to educate and empower those new to system trading with decisions to make.

Skate.

 

[Skate]

@Richard Dale, I appreciate your assistance in providing the "S&P/ASX Australian Indices Methodology PDF." It has shed light on an important aspect that I would like to further clarify.

Currently, I am subscribed to the "Silver" subscription level and have learned from the PDF that the rebalancing of the S&P/ASX Australian index series takes place regularly. The trading data is referenced to the second-to-last Friday of the month prior to the rebalancing.

Specifically, the All Ordinaries index, which is the index I exclusively trade, undergoes annual rebalancing, occurring after the market closes on the third Friday of March. Consequently, the entire S&P/ASX index series is rebalanced after the market closes on the third Friday of March.

Now, here is my question
Consider a scenario where my strategy, implemented through Amibroker with the "Filter" Watchlist and the selection of the "All Ordinaries," generates a buy signal before the market closes on the third Friday of the month of March. In this case, after the All Ordinaries is rebalanced, the buy signal gets downgraded and removed from the "All Ordinaries Index." This poses a significant dilemma, as it implies that I would never receive a "SELL" signal, which could have disastrous consequences for a trader who relies on following strategy signals.

# Am I correct in assuming this, and if so, do you have any suggestions on how I can overcome this issue?

Thank you for your attention to this matter, and I eagerly await your guidance on addressing this challenge effectively.

Skate.

 

[Richard Dale]

# Am I correct in assuming this, and if so, do you have any suggestions on how I can overcome this issue?

AmiBroker is a back testing/signals generation system but it clearly can only operate against what is provided.

There's no "portfolio manager" facility built into AmiBroker that would allow you to continue to track an open position in a previously generated signal if that security is no longer in the watchlist.

An alternative here would be for you to create a watchlist of "Open Positions" for a given strategy and run exit signals/trailing stop levels etc. against that. I'd advise doing this as Static Watchlist in NDU (in the Watchlist Library) rather than directly within AmiBroker. The reason for this is that any symbol changes are automatically handled by NDU's Static Watchlist. AmiBroker doesn't have any ability for data vendor plugins to rename symbols, so that's quite annoying (we have asked for it, multiple times).

At the Platinum level, this is not required as your backtesting code accesses a "Current & Past" watchlist where the security will remain, and you'd use the NorgateIndexConstituentTimeSeries function in your trading rules.