Dump it Here

[Skate]

The Beauty of Paper Trading is You can Race One against the Other

Well here is the race
This is a short backtest of 720 days for a comparison.



Skate.

 

[Captain_Chaza]

IMHO Paper trading should as close to the real thing as possible
I hope you are using Monday mornings OPEN Price and not Friday's CLOSE
Otherwise it will just be toilet paper and nothing will be learnt

 

[Skate]

IMHO Paper trading should as close to the real thing as possible
I hope you are using Monday mornings OPEN Price and not Friday's CLOSE
Otherwise it will just be toilet paper and nothing will be learnt

The system triggers signals "after the close" each Friday
In real trading, these signals would be placed in the pre-auction at the offer price, with a 3% premium above Friday's closing price. However, since this is just a paper trading exercise, the trades will be simulated rather than actually executed. The goal is to see how the signals would have performed if traded according to the plan.



Skate.

 

[Captain_Chaza]

Crikey Capn Skatie
You play around with Backtests as if they are a new toy you have found
I find them USELESS and extremely dangerous and misleading

I am only interested in the NOW and FUTURE learning experiences

 

[Captain_Chaza]

The system triggers signals "after the close" each Friday
In real trading, these signals would be placed in the pre-auction at the offer price, with a 3% premium above Friday's closing price. However, since this is just a paper trading exercise, the trades will be simulated rather than actually executed. The goal is to see how the signals would have performed if traded according to the plan.

View attachment 161897

Skate.

On first glance of TWO the 3% Premium does does not seem to have applied on the IVZ and the LRS

 

[Skate]

@Captain_Chaza here is my evaluation of the two systems
In a nutshell - # System 1 seems better suited to trading the "ASX All Ordinaries" based on this backtest evaluation. I'll let the metrics speak for themselves with regards to # System 2

# System 1
Trading against the ASX All Ordinaries

# System 2
Your suggestion to backtest using the ASX 300 excluding the ASX 200

# System 1 had a higher net profit (+81.72%) compared to System 2 (+51.00%).

# System 1 had a higher annual return (34.75%) than System 2 (22.85%).

# System 1 had a higher net risk adjusted return (171.95%) than System 2 (116.59%).

# System 1 had a higher number of winning trades (81 vs 67) and a higher percentage of winning trades (44.02% vs 51.54%).

# System 1 had a higher average profit per winning trade ($2236 vs $1674)

# System 1 had a lower max drawdown (-12.42% vs -10.11%)

# System 1 had better risk/reward metrics overall - higher Profit Factor, higher Risk-Reward Ratio, higher CAR/MaxDD, higher RAR/MaxDD.

# System 1 had a higher expectancy ($444 vs $392).

Overall
System 1 appears to have significantly better performance across key metrics like net profit, risk-adjusted returns, winning trades percentage, drawdowns, and risk/reward metrics. The backtest results indicate System 1 has superior performance compared to System 2 for this strategy and time period.



Skate.

 

[Skate]

I find them (backtests) USELESS and extremely dangerous and misleading

@Captain_Chaza, that's what is so exciting about traders we are all different, having different methods of reaching the same outcome. (profits)

Backtesting provides several important benefits
I understand your scepticism about backtesting. You raise some valid points about the limitations and potential pitfalls of relying too heavily on backtesting alone. Backtests are only as good as the data and assumptions used - garbage in, garbage out as @Richard Dale pointed out recently. Using the wrong data can lead to misleading results.

Past performance does not guarantee future results
Backtests don't account for real-world trading realities like slippage, liquidity issues, and execution delays. Performance in live trading may differ. I'm first to admit that it's easy to over-optimize and curve fit based on past data, leading to unreliable results. This needs to be avoided at all costs.

Psychological factors like emotion and discipline cannot be modeled in backtests
Traders may put too much faith in backtesting results without considering real-world performance. You're right that backtesting has limitations and one should not rely on it alone.

Consider real-world trading conditions
I use backtesting as a gauge of strengths/weaknesses, not as the final word. I focus on principles and logic rather than just backtest metrics. Traders should consider backtesting as one tool in the process, along with risk management, and ongoing review. Remember backtesting supports, but does not guarantee live performance. Be cautious when applying results. A balanced approach is best.

In summary
Backtesting provides an invaluable estimate of strategy performance before trading live. Comparing strategies by backtesting them on the same data allows you to make an informed choice on which strategy to trade going forward. It is a key tool for developing and evaluating trading systems.

Skate.

 

[Skate]

On first glance of TWO the 3% Premium does does not seem to have applied on the IVZ and the LRS

@Captain_Chaza, it is hard to add a 3% PREMIUM to a low-value amount, however, the applied "mathematical code" tries its best.

Skate.

 

[Skate]

You play around with Backtests as if they are a new toy you have found
I find them USELESS and extremely dangerous and misleading

The Limitations of Amibroker Backtest Portfolios as Trade Signals
A key disadvantage of using Amibroker backtest portfolio models as trade signals is the mismatch between backtest data and real-time market prices, especially on open gaps.

For example, a model identifies stock XYZ as a buy at Friday's $5 close with a 3% filter, allowing entry up to $5.15. But XYZ opens Monday at $5.50.

The Exploration Analysis method would NOT take the trade, as the $5.50 open exceeds the $5.15 limit. However, the Backtest Portfolio method WOULD buy XYZ at $5.50, disregarding the gap up.

Once executed, trades can't be deleted or modified in the backtest portfolio. So the intended $10k position size is exceeded, locking up buying power.

This illustrates the pitfalls of relying on "backtest portfolios" for trade signals
(1) Backtest data differs from real-time prices
(2) Open gaps violate predefined entry/exit limits
(3) Can't modify or delete portfolio positions in hindsight

The bottom line is that backtests should inform trading, not dictate it. Use real-time analysis, not just mechanical signals. Backtest portfolios don't react dynamically like a trader. Understand the limitations before relying on backtest outputs.

Skate

 

[Skate]

"Backtest Portfolio Signals"
To illustrate the issues with backtest portfolio signals, let's look at what happens when the markets open on Monday.

The issue in using Amibroker's "Portfolio Backtest Report" it recalculates all metrics using Monday's opening prices. This means the data can shift significantly from the original backtest results.

For example, a position that looked attractive on Friday's close based on the backtest may no longer meet the entry criteria at Monday's open. However, the portfolio model has already executed the trade.

I will display the "before and after" Portfolio Backtest Reports to demonstrate how the numbers change from Fridays close to Mondays open - so you can understand what I'm talking about.

The key takeaway is that backtest portfolio models don't adjust like a discretionary trader would. The backtest locks in positions that may no longer be prudent when the market opens. This reiterates why mechanically relying on backtest portfolio signals has inherent limitations compared to real-time human analysis and risk management.

Skate.

 

[Skate]

Amibroker Exploration Analysis
The screenshot above shows the raw trading signals generated by the "Dual Breakout Strategy" using "Amibroker Exploration Analysis". To simplify the initial paper trading exercise, only the first 10 signals were used to fill the 10-position portfolio. This allowed focus on a smaller subset of trades to evaluate the strategy.

The full Exploration Analysis displays every trade signal in the backtest for the Dual Breakout system. But for practical purposes, the first 10 signals provide a reasonable sample to paper trade. This approach prevents taking on an unrealistic number of positions while assessing the viability of the strategy.

Starting with a limited portfolio makes it easier to track trades closely and get a feel for how the system performs. Once the strategy is proven through paper trading, the portfolio could then be expanded to hold more positions as warranted by backtesting results and real-world performance.

Skate.

 

[Skate]

Exploration Analysis code versus the Backtest Portfolio method
One way to trade signals from Amibroker backtests is to use the Exploration Analysis code. This lets you set hardcoded limits on trades to manage risk.

For example, the code above has a 3% filter that calculates a buy offer price as 3% above the prior close. It rounds the price up and determines position size based on the buy offer. The goal is to buy at the pre-auction price or "not at all". If the opening gap exceeds the 3% limit, "no trade is taken". This enforces discipline around entries.

In contrast, the Backtest Portfolio method in Amibroker recalculates signals at each open using the new prices. If a stock gaps up more than 3%, the backtest portfolio will still take the trade at the higher open price, ignoring the limit. As I said previously, once executed, these portfolio trades cannot be modified or deleted in hindsight. So unwanted trades can be triggered on gaps, locking up buying power unintentionally.

The Exploration Analysis code avoids this issue by "filtering trades" against preset limits. The backtest portfolios react blindly to new prices, even if signals are no longer valid on gaps. The key takeaway is that hardcoded stops and limits are preferable to purely mechanical backtest signals. Exploration Analysis allows disciplined risk management versus backtest portfolios which lack hindsight and discretion.

So while backtests are useful for strategy development, trade signals should come from real-time analysis with human oversight. Understand the limitations before using backtest portfolios for entries and exits. The "Exploration" code enforces the risk controls you want. The backtests may generate signals that seem attractive but require additional scrutiny before implementation.

Skate.

 

[Skate]

I sometimes describe Amibroker as using "dumb math" which may seem unkind, but it highlights limitations in handling dynamic real-world numbers versus human discretion. Specifically, Amibroker calculates position size based on static equity while I dynamically adjust for changing account balance, putting more money to work as the account grows.

Another issue is that Amibroker backtests rely on closing prices for trades. However, in live trading, we can't know the opening price in advance when the market opens each day. So there is often a gap between the prior close and the current open. My trades execute at these unknown opening prices, which frequently diverge from the backtest assumptions.

To manage this gap risk, I code stops and limit orders into my strategies. But sometimes price gaps are larger than my coded limits, requiring manual intervention to override the signals. Amibroker backtests don't account for this human discretionary adjustment I apply in live trading.

Company announcements
Amibroker also can't filter company announcements, like takeovers or capital raises. I manually check news and may override buy signals if fundamental conditions have changed. Backtests generate signals oblivious to real-world events. Additionally, backtests assume perfect order fills but real trades have execution variability. The simplified backtest gives different results than the imperfect realities of live markets.

Given these limitations, I use metrics like the Megan ratio and geometric mean to evaluate systems beyond just net profit. These help assess the consistency of returns and account for compounding growth when reinvesting profits.

A trading plan overrides a trading strategy
The key point for new traders is understanding that Amibroker is a useful tool but has gaps versus live trading. The backtest serves as a guide but not a crystal ball. I combine quantitative signals with qualitative judgement, discretionary risk management, and real-world awareness. Experience shows mechanical backtest results should inform but not dictate actual trades. Maths is just one input to holistic trading success.

It's time to hop off the soapbox



Skate.

 

[martyjames]

Hi Skate

Have you looked at using Realtest instead of Amibroker? I switched to RT about 2 yrs ago and am so glad i did. Its WAY simpler to code atrategies in RT than AB and makes AB look very 'clunky' in my eyes. I have developed 2 systems in RT that i trade that i would have struggled to code in AB. Plus the support forum is friendly and very helpful (complete opposite to scary AB forum!!)


cheers
Marty

 

[Skate]

Hi Skate

Have you looked at using Realtest instead of Amibroker? I switched to RT about 2 yrs ago and am so glad i did. Its WAY simpler to code atrategies in RT than AB and makes AB look very 'clunky' in my eyes. I have developed 2 systems in RT that i trade that i would have struggled to code in AB. Plus the support forum is friendly and very helpful (complete opposite to scary AB forum!!)


cheers
Marty

Hi Marty, thanks for the recommendation to check out Realtest. I appreciate you taking the time to share your experience switching from Amibroker to Realtest. It's helpful to hear that you find Realtest simpler for coding strategies and that the forum is more beginner-friendly.

While I'm pretty happy with Amibroker so far, I'm certainly open to exploring other options, especially if they can make strategy development easier. I'll take a look at Realtest and see how it compares. Even if I don't make a switch now, it's great to be aware of alternatives in case I do want to try something new down the road. I'm glad you found an option that works better for your trading system development needs.

Appreciate you looking out for me, and thanks again for the insight!

Skate

 

[qldfrog]

Hi Marty, thanks for the recommendation to check out Realtest. I appreciate you taking the time to share your experience switching from Amibroker to Realtest. It's helpful to hear that you find Realtest simpler for coding strategies and that the forum is more beginner-friendly.

While I'm pretty happy with Amibroker so far, I'm certainly open to exploring other options, especially if they can make strategy development easier. I'll take a look at Realtest and see how it compares. Even if I don't make a switch now, it's great to be aware of alternatives in case I do want to try something new down the road. I'm glad you found an option that works better for your trading system development needs.

Appreciate you looking out for me, and thanks again for the insight!

Skate

and to be honest I did not find RT that simpler, actually easier IMHO to make a system butvery hard to ensure the code is doing what is supposed to do.
But support is outstanding, truly and yes, simple code is MUCH simpler.
I did covert one system, then compared with AB this revealed some AB issues..but also some bizarre RT signal..then I got no more money to trade and busy with move and selling farm.and trial ended

 

[Richard Dale]

However, the Backtest Portfolio method WOULD buy XYZ at $5.50, disregarding the gap up.

The issue here is one of programming skill. You can program AmiBroker to do what you wish.

e.g. https://www.amibroker.com/kb/2014/11/26/handling-limit-orders-in-the-backtester/

This is a short backtest of 720 days for a comparison.

Once again you have survivorship bias and lookahead bias occurring. Backtesting on the CURRENT constituents means you are ignoring any historical changes. Whilst 720 days doesn't seem that long, the last 2 years there have been 140 additions to the All Ordinaries index at different times and 136 removals. That's quite a turnover of constituents for an index that nominally has up to 500 constituents.

Trying to make any sort of decisions about trading system's performance with using a survivorship bias data set along with look-ahead issues (pre-inclusion bias) is just nonsensical.

Here's a worked example of the flaw:

LIN.au is a current member of the All Ords (as at 3 Sep 2023).
Your backtest is run against the current All Ords, with a start date of two years ago.
Backtest signals were generated against LIN.au from the start of your backtest (Sep 2021) through to today.

However, LIN.au only entered the All Ords on 20 Mar 2023.

So that entire "10 bagger" period on LIN.au from your backtest is completely misleading. It wasn't a member until Mar 2023 yet your system probably would have taken trades against it. There was no way of knowing in Sep 2021 that LIN.au was going to become a constituent of the All Ords 1.5 years in the future, unless you have a time machine.

Others have written about this:

How much does not having survivorship free data change test results?

Over the last month several people have asked me how important it is to have survivorship-free data. For any researcher this is an important question to understand how the different data can change…

alvarezquanttrading.com

Profitable systematic trading is indeed possible, and trading rules can be simulated - but using incomplete data and poor modelling won't get you there.

We offer the ability to incorporate such conditions within AmiBroker:

Norgate Data - AmiBroker Usage

A lower subscription level that doesn't incorporate these capabilities can be upgraded at any time on a (modified) pro-rata basis:

Norgate Data - Stock Market Packages

 

[Skate]

The issue here is one of programming skill. You can program AmiBroker to do what you wish.

e.g. https://www.amibroker.com/kb/2014/11/26/handling-limit-orders-in-the-backtester/



Once again you have survivorship bias and lookahead bias occurring. Backtesting on the CURRENT constituents means you are ignoring any historical changes. Whilst 720 days doesn't seem that long, the last 2 years there have been 140 additions to the All Ordinaries index at different times and 136 removals. That's quite a turnover of constituents for an index that nominally has up to 500 constituents.

Trying to make any sort of decisions about trading system's performance with using a survivorship bias data set along with look-ahead issues (pre-inclusion bias) is just nonsensical.

Here's a worked example of the flaw:

LIN.au is a current member of the All Ords (as at 3 Sep 2023).
Your backtest is run against the current All Ords, with a start date of two years ago.
Backtest signals were generated against LIN.au from the start of your backtest (Sep 2021) through to today.

However, LIN.au only entered the All Ords on 20 Mar 2023.

So that entire "10 bagger" period from your backtest is completely misleading.

Others have written about this:

How much does not having survivorship free data change test results?

Over the last month several people have asked me how important it is to have survivorship-free data. For any researcher this is an important question to understand how the different data can change…

alvarezquanttrading.com

Profitable systematic trading is indeed possible, and trading rules can be simulated - but using incomplete data and poor modelling won't get you there.

We offer the ability to incorporate such conditions within AmiBroker:

Norgate Data - AmiBroker Usage

A lower subscription level that doesn't incorporate these capabilities can be upgraded at any time on a (modified) pro-rata basis:

Norgate Data - Stock Market Packages

@Richard Dale, I agree that using current constituents for backtesting can introduce lookahead bias and survivorship bias. It is better to use a dataset that reflects the actual historical composition of the index over time.

The example you provided with LIN.au illustrates well how a stock's inclusion in the index midway through the backtest period can lead to misleading results. I appreciate you linking to the article discussing this issue in more detail.

You make a fair criticism of the potential flaws in my backtest approach. I agree it is important to use more robust historical data that properly accounts for index additions/deletions over time to avoid these biases. Thank you for raising this issue - it is something I will need to address to improve the reliability of my backtests going forward. I appreciate you taking the time to explain this and provide suggestions on alternative data sources.

Skate.

 

[Skate]

Profitable systematic trading is indeed possible, and trading rules can be simulated - but using incomplete data and poor modelling won't get you there.

@Richard Dale, thank you for raising an important issue - you make an excellent point in cautioning new traders about the potential pitfalls of using Norgate's lower-tier data subscriptions without a full understanding of the limitations. Your feedback highlights the need for a clear disclaimer on our site to ensure transparency about the biases that can be introduced by limited historical data.

Thinking of Using Norgate Data? Beware of the Pitfalls for New Traders
If you're a new trader looking to backtest potential trading strategies, you may have come across Norgate Data as a provider of historical stock market data. While Norgate can be useful, new traders should be very careful about putting too much faith in results obtained using their entry-level data subscriptions.

Norgate offers tiered subscriptions for Australian stock data. The cheapest "Silver" subscription only includes the last 10 years of data for currently listed stocks. This seems to allow you to backtest strategies at an affordable price. However, this limited data can be dangerously misleading if you don't understand the implications.

More experienced traders know these kinds of limited data sets are riddled with biases like survivorship bias and lookahead bias. Survivorship bias means failed companies are excluded since you only have data for companies still listed today. Lookahead bias means you have future index constituents in your historical data.

These biases can lead to grossly overestimating the potential returns of the strategies you backtest. Strategies optimised in such flawed data often perform much more poorly in the real world. This is why professional traders pay large sums for clean, unbiased data sets that accurately reflect historical market conditions.

As a new trader, you likely don't have the experience to anticipate these pitfalls. It's easy to fall into the trap of thinking the strategies you develop are far more profitable than they really would be. You could end up "allocating significant trading capital sub-optimally" or even "losing money on strategies" that appeared sound on biased data.

My recommendation is to think twice before using Norgate's entry-level packages for rigorous analysis as a new trader. The data gaps and integrity issues likely outweigh the small cost savings. Consider paying more for their higher-tier offerings or alternative data sources to avoid developing misleading strategies.

Skate.

 

[Skate]

"The Dump it here" thread (warning)
This thread is aimed at helping those just starting out on their trading journey. When starting out system trading with Amibroker, reliable data is crucial. So I want to provide a word of caution to budding strategy developers about potential pitfalls to avoid. Norgate Data offers lower-tier subscriptions that provide only limited histories for currently listed stocks. While affordable, these entry-level packages could lead to misleading or inaccurate backtesting results if the data limitations are not properly understood.

The false precision of detailed backtests on flawed data can provide false confidence
Testing only current stocks may overstate results, as failed companies are excluded through survivorship bias. Additionally, including future index constituents introduces lookahead bias. Short histories may fail to capture full market cycles and lead to strategies that falter when cycles change. Strategies reliant on long histories or delisted companies may be inappropriate if the data excludes those very factors. Optimising strategies for historical biases rather than future trading realities could result in poor allocation of your precious trading capital.

My advice is to thoroughly understand what is omitted from the entry-level data packages
Avoid over-reliance on long history-dependent strategies. Use discretion in strategy development rather than fully automated based on potentially biased optimisations. Focus more on conceptual viability using the data available. And consider upgrading to higher-tier subscriptions for rigorous analysis if affordable before risking significant funds.

With all due respect
The goal is not to disparage Norgate, but rather to ensure new developers use appropriate data for their needs and understand the implications of working with limited datasets. We all start somewhere, so begin wisely by accounting for data biases in your exploration. You can still gain valuable experience working creatively within the constraints of starter packages. Just avoid over-extrapolating the results.

Skate.