- Joined
- 28 December 2013
- Posts
- 6,392
- Reactions
- 24,319
ENHANCING THE QUALITY OF HISTORICAL SIMULATIONS
In this section, we discuss how to increase the quality of a backtest by focusing on six areas
known to present challenges, namely: data quality, data representativeness, statistical integrity,
modelling and generalisation, costs and constraints, and performance evaluation.
Data Quality
Survivorship Bias: Survivorship bias is defined as a type of selection bias that occurs
when analyses are conducted only on the data that have survived a selection process while ignoring
those that did not. This bias can lead to skewed results and incorrect conclusions because the nonsurviving entities typically differ from the survivors in significant ways.
The impact of survivorship bias is particularly pronounced in the analysis of mutual and
hedge fund performances, as well as investment strategies. For example, including only those
funds that are still active at the end of the period can significantly overestimate the average fund
performance, as the worst-performing funds go out of business and are not reported on.
Point-in-Time Considerations and Restated Data: Point in time refers to ensuring that
data used in analysis or backtesting is reflective of the information that would have been available
to researchers at that specific point in time. This is particularly relevant in the case of restated data,
which involves adjustments made to previously reported financial figures, such as earnings,
revenue, or other financial metrics, due to errors, accounting changes, or compliance with new
accounting standards. Restatements can significantly alter historical financial statements, affecting
the accuracy of financial analysis and the models that rely on those statements. For practitioners,
it is essential to manage restated data carefully; this often means using a point-in-time database to
capture financial data as it was originally reported, without incorporating restatements or
adjustments made after the fact.
Incorrect and Missing Data: Incorrect data are any errors or inaccuracies in the dataset.
These inaccuracies can stem from various sources, such as data entry errors, issues with data
collection methods, or problems in data transmission. Before analysis, data should be thoroughly
cleaned, which involves checking for and correcting errors, ensuring consistency across datasets,
and verifying data accuracy against reliable sources.
Missing data occurs when information is absent from the dataset. This can happen for
various reasons, such as system errors, data corruption, or when the data was never recorded or
collected in the first place. Missing values can be imputed using assorted statistical techniques,
such as mean imputation or regression imputation, or more sophisticated methods like multiple
imputation or machine learning-based approaches. Depending on the context, forward filling may
be a good option to avoid introducing additional complexity through imputation. In some cases,
missing data can be augmented from alternative sources, although this approach requires careful
consideration to ensure the compatibility and reliability of the augmented data.
Dealing with Outliers: In the development of investment strategies, it is essential to
evaluate the role of outliers – that is to say, extreme values that deviate significantly from the rest
of the data. Strategies that capitalise on these rare occurrences might not be sustainable, as the
outliers may represent unique, non-recurring events. Consequently, relying on such anomalies
could lead to strategies that perform exceptionally well under specific conditions but fail to
generalise across different market environments.
When incorporating models that are inherently sensitive to outliers, such as linear
regression, care must be given to the nature and impact of these extreme data points. Practitioners
must ascertain whether outliers arise from errors in data collection or entry – such as misreported
financial figures – or if they reflect genuine market phenomena that could offer valuable insights.
Data Representativeness
Sample period selection bias is the primary concern in the area of data representativeness.
This type of bias occurs when the time frame chosen for a backtest inadvertently influences the
results, leading to conclusions that may not be robust across different market conditions or time
periods. When designing all-weather investment strategies (Lopez de Prado 2019), it is essential
to include data from a wide range of market conditions rather than periods of optimal performance
alone. This ensures the strategy's robustness across different market cycles. However, when
creating tactical investment strategies, which, as Lopez de Prado (2019a) describes, are designed
for specific market conditions, it may be more appropriate to select targeted time frames. The MinTrack-Record-Length algorithm described by Bailey and Lopez de Prado (2012) can help
determine the number of observations needed to validate a strategy's effectiveness across diverse
market scenarios.
Statistical Integrity
Data Mining and Data Snooping: The concepts of data mining and data snooping refer
to the process of searching through large datasets to identify patterns, relationships, and trends that
can be used to develop investment strategies. This approach to strategy development should be
avoided, as it very easily leads to selection bias through the improper application of data analysis
techniques to uncover patterns in data that appear to be of statistical importance. Data snooping
(p-value hacking) involves researchers repeatedly probing various subsets of data or conducting
5 numerous tests (potentially on the same data) until they achieve a result that seems meaningful
(White 2000; Sullivan, Timmermann, and White 1999).
Accounting for Selection Bias under Multiple Testing: Backtests need to account for
the number of trials run and their statistics so that a discount measure can be applied to the
performance metrics (Lopez de Prado 2018, 2020). This is covered in greater detail in the next
section.
Modelling and Generalisation
Look-Ahead Bias: Look-ahead bias is the mistake of using data from the future as if it
were point-in-time data to make investment decisions. We separate this from the problem of not
using point-in-time data as we define it because of the research process. This approach can lead to
inflated performance metrics and unrealistic expectations as to the profitability of investment
strategies.
Most typically, this mistake is made by not applying the appropriate lag to the indicators
or signals. It also commonly occurs when computing statistics on the entire data sample rather than
a rolling computation – for instance, using the entire sample when computing the mean and
standard deviation of a z-score to normalise a signal.
Introducing an Embargo Period: An embargo period is a hold-out sample of data, a test
set, typically the most recent two to three years. Practitioners should fit models and make design
choices regarding their trading strategies based on the in-sample period, and once the strategy is
ready to be validated, the results can be produced for the embargo period. Any market anomaly
that was exploited in the training set should also be present in the embargo period.
Costs and Constraints
An oft-overlooked step is the incorporation of trading costs and the constraints relating to
short selling and liquidity.
Transaction Costs: Neglecting to account for transaction costs leads to a higher false
positive rate. Higher turnover strategies incur greater expenses, necessitating a more substantial
return to be viable. Omitting these costs from an evaluation leads to inflated performance metrics.
Borkovec and Heidle (2010) provide a good introduction to the types of costs and their
impact on trading. Notably, they split costs into three types. The first, brokerage costs, are easily
estimated using historical data and include commissions and fees; two examples of fees are
custodial and transfer fees. The second, trading costs, consist of four components: delay costs, bidask spread, market volatility/trend costs, and market impact. Finally, opportunity costs are the
costs of not completing the full order. The authors highlight that market impact has the highest
cost (and is difficult to model), whereas commissions and spread have the lowest effect and are
easily estimated. Transaction costs can be estimated either empirically, using historical data, or
analytically by attempting to build a model.
If you disagree with someone please don't attack or insult them, but present your argument or alternative view in a polite and respectful way. This leads to constructive discussion and debate, while aggression and personal attacks lead to nothing but off topic posts, so please think very carefully before you post.
I also note that all so-called "trading systems" appear have been snowflaked (discontinued reporting) on one day of negativity (where they gave up 2 months of gains in 1 day). Robust - nope!
@Skate # (1) I remain skeptical about using a one indicator system when all of the entries and exits are based solely on this one indicator. Usually this type of system doesn't perform well in all market conditions and therefore makes it dangerous for new traders to think that is all that they need to do to trade the markets, it could lead to big losses for new traders. If there is more to the system that you are using then this should be made clear to new traders. #(2) The danger, in my opinion, is if a new trader thinks that it's easy to develop a system like this that works consistently in all market conditions, then they may get themselves into trouble.
@Richard Dale let me address some of your statements:
#1. "The so-called "proven" "trading systems" presented in this stupidly-long thread (hint: create a new thread for each system, already) are sadly lacking"
My Response to Richards's First Statement
@Richard Dale in a previous post, I listed every time I used the word "proven" and its context. I have never claimed that the systems I posted about are "proven": https://www.aussiestockforums.com/threads/dump-it-here.34425/post-1291439
Skate.
I'm sure we'll continue to see more platitudes/motherhood statements posted in this thread.
The "wisdom" provided that describes "false signals", "improving precision". "increasing likelihood of successful trades", "more informed decisions", "enhancing accuracy" etc. are unjustified and unproven platitudes. These platitudes seriously look like they're written with Chat-GPT.
Either stop using such unproven (or unprovable) statements or provide empirical/statistical proof incorporating aspects of this paper, or I will continue to call out this practice of poor terminology. If you hadn't guessed it, I'm all for the scientific method and will call out poor/bad science when I see it.
If you hadn't guessed it, I'm all for the scientific method.
Have you ever wondered why you never seem to get positioned?and for the novices .. a truly bad example
i will call this the lunatic knife-catching style
earlier this morning i picked up ( extra ) TEG , the order was for 0.7 cents but the market re-opened at 0.6 cents ( lucky me strikes again )
and will be looking to reduce @ 1.5 (ish ) ... eventually , and dilute previous losses ( i have held since 2011 )
now this will NOT ( in my prediction ) crystallize on overall profit , but if successful will dilute my losses in this stock ( and increase the holding )
... but the announcement was negative , and the market reaction came right at me , ( thanks sellers )
We use cookies and similar technologies for the following purposes:
Do you accept cookies and these technologies?
We use cookies and similar technologies for the following purposes:
Do you accept cookies and these technologies?