Dump it Here

[MovingAverage]

The problem I have with this thread (this is not a personal attack) is that a lot of folks talk about testing over a period of time and generally the assumption is the longer the time frame the better. This is rubbish.

I bet if I was to say I did a backtest (over whatever time period you care to suggest--I don't care) and my back test executed 5000 trades who would suggest that is good (statistically relevant)?

 

[Cam019]

The problem I have with this thread (this is not a personal attack) is that a lot of folks talk about testing over a period of time and generally the assumption is the longer the time frame the better. This is rubbish.

I bet if I was to say I did a backtest (over whatever time period you care to suggest--I don't care) and my back test executed 5000 trades who would suggest that is good (statistically relevant)?

If you had 120 optimizable variables - I'd still probably say it wasn't enough.

However, if you ignore my ridiculous comment for a moment - yes 5,000 trade sample size over whatever time period is statistically significant.

I do just want to say that I have been harping on about length of backtest, but that is in relation to the timeframe I trade on - monthly. I do need to go back this far to get a statistically significant sample size.

Due to trading this timeframe, I also want to have as few optimizable parameters as possible to get enough sample size. If I had 6 more variables in my system, I couldn't get enough sample size to trade a monthly system.

 

[ducati916]

This is what should be important to system traders, but is not what you suggested in point 5.

View attachment 133658

That's correct.

That is because I do not subscribe to that methodology in approaching markets. Probably 2 examples as to why.

(i) A company's CEO runs off with the secretary. Share price tanks, gapping down 30%.
(ii) A new variant of a disease is announced, market as a whole gaps down.



A random (future event) can change a trend immediately.

Thus reliance on a backtest are potentially futile or very dangerous.

So a further thought on how markets (may) change, effecting a very different outcome to one envisaged. So since 2000 the Fed has ridden to the rescue in each market blow-up. This has been increasing: 2008, 2020 being 2 obvious examples.

Moving forward, is it a 'given' that they will do so the next time? Given that they may have to make a choice between 'stocks' and their 'fiat currency'. Either choice is bad for stocks. How relevant is a backtest?

jog on
duc

 

[Cam019]

How relevant is a backtest?

How relevant is someones subjective opinion?

 

[MovingAverage]

If you had 120 optimizable variables - I'd still probably say it wasn't enough.

However, if you ignore my ridiculous comment for a moment - yes 5,000 trade sample size over whatever time period is statistically significant.

I do just want to say that I have been harping on about length of backtest, but that is relation to the timeframe I trade on - monthly. I do need to go back this far to get a statistically significant sample size.

The reality is that the number of actual trades in your backtest needs to be put into the context of how many possible trade were available during that time period--what you take and what are available are often very different. In reality there are many constraints that limit us from taking all possible trades--capital constraints, risk management, position sizing etc etc etc. So for example, if during that period for which my backtest took 5000 trades there were 1,000,000 possible trades (couldn't take them all because of constraints) I'd suggest 5000 may not be statistically relevant. But if the 5000 trade were in the content of 6000 possible trades I'd suggest it's likely to be statistically relevant.

 

[MovingAverage]

(i) A company's CEO runs off with the secretary. Share price tanks, gapping down 30%.


jog on
duc

Company CEO's have run off with their secretary before so you'd reckon historic data would have that baked in

 

[Skate]

The problem I have with this thread (this is not a personal attack) is that everyone talks about testing over a period of time and generally the assumption is the longer the time frame the better. This is rubbish.

Nailed it again.

I like to use a rough guide which I found here a while back.

My personal preference is to go back as far as the data allows me to - 1992/1993.

Martyn Tinsley who @Cam019 references, BTW, is well-credentialed that I have already made a series of posts on.

Does your trading system "really" have an edge?




Recommended viewing
I'm suggesting if you get any spare time watch the short videos that Martyn Tinsley has produced. Martyn embarks on a journey to challenge the "backtesting status quo" that prevails among system traders that are often discussed in this thread. Martyn highlights the pitfalls of herd-mentality with regards to "backtesting & optimization", which will prove to be extremely informative.

Optimising parameters
Traders often struggle to optimise their systems in order to find parameters with the most effective edge. This often results in the systems losing money in practice, especially when traded on a live account. Without understanding this fully, traders are likely to continue choosing the wrong parameters to trade.

Parameter values that provide no edge at all often appear to perform in optimisation as if they do
Every trader needs to be very aware of this anomaly. Parameter values with little or no edge can often produce far better results in optimization than the parameters that offer the best edge, & the best chance of success in the long term. If you’re not careful, it will be these sub-optimal values that you choose to trade.

Direct link to Martyn Tinsley series of YouTube videos
Martyn's educational style is a joy to watch.

https://www.youtube.com/c/Darwinexchange/videos

Skate.

 

[MovingAverage]

If you had 120 optimizable variables - I'd still probably say it wasn't enough.

However, if you ignore my ridiculous comment for a moment - yes 5,000 trade sample size over whatever time period is statistically significant.

I do just want to say that I have been harping on about length of backtest, but that is in relation to the timeframe I trade on - monthly. I do need to go back this far to get a statistically significant sample size.

Due to trading this timeframe, I also want to have as few optimizable parameters as possible to get enough sample size. If I had 6 more variables in my system, I couldn't get enough sample size to trade a monthly system.

If you're an AB user and you haven't done this before, look into the difference between scan and explore. I've got scan setup to spit out all possible trades for a period of time which is useful for putting the number of trades you actually take in a backtest in context with total possible number of available trades. It is also pretty easy to code into your backtest output.

 

[ducati916]

How relevant is someones subjective opinion?

Totally irrelevant. Which is exactly the point. A historical backtest however will, due to the empirical record, make a definitive assumption.

jog on
duc

 

[Skate]

If you're an AB user and you haven't done this before, look into the difference between scan and explore. I've got scan setup to spit out all possible trades for a period of time which is useful for putting the number of trades you actually take in a backtest in context with total possible number of available trades. It is also pretty easy to code into your backtest output.

Straight from the horse's mouth
(a) The Scan is meant as a quick tool to show raw, unprocessed, non-delayed signals as they are present in Buy/Sell/Short/Cover arrays. The scan does not do ANY processing. The output of scan is not customizable. It merely displays cells/bars where buy/sell/short/cover signals occur.
(b) Exploration is meant as a generator of fully user-customizable tables, or Excel-like tabular reports. It does NOT use Buy/Sell/Short/Cover variables. It only uses Filter variable to decide which data rows to display. You can customize the Exploration layout freely. You can define your own columns and the content of data displayed. It does NOT do any extra (hidden) processing to display data.
(c) The Backtester is a tool to perform trading system performance testing on historical data. It performs a LOT of internal processing, including trade delays, removing redundant signals, position-sizing, applying portfolio-level constraints, etc.

Skate.

 

[ducati916]

Company CEO's have run off with their secretary before so you'd reckon historic data would have that baked in

A phrase adapted from the Book of Ecclesiastes; the author complains frequently in the book about the monotony of life. The entire passage reads, “The thing that hath been, it is that which shall be; and that which is done is that which shall be done: and there is no new thing under the sun.”

The question however is twofold: will this remain true moving forward and is that actually a true statement about financial markets?


jog on
duc

 

[Skate]

If you're an AB user and you haven't done this before, look into the difference between scan and explore

"Skate's Monthly Momentum Strategy"
@MovingAverage made a great post highlighting the differences between "Scan & Explore". I'll take it one step further & show the practical example of ways to trade using the "Backtest" versus the "Exploration Analysis". I have previously canvassed the different & associated pitfalls between the two.

Using Backtest to generate signals
The Backtester is a tool to perform trading system performance testing on historical data. It performs a "LOT of internal processing", including trade delays, removing redundant signals, position-sizing, applying portfolio-level constraints, etc.




Using Exploration Analysis to generate signals
Using the "Exploration Analysis" generates fully user-customizable tables. You can customize the Exploration layout freely. You can define your own columns and the content of the data displayed. This is my preferred method to generate signals as the information is invaluable to "me"



Summary
As you can see, they are the same signals but using the "Exploration Analysis" allows me to add additional columns of information that display more information through additional coding.

Skate.

 

[MovingAverage]

The question however is twofold: will this remain true moving forward


jog on
duc

I reckon it will...human nature is what it is and CEO's (some but not all) will continue to "screw the crew". As sure as the sun comes up is man's desire to sow his seeds I think someone else said it here a few posts back--human nature hasn't changed a lot

 

[Skate]

Not now but sometime in the future
There will be members who read the posts from today as well as last week, which will resonate with them more than others. Which point of view will they accept is not for us to decide. "The Dump it here" thread is about ideas - "not a contest of ideas".

Civility when expressing an opinion
When members express an opinion, what is important "is the words" they use to express that alternative point of view or express an opinion is the true measure of this thread. I truly appreciate when valued, experienced members take the time to express those ideas & even though at times I don't agree, I take them on board & feel wealthier for it.

Acceptance
We can all agree to disagree - we can all contest our ideas but to express those views in a manner that's has been promoted recently has been a real credit to all members. Your belief system defines whether you agree or disagree with a member's post but if you don’t listen you forgo the right to understand their point of view.

A few members with conviction
I enjoy reading those who prosecute their case forcefully with conviction & when they have something to say it pays to listen. Some posts I re-read to pick up the gems they causally scatter.

Well done guys,

Skate.

 

[Skate]

Something fresh & trade-related
@ducati916 made a remarkable post about Crypto today that I would like to condense & add my opinion.

Cryptos is built on fairy dust
"Crypto's is basically like stock in an empty company that doesn't do anything except promote the sale of its own stock. Crypto's such as Bitcoin is a fool's investment of those who trade hoping to offload it on someone dumber than them who will pay more for it. Crypto trading is not economically distinguishable from Ponzi Schemes".

BOE stands for 'built on enthusiasm'
Bitcoin is built on the enthusiasm of others that doesn’t have any intrinsic value but mainstream adoption is showing no signs of letting up anytime soon. Bitcoin is currently in unchartered territory & nobody knows when the current bull run is going to stop or take a pause.

I can't understand it
Ultra-easy monetary policy has pumped trillions of dollars into cryptocurrencies & perhaps most important of all is the institutional interest which has skyrocketed this year expanding their investments in the world’s premier digital currency. Big names including Fidelity, BlackRock & Citigroup have followed suit are just a few. Also, well-respected long-term investors are getting in on the act with high volumes of money being splashed about.

Summary
I want to finish with Duc's assessment:
"Bitcoin is a fool's investment of those who trade hoping to offload it on someone dumber than them who will pay more for it".

Skate.

 

[ducati916]

I reckon it will...human nature is what it is and CEO's (some but not all) will continue to "screw the crew". As sure as the sun comes up is man's desire to sow his seeds I think someone else said it here a few posts back--human nature hasn't changed a l

I reckon it will...human nature is what it is and CEO's (some but not all) will continue to "screw the crew". As sure as the sun comes up is man's desire to sow his seeds I think someone else said it here a few posts back--human nature hasn't changed a lot

So if we agree on 'human nature', then we will see further innovation within financial products, to which the true ramifications are unknown to their creators and especially the users.

So in 2008, CDOs cubed was an extension of an already dangerous idea. Currently, the world derivative exposure is now at a 'Quadrillion + (1000 Trillion). Is that priced into the market? Let's say that it is; then Mr Skate's idea of an acceleration of downside is relevant (2020) when trying to build a system. Both 2008 and especially 2000, gave traders ample time to exit if they were paying attention.

2020 was much faster.

The next blow-up will be faster yet, driven by unseen, interlinked exposures, that won't even start in the stock markets. Is your portfolio ready for a 30% decline in a major index over a 24/48hr time period from your backtesting? The 'speed' of a serious decline is accelerating. Of course this is more of an exit issue (potentially) than an entry issue.

jog on
duc

 

[MovingAverage]

So if we agree on 'human nature', then we will see further innovation within financial products, to which the true ramifications are unknown to their creators and especially the users.

The next blow-up will be faster yet, driven by unseen, interlinked exposures, that won't even start in the stock markets. Is your portfolio ready for a 30% decline in a major index over a 24/48hr time period from your backtesting? The 'speed' of a serious decline is accelerating. Of course this is more of an exit issue (potentially) than an entry issue.

jog on
duc

For sure--have no doubt that further innovation and regulatory changes will drive changes in the market and its behaviour. But just so we are clear, I do subscribe to the theory that markets and their behaviour has and will continue to change but that doesn't mean past market behaviour is irrelevant.

Whether the next blow-up will be faster yet, neither you, I or anyone else can predict what the markets will do in the future so as they say...time will ultimately tell us how the markets will behave in the future. I am not advocating nor have I advocated that designing a mechanical system on historic data will tell you how your system will behave in the future, but using techniques like walk forward testing will and does give you a degree of insight into how your system will deal with future market behavior. There is no such thing as certainty in trading (no matter what your trading style is) but there are degrees of probably and I am very comfortable with that.

 

[qldfrog]

If you're an AB user and you haven't done this before, look into the difference between scan and explore. I've got scan setup to spit out all possible trades for a period of time which is useful for putting the number of trades you actually take in a backtest in context with total possible number of available trades. It is also pretty easy to code into your backtest output.

I would agree with that
Just but there is a fine line between all candidates and final choices..
many systems mine included roughly take the smallest SP first; which introduce a bias mostly to small caps..
[rough approximation the smaller the SP the smaller the company..]
if ranking is doing most of the filtering vs the selective being done for a smaller pool of candidates l, it can make a big difference;
Hope I convey my thoughts?
Anyway, I agree that the total number of trades possible vs nb of trades taken is indeed statistically relevant

 

[qldfrog]

for @Cam019 :
without trying or needing to relaunch the debate, and while I believe this is worth zip:
I run my current strategies from the 1/1/1992 :
Remember when buffaloes, donkeys or man power was still the #1 machinery used in what is 2021's #1 economic power ;-)
Note that these have been designed and optimised for the 2019 onward periods
These are the results on both XAO (with history) and my FPO definition
They are mostly live or will be on FPO.

you will note that my results are more statistically relevant than some other offered and the results are not too shabby especially looking at max DD
if I take the last one :QFsec

less pleasant shape QFDuc:

and the laggard:
DLGuppy whose only purpose is to act shiftily to be out on crash and is by far the worst

Not too bad when you consider that these have been designed and optimised for the 2019 onward periods...
So Yes feel free to have a quick check since WWI if you want but develop for recent times
Please note that there was no inflation indexation in that code

 

[Cam019]

feel free to have a quick check since WWI

This made me laugh!

Look, we could discuss our beliefs until we are blue in the face, or dead. However, I would much rather agree to disagree and accept each other are going to do things the way they believe is right. I think @MovingAverage said this earlier, but everyone doing different things is what creates trading opportunites in the market. If we all did the same thing, the same way, our systems wouldn't work.

I appreciate you putting that up, even if you think it has no value.

Just one question from me.

Did you purposely design the systems to have approx. less than 50% exposure, or is that simply a bi-product of what I call 'curve-fitting' your system to more recent market data?