Backtested Systems, Lies, Damned Lies, & Statistics

[ducati916]

Snake

The mission of arbitrage is to correct inefficiencies. People who do this pick apart strategies, concepts etc, not limited to trading/investing systems though, until they find an inefficiency or a series of inefficiencies. A bit like investing in undervalued stocks, or not?

My apologies, I assumed that you knew what an arbitrage actually was.
It is a risk free trade taken between two securities of the same issuer, but priced to provide a spread.

The spread is the risk free return. By way of example;
A Preferred convertible is selling at 1000, convertible into 25 shares of common; thus the common is fairly valued at $40.00

If however, the common is selling at $43.98, there is a 9% spread.
By purchasing the Preferred at $1000.00 and selling short the common at $43.98 or above, you lock in a 9%+ risk free return in 1/day or 1,980% annualised. If you should leverage that say 10 times, and why would you not, that is a 90% risk free return/day

But, inefficiencies are not always found.

Not every day.
But certainly enough to keep me interested.

Therefore, move on to the next strategy, concept etc.etc. and start again. The window of opportunity is extremely important, because you are competing with other people doing the same, looking for inefficiencies. If they beat you to it you can miss out.

No, you keep a % of your cash available for this strategy, and have alternate strategies available to fill in the slower times.

So how can arbitrage be the holy grail if time is not an issue?

Because you cannot lose money.
That's pretty cool in my book.
jog on
d998

 

[tech/a]

There are 3 issues,re arbitrage.

(1) Occurence.
(2) How long the inefficiency manifests itself.
(3) Highly correlated to (2) Finding one/them.

Even if they could be spotted once a year then $ x standing buy in an account would be risk free and your bank manager would see to it that his as well as all of your savings were available for immediate use.

 

[markrmau]

My apologies, I assumed that you knew what an arbitrage actually was.
It is a risk free trade taken between two securities of the same issuer, but priced to provide a spread.

That is a rather limited view on arbitrage. Arbitrage is not limited to mispricing between securities.

A few months ago, the SMH provided a coupon to get a free salad bar roll from Mcdonalds.

The SMH cost about $2 or what ever, and the roll is priced at $5 or so. So if you wanted the roll from Mcdonalds, a simple arbitrage deal was to buy the SMH.

But the point is that it is still arbitrage. In a completely efficient market, all the SMH's would have been snapped up, and poor old Mcdonalds would have excessively long queues until the time value of the person lining up came to about $3

 

[happytrader]

Snake



My apologies, I assumed that you knew what an arbitrage actually was.
It is a risk free trade taken between two securities of the same issuer, but priced to provide a spread.

The spread is the risk free return. By way of example;
A Preferred convertible is selling at 1000, convertible into 25 shares of common; thus the common is fairly valued at $40.00

If however, the common is selling at $43.98, there is a 9% spread.
By purchasing the Preferred at $1000.00 and selling short the common at $43.98 or above, you lock in a 9%+ risk free return in 1/day or 1,980% annualised. If you should leverage that say 10 times, and why would you not, that is a 90% risk free return/day



Not every day.
But certainly enough to keep me interested.


No, you keep a % of your cash available for this strategy, and have alternate strategies available to fill in the slower times.



Because you cannot lose money.
That's pretty cool in my book.
jog on
d998

Hi Duc

Do feel free to expand and educate us on your views and share with us the benefit of your experience. We are all ears and eyes here.

Thanks in advance

Cheers
Happytrader

 

[stevo]

Duc
Arbitrage opportunities, as you have mentioned before, don't come up all that often. Do I assume that these opportunities form a minor part of the strategies that you use?

You keep mentioning expectancy and I am not sure why - it's just another calculation. What measures would you use to determine if a system has a reasonable chance of working going forward? How did you determine that arbitrage was a goer?

Monte Carlo analysis is useful to investigate the possible outcomes of the same system. A single backtest run doesn't really tell me much but a 1000 simulations start to tell me things. I also don't test on individual stocks only - I always portfolio test 100's of stocks.

I plot monthly returns, yearly returns, drawdown, as well as the equity curve. I can then compare actual results achieved over the years with results that have been simulated and see if these results fall within the expected outcomes.

I like graphical outcomes, and I am not talking about charts when I say this.
A picture like the one below tells me a lot. I ran 1000 Monte Carlo simulations on 3 systems - random buy and sell, standard weekly MACD and a system I have been working on - Rocket Science (RS).

Just looking at the graphs you can see that random averaged around 4% compound annual return, MACD around 11% and RS around 28%. Don't get too hung up on the absolute values - look at the differences between the system. Using different position sizing strategies can have dramatic effects on returns.

What is more interesting is the range of possible results is quite tight for RS - around 25% to 31%. Both the MACD and random had enough variation in the outcomes to make them untradeable - regardless of the low returns.

Obviously there is more to system testing than one graph - I can do the same sort of graph for drawdown - the random system has horrible drawdown, as does MACD.

You can do the same simulations with buy and hold strategies - some wonderful results and some incredibly bad drawdowns. But it's the variation of possible outcomes that kills buy and hold style strategies for me.

As I stated before if I have to do p-Values or t-tests to see the difference then the system is not worth trading. But an equity curves and quarterly returns bar chart gives me a hint of the future if I traded a system.

stevo

 

[tech/a]

Stevo what are you using for your tables? (software)

 

[stevo]

tech
AmiBroker to generate the Monte Carlo data and Excel to get the frequency histograms, although if I want I can use Amibroker and TradeSim as well.

stevo

 

[It's Snake Pliskin]

My apologies, I assumed that you knew what an arbitrage actually was.
It is a risk free trade taken between two securities of the same issuer, but priced to provide a spread.

:bs:

Duc,

I was talking about arbitrage not "an arbitrage".

Perhaps you could go beyond your definition and explain why it is used by entrepreneurs all the time. It is not limited to securities, and is certainly rendered inneffective once the window of opportunity has passed. Maybe a text book can be quoted or something.....

 

[Bobby]

:bs:

Duc,

I was talking about arbitrage not "an arbitrage".

Perhaps you could go beyond your definition and explain why it is used by entrepreneurs all the time. It is not limited to securities, and is certainly rendered inneffective once the window of opportunity has passed. Maybe a text book can be quoted or something.....

Go Snake,
I'm not the only one sick of this humdrum- jog on humbug, hope you had a good laugh ! :

Bob.

 

[ducati916]

stevo

Duc
Arbitrage opportunities, as you have mentioned before, don't come up all that often. Do I assume that these opportunities form a minor part of the strategies that you use?

No they don't.
I'll find five, six opportunities each day, but I will only get executed on average one deal per month. With enough leverage, the returns are well worth having for zero risk.

You keep mentioning expectancy and I am not sure why - it's just another calculation. What measures would you use to determine if a system has a reasonable chance of working going forward? How did you determine that arbitrage was a goer?

I mention, or harp on about expectancy for the following reason.
The inputs to the calculation are derivatives.
Being derivatives, their adequacy is governed by the quality of their generation. ****e in ****e out. Therefore, if the statistical significance is non-existent, the expectancy calculation is worthless.

The major flaw with *monte carlo* is that the time period is too short.
If your time period is long enough, then, you should start to see statistical significance, and the derived calculations will have relevance.

I spoke to one systems tester, and he didn't even bother with monte carlo.
Yet the same calculations were used.

Monte Carlo analysis is useful to investigate the possible outcomes of the same system. A single backtest run doesn't really tell me much but a 1000 simulations start to tell me things. I also don't test on individual stocks only - I always portfolio test 100's of stocks.

Agreed.
Monte carlo is definitely a step in the right direction.
I know that both you and tech/a utilize it.
TT only went (I believe from 1996) and that to my mind is just barely scraping a pass mark, if we say a *business cycle* is 4yrs to 7yrs.
It does not cover *market cycles* which are much more variable.
1996 to today in the ASX also does not include a secular bear market.
This I believe is important information that needs consideration within the testing parametres, if you wish to achieve true statistical significance, which will then provide you with a true expectancy or PROBABILITY.

As I stated before if I have to do p-Values or t-tests to see the difference then the system is not worth trading. But an equity curves and quarterly returns bar chart gives me a hint of the future if I traded a system.

It will only give you a hint of the future if it is statistically significant
If it is not, then you are defining a trend
A trend is a direct prediction of the future, and must prove itself either correct, or incorrect, as it will either continue, or end. This is true because the paradigm that you are currently testing in is DETERMINISTIC.

markrmau

That is a rather limited view on arbitrage. Arbitrage is not limited to mispricing between securities.

Of course, but you wouldn't expect a thesis on arbitrage in the space of one post, any more than a tecchie could summarize TA in a single post.

Snake

I was talking about arbitrage not "an arbitrage".

Perhaps you could go beyond your definition and explain why it is used by entrepreneurs all the time. It is not limited to securities, and is certainly rendered inneffective once the window of opportunity has passed. Maybe a text book can be quoted or something.....

If you can find a quote directly attributable to myself, asserting, entrepreneurs use it all the time then I shall do so.
Of course, that will never happen, as I never said any such thing.
I was illustrating a statistically significant methodology something that obviously holds no interest for yourself.

In regards to its limitation to securities, you are correct, there are a number of arbitrage techniques. The opportunity lasts as long as the price inefficiency, how long is a piece of string?

bob

Go Snake,
I'm not the only one sick of this humdrum- jog on humbug, hope you had a good laugh !

Bob.

Even the peanut gallery are jumping in. Obviously ignorance is bliss, and using a hammer for a screw insertion carries the appeal of speed, and seemingly identical results.

jog on
d998

 

[tech/a]

Duc

How do you determine a statistically significant amount of data?

Currently the discussion revolves around T/T and other longterm methods.
If you were a short term futures trader using 5 min tick data when is enough data enough data.

Fundamentally how long is enough fundamantal data?
How can you vouch for accuracy in the data presented in a balance sheet/s
and Profit and loss stataements? You cant tell me that everything presented fundamentally shows a complete picture,after all its presented to satisfy share holders and in the best possible light.

Again I'll ask the question--Why would you want a method to perform in an environment in which it is not desighned to perform well in?
Why would you trade a bullish method in a bear market?
Why would you trade a stock which has fundamentally great numbers when you knew that the market as a whole was going to be against it?
EG say a gold stock when Bank reserves are selling gold (as an example).

Whats stops you or anyone from suffering from analysis paralysis?
Yeh I know the 100% risk free mantra but in the real world while rarely possible not totally practical.


See Corporate and Social Responsibilities Thread on Reefcap.

 

[Milk Man]

Duc

How do you determine a statistically significant amount of data?

Currently the discussion revolves around T/T and other longterm methods.
If you were a short term futures trader using 5 min tick data when is enough data enough data.

Fundamentally how long is enough fundamantal data?
How can you vouch for accuracy in the data presented in a balance sheet/s
and Profit and loss stataements? You cant tell me that everything presented fundamentally shows a complete picture,after all its presented to satisfy share holders and in the best possible light.

Again I'll ask the question--Why would you want a method to perform in an environment in which it is not desighned to perform well in?
Why would you trade a bullish method in a bear market?
Why would you trade a stock which has fundamentally great numbers when you knew that the market as a whole was going to be against it?
EG say a gold stock when Bank reserves are selling gold (as an example).

Whats stops you or anyone from suffering from analysis paralysis?
Yeh I know the 100% risk free mantra but in the real world while rarely possible not totally practical.

Reminds me of that post you stated was "the best ive seen on any forum". Basically says only to trade long in bulls and be careful in sideways, but either stay out or go short in bears. Some form of portfolio stop could be utilized but it may prove to be "statistically insignificant"; just like my bank a/c. esok:

 

[tech/a]

Milk.

This is a tact I am currently looking into.But more on that later.

 

[stevo]

Duc

TT only went (I believe from 1996) and that to my mind is just barely scraping a pass mark, if we say a *business cycle* is 4yrs to 7yrs.
It does not cover *market cycles* which are much more variable.

10 years is actually pretty good, but obviously doesn't include all past market activity. If you test 500 stocks over 10 years you get roughly 1.25 million trading days worth of data. But the more the better. I personally don't trust data back more than about ten years, and even then it is getting a little hairy. It's better not to test than to test on bad data.

I like to think of all the data for all the stocks joined end to end as a single stock - 10 years of data on 500 stocks is the equivalent of 5000 years worth of data for one stock. I am sure that you will point out the error in this thought but it does highlight the value of portfolio testing versus single stock back-testing for a given set of criteria.

I have been looking for price data out to 2020 but nobody seems to have it! If anyone knows a supplier I would be prepared to pay substantial sums for it - although not my soul.

stevo

 

[ducati916]

tech/a

How do you determine a statistically significant amount of data?

From pg2

Sample Size
One crucial prerequisite prior to embarking upon the *testing* or study is the requirement to perform a sample size, or POWER CALCULATION In the words of D. Altman "a trial should be big enough to have a high chance of detecting, as a statistically significant, a worthwhile effect if it exists, and thus to be reasonably sure that no benefit exists if it is not found in the trial."

As regards duration, common sense is enough of a guide to determine the appropriate time period to be used

TT was designed as a long term methodology, therefore long term duration of data should be utilized.

Intrestingly, you may if you subscribe to the fractal theory of financial markets utilize 1mins charts for your long term TT methodology, and you would starting from 1996 - 2005 have far more data available.

As for daytrading who knows, who cares.
Obviously daytraders.
I would suspect in actuality they need the same amount of data as TT.
What you are looking for is statistical significance, the law of large numbers rules within statistical methodologies.

Fundamentally how long is enough fundamantal data?

13 weeks, or Q1.

How can you vouch for accuracy in the data presented in a balance sheet/s
and Profit and loss stataements? You cant tell me that everything presented fundamentally shows a complete picture,after all its presented to satisfy share holders and in the best possible light.

They would never seek to mislead, or lie to me.
The data is 100% reliable.

Again I'll ask the question--Why would you want a method to perform in an environment in which it is not desighned to perform well in?
Why would you trade a bullish method in a bear market?

Statistically significant methodologies are robust methodologies.
Therefore, if it performs in all markets, under all conditions, even adverse ones, this is a methodology I can back with cash.

Why would you trade a stock which has fundamentally great numbers when you knew that the market as a whole was going to be against it?

Because that is when your risk is at its lowest, and your potential returns are at their highest.

Whats stops you or anyone from suffering from analysis paralysis?

An interesting question.
First and foremost, there must be a differentiation of quantitative methods from qualitative methods. The quantitative methods must be consistent, comparable, show causation, correlation and be based or founded in the simplest calculations possible. The qualitative methods must support the quantitative on a common sense basis, and again be based on simple logical arguments, that are easily understood by the layman.

Increasing use of esoteric methodologies, higher math, quantum math, elaborate paradigms, etc will almost certainly lead to a mental meltdown, and analysis paralysis, or extreme emotional bias.

stevo

10 years is actually pretty good, but obviously doesn't include all past market activity. If you test 500 stocks over 10 years you get roughly 1.25 million trading days worth of data. But the more the better. I personally don't trust data back more than about ten years, and even then it is getting a little hairy. It's better not to test than to test on bad data.

Agreed.
If the data is corrupt, the output must by definition be corrupt, and worthless

I like to think of all the data for all the stocks joined end to end as a single stock - 10 years of data on 500 stocks is the equivalent of 5000 years worth of data for one stock. I am sure that you will point out the error in this thought but it does highlight the value of portfolio testing versus single stock back-testing for a given set of criteria.

This would be fine, if, all the variables within individual stocks were consistent. That they are not, must therefore invalidate this convenient theory.

I have been looking for price data out to 2020 but nobody seems to have it! If anyone knows a supplier I would be prepared to pay substantial sums for it - although not my soul.

I of course have it.
But if I send it, you gotta keep it under your hat.

jog on
d998

 

[It's Snake Pliskin]

If you can find a quote directly attributable to myself, asserting, entrepreneurs use it all the time then I shall do so.
Of course, that will never happen, as I never said any such thing.
I was illustrating a statistically significant methodology something that obviously holds no interest for yourself.

In regards to its limitation to securities, you are correct, there are a number of arbitrage techniques. The opportunity lasts as long as the price inefficiency, how long is a piece of string?

It was a rhetorical question, one you were obviously not able to detect.:screwy:

Good luck with your grail and watch that window of opportunity!

 

[tech/a]

The quantitative methods must be consistent, comparable, show causation, correlation and be based or founded in the simplest calculations possible. The qualitative methods must support the quantitative on a common sense basis, and again be based on simple logical arguments, that are easily understood by the layman.

While open to arguement,I think thats is exactly what and where the likes of myself and Stevo are at.

 

[stevo]

What you are looking for is statistical significance, the law of large numbers rules within statistical methodologies.

Continuing down the path of disaggregating data (love that word - disaggregating) more robust systems would be developed using daily data over agreggated weekly data. In practice this does not seem to hold up, although I am sure many would disagree with me.

The concept that the longer term the system the more data required has steered me away from designing systems based on a monthly or quarterly time frames. The large number of trades in a backtest can give some comfort, but it is possible to generate more trades without increasing statistical significance.

stevo