Dump it Here

[ducati916]

No my general point is not that more data is relevant--like I said, a statistically relevant amount of data is what is need. They are two very different things and should not be confused. It may seem like one and the same, but trust me in the world of statistics they are chalk and cheese.

Statistical confidence increases with the number of observations of the variable under observation, which is statistically relevant data, to which I am referring. I am not advocating adding in data re. hemlines etc.

So let us get a little less abstract and more specific.

Take Mr Skate's various systems: these have been designed with the 'exit' as the (or one of the prime) variables. The premise being: money is made on the exit.

Now, return to last year when the markets crashed. Did the exits save Mr Skate? To a point, yes, he still exited the market holding profits. But essentially the profits were decimated.

How did other mechanical systems hold up last year? They didn't.

Now they were backtested, and all gave the illusion of having a max drawdown of 'X'. They were all wrong. They were all wrong because of the very issue highlighted.

If, they had 'recognised' the signs as they were building, they would have exited much earlier. That is the value of studying, backtesting, whatever, the outliers that we have had far more frequently than the relevant statistical data has forecast in terms of probabilities.

jog on
duc

 

[qldfrog]

Your general point is that more data is better than less data. I am saying that that is simply not the case. We agree that specific samples of data are required.

jog on
duc

MA has the view that all data is the equal so that :
what was true a generation ago is still true.
I would say
Yes and no, true as to human psyche, false as to its implementation and so day to day influence on the market
This is where we differ, and as a consequence i am in the view that "too old" zillion data testing is useless.
But yes if you have relevant data, the more the better.we all agree

 

[qldfrog]

Statistical confidence increases with the number of observations of the variable under observation, which is statistically relevant data, to which I am referring. I am not advocating adding in data re. hemlines etc.

So let us get a little less abstract and more specific.

Take Mr Skate's various systems: these have been designed with the 'exit' as the (or one of the prime) variables. The premise being: money is made on the exit.

Now, return to last year when the markets crashed. Did the exits save Mr Skate? To a point, yes, he still exited the market holding profits. But essentially the profits were decimated.

How did other mechanical systems hold up last year? They didn't.

Now they were backtested, and all gave the illusion of having a max drawdown of 'X'. They were all wrong. They were all wrong because of the very issue highlighted.

If, they had 'recognised' the signs as they were building, they would have exited much earlier. That is the value of studying, backtesting, whatever, the outliers that we have had far more frequently than the relevant statistical data has forecast in terms of probabilities.

jog on
duc

We all remember the black swan zillion of lifetime probability not to happen..which happened in the last crash.quite interesting article at the time so always remember stats can tell you whatever you want them to say ..based on the data selection...

 

[MovingAverage]

Now they were backtested, and all gave the illusion of having a max drawdown of 'X'. They were all wrong. They were all wrong because of the very issue highlighted.

I am glad you have highlighted this, but it illustrates one of the major issues I have with a lot of people that trade mechanical systems and not the systems per se. I say I have a major issue with the people deliberately but a lot of people trading mechanical systems do not have sufficient understanding of how to properly analysis system that deal with time series discrete data.

Most people will do a single backtest across 20 years of data and to further add to the system they will over optimize (or curve fit). They get a fantastic looking system that averages a CAGR of 35% and a drawdown of 20%. Guess what, then they jump straight into live trading that system and as you allude to they encounter a tough market and experience a 35% drawdown and maybe after a few years they're at a CAGR of 8%.

Well based on your comment I'm guessing you would blame the system--but I don't because that person didn't comprehensively test that system to full understand the systems bounds and perhaps that 8% CAGR and 35% DD was well within the systems behavior. Anyone that runs a single simulation of 20 years of data and then goes live is seriously asking for trouble but I don't blame the system.

If you do proper MC analysis, and I'm not talking about that crappy MC testing in Amibroker, but proper statistical analysis you will have a far more insightful understanding of what to expect from your system. The reality is that a system will have a range within which key parameters will fall if correctly tested. Proper MC analysis (not Amibroker MC rubbish) will yield a distribution (or range) within which you can expect your system to perform. For example, proper statistical analysis will reveal that the DD will have a mean of 18% with a certain SD from the mean at 95% confidence interval. This applies to any system parameter you wish to track. In other words, proper statistical analysis will give you a mean and what you can expect in terms of extremes from that mean.

The other thing that a lot of people don't do is out of sample testing. It's one thing to refine and tune your system over certain data, but you better test that on data that you didn't test and refine your system on. In other words, this will give you some insight in your systems ability to perform on future data. This is one of the biggest oversights a lot of system traders make.

Anyway, the point I'm making is that in the scenario you provide--I don't blame the system I blame the person for not having engaged in proper MC testing of their system.

 

[MovingAverage]

MA has the view that all data is the equal so that :
what was true a generation ago is still true.
I would say
Yes and no, true as to human psyche, false as to its implementation and so day to day influence on the market
This is where we differ, and as a consequence i am in the view that "too old" zillion data testing is useless.
But yes if you have relevant data, the more the better.we all agree

I'm kind of saying that. What I saying is that if you test and refine your system over a period 1990 to 1995 and your then perform out of sample testing (not optimizing of system parameters) over the discrete period of 1996-2000, 2001-2005, 2006-2010, 2011-2015, 2016-2020 and all sim results across those years yield acceptable results then I would say your system appears to work well on future data. That is why a say old data (in this case 1990 to 1995) is not irrelevant. Maybe it isn't always the case but to discount old data outright just because times were different might be doing you a disservice.

 

[qldfrog]

I'm kind of saying that. What I saying is that if you test and refine your system over a period 1990 to 1995 and your then perform out of sample testing (not optimizing of system parameters) over the discrete period of 1996-2000, 2001-2005, 2006-2010, 2011-2015, 2016-2020 and all sim results across those years yield acceptable results then I would say you system appear to work well of future data. That is why a say old data (in this case 1990 to 1995) is not irrelevant. Maybe it isn't always the case but to discount old data outright just because times were different might be doing you a disservice.

I will have to go but.
If you do so, can I genuinely ask: do you use price as a realm selection? for example to filter out 2c stocks etc..?
If you do then how do you reconcile the fact a 2c worth in 1992 is probably a 5c worth or more today?
Sure possible to use a cpi indexed price as a tgreshold but outside the beginner coding usage.
So i just want beginners to think about what their code backtest is supposedly testing with 30y old data.
Nice discussion.i agree we disagree?

 

[MovingAverage]

I will have to go but.
If you do so, can I genuinely ask: do you use price as a realm selection? for example to filter out 2c stocks etc..?
If you do then how do you reconcile the fact a 2c worth in 1992 is probably a 5c worth or more today?
Sure possible to use a cpi indexed price as a tgreshold but outside the beginner coding usage.
So i just want beginners to think about what their code backtest is supposedly testing with 30y old data.
Nice discussion.i agree we disagree?

I like having a respectful disagreement...after all if everyone traded the same we'd have no market to trade .

I live trade a number of systems and no I don't apply a price filter directly, but I see your point. I do apply a turnover filter (must have min $ turnover) which is derived from price and volume, but it isn't a fixed value and it will vary depending on the trade size. I haven't thought it through too much but I guess your point might be relevant to me and I'll take it onboard and think it through.

 

[Lone Wolf]

The other thing that a lot of people don't do is out of sample testing. It's one thing to refine and tune your system over certain data, but you better test that on data that you didn't test and refine your system on. In other words, this will give you some insight in your systems ability to perform on future data. This is one of the biggest oversights a lot of system traders make.

One trap about out-of-sample testing that's easy to fall into is repeatedly using the same data sets. You test your system on in-sample data, looks great. You test it on out-of-sample data, it falls apart. What do you do? You go back to the system and make some adjustments. Test it again on the same data and it looks much better this time. But really, you've just made your out-of-sample data part of the in-sample data set. You can't keep making adjustments until the same out-of-sample test comes good.

 

[Quanta]

Hey Guys,

Quick question MovingAverage, when you state:

If you do proper MC analysis, and I'm not talking about that crappy MC testing in Amibroker, but proper statistical analysis you will have a far more insightful understanding of what to expect from your system.

How do you go about this? Do you use Amibroker and import your results into Excel for analysis?

I'm on the learning curve, most of the stuff discussed to test robustness I do as it just made sense, next step is expanding data analysis.

Thanks in advance.

 

[ducati916]

1. I am glad you have highlighted this, but it illustrates one of the major issues I have with a lot of people that trade mechanical systems and not the systems per se. I say I have a major issue with the people deliberately but a lot of people trading mechanical systems do not have sufficient understanding of how to properly analysis system that deal with time series discrete data.

2. Most people will do a single backtest across 20 years of data and to further add to the system they will over optimize (or curve fit). They get a fantastic looking system that averages a CAGR of 35% and a drawdown of 20%. Guess what, then they jump straight into live trading that system and as you allude to they encounter a tough market and experience a 35% drawdown and maybe after a few years they're at a CAGR of 8%.

3. Well based on your comment I'm guessing you would blame the system--but I don't because that person didn't comprehensively test that system to full understand the systems bounds and perhaps that 8% CAGR and 35% DD was well within the systems behavior. Anyone that runs a single simulation of 20 years of data and then goes live is seriously asking for trouble but I don't blame the system.

4. If you do proper MC analysis, and I'm not talking about that crappy MC testing in Amibroker, but proper statistical analysis you will have a far more insightful understanding of what to expect from your system. The reality is that a system will have a range within which key parameters will fall if correctly tested. Proper MC analysis (not Amibroker MC rubbish) will yield a distribution (or range) within which you can expect your system to perform. For example, proper statistical analysis will reveal that the DD will have a mean of 18% with a certain SD from the mean at 95% confidence interval. This applies to any system parameter you wish to track. In other words, proper statistical analysis will give you a mean and what you can expect in terms of extremes from that mean.

5. The other thing that a lot of people don't do is out of sample testing. It's one thing to refine and tune your system over certain data, but you better test that on data that you didn't test and refine your system on. In other words, this will give you some insight in your systems ability to perform on future data. This is one of the biggest oversights a lot of system traders make.

6. Anyway, the point I'm making is that in the scenario you provide--I don't blame the system I blame the person for not having engaged in proper MC testing of their system.

1. The issue that I have with mechanical systems and I suppose by extension the people that design them is that they, for the most part, seem to be blind to the intrinsic flaw that lies at the heart of mechanical based trading.

2. A mechanical system is an automated strategy. Certain strategies will do well in certain market environments and potentially outperform their initial code. Markets change. Sometimes that change is subtle and the system still returns a profit, sometimes it is radically different and a period of underperformance will ensue. Sometimes the monkey wrench gets tossed in and s**t blows up. Now you could have multiple systems (strategies) and chop and change them as market conditions change. That however is 'discretionary', which (possibly) systems traders decry?

3. Being the polite chap that I am, yes, it is the 'system's' fault.

4. Well good. This is the topic that came up a few posts back and (seemingly) was resoundingly ignored. This is something worthy of a detailed discussion. The results, if properly done, will toss many systems currently being traded on the scrap heap.

5. This is where the MC is critical. The data that we have, even if you had data from 1880 to the current day, is not sufficient. Now that statement may seem to contradict my position. It doesn't. This point is raised by @Lone Wolf in his post.

6. And this should be rectified.

* I see another post, just above is curious.

jog on
duc

 

[MovingAverage]

Hey Guys,

Quick question MovingAverage, when you state:


How do you go about this? Do you use Amibroker and import your results into Excel for analysis?

I'm on the learning curve, most of the stuff discussed to test robustness I do as it just made sense, next step is expanding data analysis.

Thanks in advance.

Hey Quanta,
I use two applications depending on what I want to do. I use TradeSim and/or Excel. With both applications I dump sim outputs from Amibroker into CSV files and then just import into TradeSim or Excel. You really can do some great statistical analysis with Excel. TradeSim I really like because it is specific to stocks and you can drill down into more trade related stuff than Excel. But you can do a whole lot with Excel and enough analysis to decide if you want to go live or not. Only downside to TradeSim (and it isn't major) is that it expects the CSV data to be in a particular format--they openly publish the CSV input format. So for TradeSim it is not a simple matter of exporting from Amibroker (which you can do for Excel). In the AFL code of your system you have to write the CSV file in the format required for TradeSim. If you're comfortable with AFL then this should NOT (thanks frog) be a big deal.
Good luck.
MA

 

[qldfrog]

If you're comfortable with AFL then this should be a big deal.

This should NOT be a big deal .
This exchange has been quite interesting.Thanks Mr Skate to allow us to highjack your thread ?

One last point from me: this is not black and white: of course you need sufficient data to be statistically relevant, of course you need out of sample data trials to be able to judge possible responses inc DD but yes markets change and old data probably looses relevance if time affected: cpi, new robot trading,etc.
Overall, it is still better to work with these flaws than just randomly throwing money at shares and pray!
And this is what most beginners do, i did too ...with either success in boom time that they parade around and become so called experts at the BBQ events, or big losses and become investment property addicts for the rest of their life?

 

[MovingAverage]

This should NOT be a big deal .

Fixed

 

[MovingAverage]

6. And this should be rectified.



jog on
duc

I often think that a lot of mechanical system traders would benefit hugely from a course in statistical analysis--even a basic one. Anyhow, as you say...Jog On

 

[ducati916]

A mechanical system is an automated strategy. Certain strategies will do well in certain market environments and potentially outperform their initial code. Markets change. Sometimes that change is subtle and the system still returns a profit, sometimes it is radically different and a period of underperformance will ensue. Sometimes the monkey wrench gets tossed in and s**t blows up. Now you could have multiple systems (strategies) and chop and change them as market conditions change. That however is 'discretionary', which (possibly) systems traders decry?

No one seems to have picked up on this point, maybe, irrelevant to mechanical systems traders.

So over the past few months, the market changed. Interest rates became a thing and volatility, less of a thing. Those that follow my other thread will probably have noticed, less on the VIX more on TNX and DXY. Tech. in pretty much the same time frame went from hot to meh to shite.

Now I don't analyse the stocks being selected in Mr Skate's systems, mostly because they are Aussie stocks. However I do know that @Skate employs a 'ranking' methodology. Now (again) I'm not sure how they rank stocks, but it should rank them based on sectors and stocks within those sectors. For example: rising curve ++financials - - Tech. Stocks that fall in the financial sector get a higher ranking than Tech. There are all manner of subtle variations.

The reason is that the filter may grab a Tech stock that has some good news attached, whatever: once that 'good news' plays out over a day or two, that stock will revert to its sector. It will not continue to outperform, whereas, even average/shite financials will run. Possibly the systems selection strategy might cover this, but I doubt it. Possibly Mr Skate can expand on this criteria and whether the scans are or could be (made) to be discriminating enough.

jog on
duc

 

[Skate]

The issue that I have with mechanical systems and I suppose by extension the people that design them is that they, for the most part, seem to be blind to the intrinsic flaw that lies at the heart of mechanical based trading. Certain strategies will do well in certain market environments and potentially outperform their initial code.

I'm amazed
The recent flurry of quality post prosecuting each side of the argument is astounding. Read individually they all have a certain degree of merit - but @ducati916 has a valid point about mechanical-based trading. Trading a system that's ridged (inflexible) on a constantly changing market.

Possibly Mr Skate can expand on this criteria and whether the scans are or could be (made) to be discriminating enough.

Great point
And by the why - another thought-provoking post. The series of recent posts go to the heart of the question.

"Why do we buy a position trading a mechanical system?".

Summary
Over a few posts (so I can chunk the answer) I'll try to answer the question.

Skate.

 

[Skate]

Backgrounding first - Falling Interest rates
At times circumstances shape your future & the falling interest rates shaped my financial future.

I'm a 6% guy
Why 6%? - 6% returns mean I'll never run out of money in my life. Inflation is included in that calculations (inflationary forces reduces true value)

Living in a cranky & unstable world
There are no free rides in this world & financial freedom is never stable. The reason we work is to survive & create a life that one can be happy with.

Trading isn’t about getting rich
I've posted this many times, it's more about financial independence supporting yourself without an income, being able to choose to live your life on your terms.

Who decides?
The only person who can make sure you’re able to do it on your terms is you - this is the true measure of what life is all about as far as I'm concerned.

Skate.

 

[barney]

No particular input from me. @ducati916 and @Skate

You guys are very clever. I understand the premise of your Trading ideas/systems etc etc

I'm personally either too old or too lazy to implement something similar, however,

I certainly enjoy (and obviously many others also) appreciate the detail you chaps present, so thanks for that.

I often wish I had the aptitude to be more regimented like you both,

But, I am a simple musician who plays by ear, so that will never happen


ps. I'm doing ok being a lazy musician so all is good


pps Carry on as you were gentlemen. The Dump Thread will likely live on far longer than all of us. Well done Mr Skate

 

[Skate]

Back to trading
Like most "trading became an obsession" after a period of time. Returns from trading started to become the scorecard. The measures of personal development as the results provided the motivation to keep going & the motivation to improve.

Mechanical Trading
The years of study, Backtesting ideas (running 5 computers - 24/7) - exporting those statistical results to a Monte Carlo simulation was a worthwhile exercise giving a raw indication if the trading idea had the potential (probability) of profitability. In hindsight, the effort developed into rewards (if only by the luck of timing). What was significant - the confidence that the results gave at the time.

Sample of trades
This is one of my trading systems - it's a mature system traded over many years. At face value, a win rate of 40% looks a bit low but within the range of a Mechanical Trend Trading Strategy. Also, my Equity curve would certainly highlight the views that @ducati916 recently made about the fluidity of markets & the changing sectors within that market. Markets are fluid & change with every heartbeat. The jaggedness of the curve would bear this out.



The ocean
Waves are forming every minute of every day, some days the waves are larger than others (that's a given) but if I was a surfer (which I'm not) I would pick my day to enjoy the pastime. Trading to me is no different.

Skate.

 

[MovingAverage]

Back to trading
Like most "trading became an obsession" after a period of time. Returns from trading started to become the scorecard. The measures of personal development as the results provided the motivation to keep going & the motivation to improve.

This is a very interesting topic and one that I'm sure has the potential to generate another flurry of posts. "Returns from trading started to become the scorecard". Yes, returns is an important measure, but for me it is a secondary important measure and doesn't top my scorecard measure. For me personally my priority is to chase a nice tight standard deviation in returns. Why, it delivers a better level of predictability and lessens surprises. Yes, a tight standard deviation generally comes at the cost of a higher return, but I also don't like the wild ride that comes with surprises.