New posts

Reply to thread

Message

I am glad you have highlighted this, but it illustrates one of the major issues I have with a lot of people that trade mechanical systems and not the systems per se. I say I have a major issue with the people deliberately but a lot of people trading mechanical systems do not have sufficient understanding of how to properly analysis system that deal with time series discrete data.

Most people will do a single backtest across 20 years of data and to further add to the system they will over optimize (or curve fit). They get a fantastic looking system that averages a CAGR of 35% and a drawdown of 20%. Guess what, then they jump straight into live trading that system and as you allude to they encounter a tough market and experience a 35% drawdown and maybe after a few years they're at a CAGR of 8%.

Well based on your comment I'm guessing you would blame the system--but I don't because that person didn't comprehensively test that system to full understand the systems bounds and perhaps that 8% CAGR and 35% DD was well within the systems behavior. Anyone that runs a single simulation of 20 years of data and then goes live is seriously asking for trouble but I don't blame the system.

If you do proper MC analysis, and I'm not talking about that crappy MC testing in Amibroker, but proper statistical analysis you will have a far more insightful understanding of what to expect from your system. The reality is that a system will have a range within which key parameters will fall if correctly tested. Proper MC analysis (not Amibroker MC rubbish) will yield a distribution (or range) within which you can expect your system to perform. For example, proper statistical analysis will reveal that the DD will have a mean of 18% with a certain SD from the mean at 95% confidence interval. This applies to any system parameter you wish to track. In other words, proper statistical analysis will give you a mean and what you can expect in terms of extremes from that mean.

The other thing that a lot of people don't do is out of sample testing. It's one thing to refine and tune your system over certain data, but you better test that on data that you didn't test and refine your system on. In other words, this will give you some insight in your systems ability to perform on future data. This is one of the biggest oversights a lot of system traders make.

Anyway, the point I'm making is that in the scenario you provide--I don't blame the system I blame the person for not having engaged in proper MC testing of their system.

<blockquote data-quote="MovingAverage" data-source="post: 1114857" data-attributes="member: 33203">I am glad you have highlighted this, but it illustrates one of the major issues I have with a lot of people that trade mechanical systems and not the systems per se. I say I have a major issue with the people deliberately but a lot of people trading mechanical systems do not have sufficient understanding of how to properly analysis system that deal with time series discrete data.Most people will do a single backtest across 20 years of data and to further add to the system they will over optimize (or curve fit). They get a fantastic looking system that averages a CAGR of 35% and a drawdown of 20%. Guess what, then they jump straight into live trading that system and as you allude to they encounter a tough market and experience a 35% drawdown and maybe after a few years they&#039;re at a CAGR of 8%.Well based on your comment I&#039;m guessing you would blame the system--but I don&#039;t because that person didn&#039;t comprehensively test that system to full understand the systems bounds and perhaps that 8% CAGR and 35% DD was well within the systems behavior. Anyone that runs a single simulation of 20 years of data and then goes live is seriously asking for trouble but I don&#039;t blame the system.If you do proper MC analysis, and I&#039;m not talking about that crappy MC testing in Amibroker, but proper statistical analysis you will have a far more insightful understanding of what to expect from your system. The reality is that a system will have a range within which key parameters will fall if correctly tested. Proper MC analysis (not Amibroker MC rubbish) will yield a distribution (or range) within which you can expect your system to perform. For example, proper statistical analysis will reveal that the DD will have a mean of 18% with a certain SD from the mean at 95% confidence interval. This applies to any system parameter you wish to track. In other words, proper statistical analysis will give you a mean and what you can expect in terms of extremes from that mean.The other thing that a lot of people don&#039;t do is out of sample testing. It&#039;s one thing to refine and tune your system over certain data, but you better test that on data that you didn&#039;t test and refine your system on. In other words, this will give you some insight in your systems ability to perform on future data. This is one of the biggest oversights a lot of system traders make.Anyway, the point I&#039;m making is that in the scenario you provide--I don&#039;t blame the system I blame the person for not having engaged in proper MC testing of their system.</blockquote>

[QUOTE="MovingAverage, post: 1114857, member: 33203"] I am glad you have highlighted this, but it illustrates one of the major issues I have with a lot of people that trade mechanical systems and not the systems per se. I say I have a major issue with the people deliberately but a lot of people trading mechanical systems do not have sufficient understanding of how to properly analysis system that deal with time series discrete data. Most people will do a single backtest across 20 years of data and to further add to the system they will over optimize (or curve fit). They get a fantastic looking system that averages a CAGR of 35% and a drawdown of 20%. Guess what, then they jump straight into live trading that system and as you allude to they encounter a tough market and experience a 35% drawdown and maybe after a few years they're at a CAGR of 8%. Well based on your comment I'm guessing you would blame the system--but I don't because that person didn't comprehensively test that system to full understand the systems bounds and perhaps that 8% CAGR and 35% DD was well within the systems behavior. Anyone that runs a single simulation of 20 years of data and then goes live is seriously asking for trouble but I don't blame the system. If you do proper MC analysis, and I'm not talking about that crappy MC testing in Amibroker, but proper statistical analysis you will have a far more insightful understanding of what to expect from your system. The reality is that a system will have a range within which key parameters will fall if correctly tested. Proper MC analysis (not Amibroker MC rubbish) will yield a distribution (or range) within which you can expect your system to perform. For example, proper statistical analysis will reveal that the DD will have a mean of 18% with a certain SD from the mean at 95% confidence interval. This applies to any system parameter you wish to track. In other words, proper statistical analysis will give you a mean and what you can expect in terms of extremes from that mean. The other thing that a lot of people don't do is out of sample testing. It's one thing to refine and tune your system over certain data, but you better test that on data that you didn't test and refine your system on. In other words, this will give you some insight in your systems ability to perform on future data. This is one of the biggest oversights a lot of system traders make. Anyway, the point I'm making is that in the scenario you provide--I don't blame the system I blame the person for not having engaged in proper MC testing of their system. [/QUOTE]

Verification