Amibroker Beginnings

Just brought ab on the weekend,
This thread and the faq thread is golden!

Keep up the good work every1!

J

Interesting little thread... I wonder how many others like this are out there
(I will keep searching...)

howardbandy said:

Note the number of consecutive losers -- 28. The ratio of winning trades to losing trades is 105 to 439 -- 80.6% losers. Personally, if I saw a system go 27 trades in a row with a loss each time, I could not place the 28th or 29th trade. In fact, the probability of 14 consecutive losing trades is 0.0488. When the 14th trade in a row lost (and the probability of that happening is <0.05), the system is telling you that it is broken and should not be traded any more.

Thanks for listening,
Howard

Click to expand...

Hi Howard,

I have to disagree with this. I understand your qualifications, your background and where you are coming from, i even have your book, but i do not believe that shutting off a system because an event with a probability of less than 5% is correct.

Let's make a simple example here, a system with a reasonbly high amount of trades per year, lets say 150, and a win rate of 50%. The chance of 1 losing trade is 50%, the chance of two in a row is 25%, and so on. Here are the figures using the formula (WinRate)^Streak.


So at the 5% level, we are shutting the system off when i has 5 consecutive losing trades in a row. This is for one test, which is far too shallow in my opinion. What you need to do is calculate the chance of hitting this streak over a certain number of repetitions. As the system trades 150 times a year, we have 146 possible trades that can start a streak of 5 losing trades in a row. Using an formula that i have adapted, we can calculate the probability of hitting a streak of 5 losing trades in a row having the opportunity to start this streak 146 times.

The formula i will use is simple, (1-(1-(WinRate^Streak))^Repetitions.


This means, by my calculations, that there is a 95% chance that we will have a streak of 5 losing trades in a row, if we do 150 trades in a year. If we run this system in it's current state, there is a just a 4.9% chance that we'll still be using this system at the end year, even though the "abnomily" that causes us to shut the system off, is just an expected result because of the nature of probability over a large number of repetitions. Even shutting it off with a score of <.01, there is more than a 50% chance of shutting the system down.

Shutting it down with a score of <.05 will only work if you are doing 5 trades, if you are doing 6 trades, you'll have 2 chances to start the same streak, meaning that what was once a 5% probability of occuring, is now a more likea 6-7% chance of occuring. The more repetitions we do, the more chance we have of shutting off a perfectly good system.

Regards
Brad

Hi Brad --

You can pick whatever level that you want to consider as "the critical level" that you want to. 5% is typical.

The critical level will depend on the out-of-sample performance you observed during the out-of-sample validation runs. If your system has a high percentage of winning trades, it will take fewer consecutive losing trades to indicate that it is broken. If your system has a low percentage of winning trades, then a string of losing trades will not be as unusual and it will take a longer string of them to signal trouble.

In all situations of this kind, you are trading off two kinds of statistical error -- type I and type II. Ask yourself -- Do you want to risk taking trades using a system that is broken (or at least out of synchronization) or risk not taking trades with a system that works but is in a rare event? The higher the critical level is, say 0.10, the more likely the system will signal trouble when it is not broken. The lower that critical level is, say 0.01, the more likely real money will be lost trading a broken system until the statistic falls enough to trigger the "put this system on hold".

My advice is to risk the lost opportunity of staying flat rather the loss of real money.

My recommendation is that you do not take trades after the system has produced an event -- a string of losing trades -- that is expected to happen less than 5% (or pick your own number here) of the time. Watch the next trade or two or three. If the system turns in some winning trades, continue trading it. If it shows more losing trades, keep it on hold or revalidate it.

My point is: At some point you will eventually stop trading a system that continues to produce losing trades. Decide ahead of time what that level will be.

Thanks for listening,
Howard

Hi Howard.

My point wasn't that 5% score is not the correct level to turn off a system, but over a large sample, we are nearly certain to hit that 5% score. This is because we are repeating the same test over and over, so eventually we are going to hit a level that we find to be an abnomily.

If the system produces 1 set of trades a year, (1 set being the amount of trades needed to hit our critical level), then shutting it off at 5% will suffice.

Just to make an example easy, lets imagine a system has 50% win rate and trades 100 times a year. For your P<0.05, you will need 5 losing trades in a row (0.5)^5 = 3.13%. Now here is the fun part Go and grab a coin and flip it 100 times, heads will represent a winning trade, and tails will represent a losing trade. I can tell you with 95% certainty that you will have a streak of 5 flips that are tails in a row.

Now i ask you this, is the coin broken? Is the coin not producing a 50/50 result? The coins bias remains unchanged, what we have seen is simply what was once thought to be an unlikely event occur, because we have repeated the task 100 times.

Therefore i'd suggest that we don't use 5% as a level for turning off a system, any system that has any sort of regular trade frequency can almost be guaranteed to be shut off, even if it's not broken, see the coin example. I'd suggest that we turn off the system if we produce 11 losing trades in a row, at a 50% win rate. To me, this would indicate that the chance of this happening is small enough to deem to system unfit. How did i come to this conclusion? The chance of 11 losing trades in a row, starting now, is just 0.05%. But if we repeat this over 100 trades, the chance of this occuring jumps up to 4.25%, which satisfies our P<0.05 level of confidence. I believe shutting the system off after 11 trades gives it enough chance to prove itself unbroken, until the time that we find that there is a less than 5% chance of this occuring over the whole sample, not just a single sample of a streak occuring from 1 trade to 11 trades, without repetition.

Regards
Brad

Hi Brad --

I am probably belaboring the point, but let me try again.

I begin with the assumption that we will have done a proper validation of our trading system, and in that process gathered statistics about the performance of the system. Those include the percentage of trades that are profitable.

As we trade the system with real money, we can observe the sequence of trades, including whether they are profitable or not.

There are other statistical methods that will help determine the health of a trading system. This is one example, but they all operate on the same principle. That is to look for results that are both unlikely and undesirable, and compare those with observed results where critical values were determined through statistical analysis of the out-of-sample results.

One statistic that might help us determine the health of the system as it is being traded is the sequence of losing trades. When a series of losing trades that should occur very rarely does occur, we can take that as an indication that the model is out of synchronization with the data series -- that is, the system is broken. We can stop taking trades with real money until one of two things happens:
1. We go through the system validation process again, selecting new values for the logic and variables. This re-syncs the system. Since there was a period of poor performance, I recommend that we watch the new system (paper trade it) to build confidence that the system will be profitable.
2. We make no changes to the system until the next scheduled revalidation point. Just wait and watch and paper trade. When the system shows enough profitable trades to bring the statistics back in line, we start making trades with real money again.

Extending my point to the limit ----
If I have a system that I expect to produce a string of, say, 10 losing trades less than 5% of the time, I should be concerned. I could continue to trade it. Somewhere between observing the string of 10 losing trades and 1000 losing trades, I will stop trading it. Is my stopping point 10? or 20? Or do I wait for the account to go bankrupt?

I pick some statistic that is unlikely, but not impossible. 5% works for me. You can pick whatever number you want to. But, and this is my point, whatever number I pick gives me a warning that my system may be broken. How I deal with the system from that point on puts the system back into the system development and validation phase.

Thanks for listening,
Howard

No problems howard, thanks for your time.
I look forward to buying your next book later this year.

My point was that, something with a score of less than 5% is bound to occur, whether the situation is out of synchronisation or not, if we repeat the test enough times. Im sure that you can agree that 5% is quite a high number if we are talking a sample say, the next 1000 trades. It's almost certain to happen.

The dilemma is, what you have rightly pointed out, do we stop at P=5%, or continue to risk our money waiting for the system to be proven more "wrong"? From my studies of probability, i'd lean towards the latter, a point where we are wrong for the sample size, not a single chosen string of a sample.

It's a really tough question as there is never a right or wrong answer, it's not black and white, but personally i feel that P=5% is still quite a high probability event. As i have said before, 5 losing trades in a row at win% of 50% is P=0.03, therefore as P<0.05 the system is broken and we are shutting it off.

My dilemma, and the flipside of this is, we are not doing 1 sample. As we trade, each new trade is another chance to start a streak. If we do 300 trades, with a "switch" to turn the system off after 5 losing trades, at Win% = 50%, im 99.99% certain that there will be 5 losers in a row, even if the system and environment doesn't change. Even if we only do 100 trades, im 95.82% certain that we'll hit 5 in a row, without a change in the system or enviroment. It's the same with a coin toss, 300 tosses and there is close to a 99.993% chance of 5 tails in a row, this is just the nature of probability and randomness.

I've heard of 12 flips in a row, the likely hood of this event occuring is 0.0244%. Here is the crux, i bet everything i own, that if you go and toss a coin 50,000 times, you'll have 12 flips of either heads or tails in a row. (if anyone wants to take me up on this, go ahead ) An event with a 0.0244% probability is around a 1 in 4096 chance occuring.

Let's say we shut our system down after P<0.05, and we put it back into validation stage. Do you then start to paper trade it and wait for the equity curve to be at a new high, re backtest in sample & out of sample or re-optimize it for current conditions?

What's my point? My point is that, there needs to be some kind of seperate way to determine whether the environment has changed, causing our system to be broken, rather than soley relying on probabilty. Everything that can happen, will happen, given enough repetitions. Therefore, assuming that the enviroment always remains static, probability can never shut down a system because every possible combination has to be expected to occur if we sample long enough.

I am not smart enough to devise a way to seperate a system from the enviroment it is operating in, and analyse the two in isolation, to determine whether the enviroment has changed. The system can never "break" if the code remains the same, the market has to change for the system to fail. I don't know how to quantitatively meausre whether the nature of the market has changed, if there is a way at all, therefore some type of judgement call has to be made as to whether the result is a true failure of the system, or a psuedo failure attributed to the system failing when infact it hasn't, we have just sampled for long enough that a low probability event has occured.

Regards
Brad

Hi Brad --

Perhaps a couple more comments, then let this rest.

Any system developer who is trading his system faces exactly the situation we are both talking about -- measured by some statistic, the results from recent trades are poorer than the out-of-sample data. And the trader must decide whether to continue trading that system or not. You can pick your criteria, and I can pick mine. What criteria each of us uses depends on our individual tolerance for risk.

You stated:
"What's my point? My point is that, there needs to be some kind of seperate way to determine whether the environment has changed, causing our system to be broken, rather than soley relying on probabilty. Everything that can happen, will happen, given enough repetitions. Therefore, assuming that the enviroment always remains static, probability can never shut down a system because every possible combination has to be expected to occur if we sample long enough."

My answer is that the statistical tests I have been talking Are the way in which to separate the model from the data.

In my description of a trading system, it is the model that is static and the environment that is dynamic rather than the other way around.

You have commented several time about Shut Down. I do not see putting a trading system on hold as a final decision. When the statistics indicate to me that the system performance has deteriorated, I stop making real-money trades, but continue to watch the system. If results from the paper trades improve, I can have some confidence that the system is still viable. If they do not improve, I must go through the validation process for that system again.

And you wrote:
"I am not smart enough to devise a way to seperate a system from the enviroment it is operating in, and analyse the two in isolation, to determine whether the enviroment has changed. The system can never "break" if the code remains the same, the market has to change for the system to fail. I don't know how to quantitatively meausre whether the nature of the market has changed, if there is a way at all, therefore some type of judgement call has to be made as to whether the result is a true failure of the system, or a psuedo failure attributed to the system failing when infact it hasn't, we have just sampled for long enough that a low probability event has occured."

My point is that I have been describing Exactly the procedure you are asking for. The judgment is the amount of risk a trader is willing to take. That translates into the statistic that is used to determine when to stop taking real-money trades.

Yes, rare things happen. Re-read my earlier comments about the trade-off between type I and type II errors. Something rare that causes me to stop taking real-money trades when the system is not broken causes me to lose an opportunity. The real danger comes when the system is broken and I continue to trade it.

Thanks for listening,
Howard

Thanks Howard
Brad