# Daily Probability



## r22mark (15 February 2010)

Current project I am working on evaluates the probability / percent chance that tomorrow will either be up or down. 

After some thinking / research I  have this:
Probability of an event = P(E)
n(E) = # favourable outcomes
n(S) = # outcomes
P(E) = n(E) / n(S)

Relating the above equation into candle terms:
up1 = # times a day was up
up2 = # times two days were up in a row
total = total number of days / bars

Results from XJO:
up1 = 1244
up2 = 646
total = 2363

Prob any 1 day being up = up1 / total
XJO: 1244 / 2363 = .53 ie. 53% possibility that a day will be up

Prob any 2 days being up = up2 / total 
XJO: 646 / 2363 = .27 ie. 27% possibility that 2 up days will occur at any time. Error here?

This is where I think I may have it wrong, calculating prob for tomorrow (XJO):
up1 = 1244 (# times 1 day up)
dn1 = 1119 (# times 1 day down)
up2 = 646 (# times 2 days up)
dn2 = 522 (# times 2 days down)
totaldays1 = 1244 + 1119 = 2363 (total # outcomes for 1 day up or down)
totaldays2 = 646 + 522 = 1168 (total # outcomes for 2 days up or down 

Today was up, then tomorrow up as well = (up1 + up2) / totaldays1 + totaldays2 ?

XJO: (1244 + 646) / (2363 + 1168) = .54 ie. 54% that tomorrow will be an up day, if today was an up day ?

Is this correct?

Mark


----------



## Knoxy (15 February 2010)

If today was up then shouldn't it simply be probability 646/1244 = 52% that tomorrow will be?

ie for the 1244 cases a day was up, 646 cases went on to give a second up day, 1244-646 times the next day was down.


----------



## RamonR (15 February 2010)

Factor in what America did overnight and to a lesser degree what Asian markets have also done.

Also if any company is a big part of the XJO then when they go ex div expect them to fall


----------



## r22mark (15 February 2010)

Knoxy: think you may have solved my dilemma, thanks

RamonR: plan on including it, just getting the basics down first

Mark


----------



## bilo83 (15 February 2010)

I think 52% is right. You can use the equation for conditional probability 
P(A/B)=P(AB)/P(B). Doesnt really tell you much without the size of up and down days and their distribution.


----------



## RamonR (15 February 2010)

Think also the the market spends 80% time going up and 20% going down.
Which means the down movement is likely to be more dramatic.

if you get this right do I get a copy of formula for help given thus far?


----------



## r22mark (15 February 2010)

bilo83: you want to apply your conditional probability formula for the example given, if it would give different / more accurate results.

RamonR: happy to post amiboker code if desired - I appreciate the help!

Mark


----------



## ginar (15 February 2010)

RamonR said:


> Factor in what America did overnight and to a lesser degree what Asian markets have also done.
> 
> Also if any company is a big part of the XJO then when they go ex div expect them to fall




You can probably add commodity prices , AUD currency changes along with what BHP ADRs have done , these things are interconnected though so possibly just one of those may do as an input .


----------



## saiter (15 February 2010)

No you are wrong.

P(Up) = 0.53
P(Up AND Up) = P(Up)*P(Up) = 0.53^2 = 0.28 -> 28%

*total* should always be equal to the barcount with regards to your question

When asking if P(X) AND P(Y) occurs, then you need to multiply the probabilities. This applies for P(X) AND P(Y) AND P(Z) AND....
When asking if P(X) OR P(Y) occurs, then you need to add the probabilities.

Again, when calculating P(X), then the total needs to equal the number of bars.


If you eventually manage to code this (I'm assuming you're using amibroker), then you will find that for most asx stocks, when observing P(Up) or P(Down) over many years, the two will be roughly equal. Obviously, you will find a difference in probabilities when you look at a much shorter time frame. IIRC, during 2005-2009 the XAO's P(Up) was almost equal to P(Down), however, during '09, some months had P(Up)=0.6 and P(Down)=0.2 whereas others had P(Down)=0.5 and P(Up)=0.2.
Hopefully the above paragraph has also highlighted another outcome that you aren't considering.


----------



## Trembling Hand (15 February 2010)

And then you have the problem of using the probablities.

Are you taking an Up day from the open of the same day or the close of the day before or from open close the same day?

Because you can have an up open(from yesterdays close) and a down day (open to close same day) And then from there you can still be either up or down from yesterdays close. 

Just a warning, ASX XJO data is of no use unless you are only concerned with close to close.


----------



## r22mark (15 February 2010)

Saiter: Yes I am using amibroker. "P(Up AND Up) = P(Up)*P(Up) = 0.53^2 = 0.28 -> 28%" Wouldn't that be true if I only had data for one day, and was then trying to work it out for two up days? I determine that today was an up day, and am trying to work out the prob that tomorrow will also be up. Maybe try to give an example in my previous XJO terms?

Trembling Hand: in an attempt to sort the basics I am using an up day is close > open. I will expand on that later.

I don't presume that this will be life altering. It's an just an area of interest. XJO is a starting point.

Mark


----------



## Trembling Hand (15 February 2010)

r22mark said:


> Trembling Hand: in an attempt to sort the basics I am using an up day is close > open. I will expand on that later.
> 
> I don't presume that this will be life altering. It's an just an area of interest. XJO is a starting point.
> 
> Mark




You are wasting your time then. ASX index data is rubbish in relation to the opening tick. see my blog.


----------



## tech/a (15 February 2010)

Trembling Hand said:


> You are wasting your time then. ASX index data is rubbish in relation to the opening tick. see my blog.




Concur

Infact is probably more accurate to fade.


----------



## r22mark (15 February 2010)

This is how I see it at the moment:
The market (XYZ = irrelevant) is like a coin. It can be up (C > O) or it can be down. This coin is up more often than down. I have found:
Numbers after up / down are # times in a row that it occurred. Up3 = up for three days in a row.  Different results. So:
up1 = 1307
up2 = 677
up3 = 348
dn1 = 1190
dn2 = 560
dn3 = 271
totalbars = up1 + dn1 = 2497

Today was an up day for XYZ. Given the above info, can it be determined what the possibility that tomorrow will be an up day? If not, what additional info is needed? 

I thought that possibility was up2 / up1 = 677 / 1307 = .52 = 52%
Likewise the possibility of having 3 up days in a row was up3 / up1 = 348 / 1307 = .27 = 27%

If you want to quote prob formula's please try to apply them to this exercise.

Once it is determined that I have this initial step correct, THEN I will do other scenarios / markets.

Mark


----------



## saiter (15 February 2010)

r22mark said:


> Saiter: Yes I am using amibroker. "P(Up AND Up) = P(Up)*P(Up) = 0.53^2 = 0.28 -> 28%" Wouldn't that be true if I only had data for one day, and was then trying to work it out for two up days? I determine that today was an up day, and am trying to work out the prob that tomorrow will also be up. Maybe try to give an example in my previous XJO terms?
> 
> Trembling Hand: in an attempt to sort the basics I am using an up day is close > open. I will expand on that later.
> 
> ...




Probability of 2 up days (i.e. Day 1 = up, Day 2 =up)
XJO: up2/total OR up1/total * up1/total

Probability of any 2 day pattern ending up (i.e. Day 1 = up or down, Day 2 = up)
XJO: up1/total


----------



## r22mark (15 February 2010)

http://www.investmentwarrior.com/Archived%20Articles/111402.htm

Their findings are what I aiming to prove / disprove.

Mark


----------



## professor_frink (15 February 2010)

r22mark said:


> http://www.investmentwarrior.com/Archived%20Articles/111402.htm
> 
> Their findings are what I aiming to prove / disprove.
> 
> Mark




I have never seen those sorts of claims before about any of the major markets. BS would be my first thought without even knowing which markets or which era is being discussed


----------



## Trembling Hand (15 February 2010)

r22mark said:


> This is how I see it at the moment:
> The market (XYZ = irrelevant) is like a coin.



  Actually when you get to trading it it isn't like a coin at all.



r22mark said:


> It can be up (C > O) or it can be down.




No you are missing a very important step when it comes to trading. Firstly are you using that formula with XJO data because you couldn't of got those stats. Pre feb 07 C = O for xjo. 

Then you actually may want to look at this formula if you ever want to apply any of this to trading,

Overnight gap +/- intraday movement > yesterdays close, is an up day.

Otherwise you are going to be forced to trade prices that are not available ie gaps and opening/closing prices.

Then of course you will not be able to apply averages of 10 years of data to a trading system. You will blow up before you have a chance of getting the 10 years to work in your favour.


----------



## r22mark (15 February 2010)

Trembling Hand: You seem to have misunderstood my INITIAL market assumption for this exercise. ie. up or down. Yes I am aware that just because today was up doesn't that it was up compared to yesterday, etc. Start simple, get the maths right, then add complexity / real life later. 

I just want confirmation that I am calculating the probabilities for this event correctly.

Regarding my initial choice of XJO - who cares? At this point it doesn't matter. Incidentally my XJO data (Premium Data) has O/H/L/C/V data from 3/4/2000 till now. Do you want to pick stock and I'll work with that?

Mark


----------



## Trembling Hand (15 February 2010)

r22mark said:


> Regarding my initial choice of XJO - who cares?




I thought you would care before you go about wasting time with untradeable prices but apparently not, 

And I would give some good thought to applying 10 years _average _data to a trading system or the basis of a trading methodology. I understand you're looking at the correct maths to apply here but I'm thinking a couple of steps ahead of you - practicality.


----------



## r22mark (15 February 2010)

Trembling Hand: at the end of this exercise I plan on having a code that spits out something like "There is  xYears of data for XJO (insert generic index). Today XJO gapped up and closed up 1% on above average volume. This is the third overall day day up. The index daily trend is up. There is a x% chance that tomorrow will be up. There is xYears of data for BHP. Today BHP was down and closed 0.5% lower on above average volume. There is a x% chance that BHP will be down tomorrow. etc"

What I do with it depends on how meaningful the results turn out to be. Might form part of an entry, adjust a stop, or be background info. Can't reach that place yet till the basics are known.

I'm sorry I don't understand what you mean with "applying 10 years average data to a trading system". Assuming any data is complete with OHLCV info, do you consider 10 years worth of data insufficient for backtesting / evaluating a system? I do appreciate your input.

Mark


----------



## Trembling Hand (15 February 2010)

Firstly Mark I Dont think you are following me here. You simply *cannot *use any index data from the ASX that includes the opening price. All that can be used is C on C and maybe some of the HL & V data, thats a big maybe. 

Away from that, using the last 10 years of data is like trying to predict if it will rain tomorrow from averaging the last 30 years of weather. Its easy enough to get a probability but for tomorrow will that apply if we are in a drought? or if its December or July.

The last 10 years has had 6 years (03-07 & 09) of the biggest bull markets we have seen in a long time. Is the market going to repeat the same patterns in the next 10 years?? Then have a look at volatility on a short term basis, daily ranges & day to day, to see how that is continually expanding and contracting over such a long time.

Then if it does repeat, will your system survive the day to day, week to week, month to month fluctuations of the inevitable moves away from the averages? You could well find a good system that works on 10 years of data but my guess is that you it will also have nasty drawdowns.

And of course I'm not a system trader but  do know that the only results that will really count are out of sample results. Are you going to test it for another 5 years to get confirmation?


----------



## r22mark (15 February 2010)

Trembling Hand, 







> You simply cannot use any index data from the ASX that includes the opening price. All that can be used is C on C and maybe some of the HL & V data, thats a big maybe.




Now I am confused. Is this due to the indices extended trading hours, and the open may not be accurate? Is this limited to ASX indices or is common worldwide?

I am limited to data available, obviously the longer the better. I have 20yrs of data for XAO, for example. Systems traders argue over the validity of new versus old data. But for this exercise, more is better. I recall reading somewhere that you need a minimum of 30 results generally to give a statistically valid inference (another formula to find). Whatever that number that is, if an event runs under that minimum it will be noted in the output text.

Mark


----------



## Trembling Hand (15 February 2010)

r22mark said:


> Now I am confused. Is this due to the indices extended trading hours, and the open may not be accurate? Is this limited to ASX indices or is common worldwide?




Just to the ASX, the staggered open distorts the first tick, the opening value, for the Indexes. See my last post on my blog 



r22mark said:


> I am limited to data available, obviously the longer the better.



 I don't think so. The market is not some static beast. Patterns come, go, breakdown and return all the time. For example have a look at this which is similar to what you are trying to do,
http://tremblinghandtrader.typepad.com/trembling_hand_trader/2007/07/spi.html

It showed that all the bull move was in the overnight gaps. trading this "method", holding only over night & short during the day would of given you $100,000 return uncompounded from $20,000 over 3 years, then look at the date. Just as I found it it broke down.

I think you better look into the more data is better idea. I don't think thats correct, not for a trading system. maybe some of the system traders could help.


----------



## professor_frink (15 February 2010)

Mark,

just on the ideas of looking for probabilities of next day returns, looking at multi day runs in the same direction, etc, it would be well worth having a look through the marketsci blog, found here:

http://marketsci.wordpress.com/

A couple of posts to get you started

short term stock market mean reversion becoming stronger

exploring multi day stock market runs


----------



## r22mark (15 February 2010)

Trembling Hand: staggered open, makes sense. Will revise code used for index results, with attention to gaps. Thanks for the heads up.

professor_frink: haven't encountered that blog, looks like good info.

Another poster suggested this:
Summing independent events, P(A and B) = P(A) * P(B) BUT working out P(B).

P(A and B) = 2 days up = up2 / totalbars = 677 / 2497 = .27

P(A) = today up = up1 / totalbars = 1307 / 2497 = .52

so P(B) = tomorrow up = P(A and B) / P(A) = .27/.52 = .52 = 52% chance that tomorrow will be up also ?
Likelihood of a third day up  = 27%.

Another method, but which one is correct?  

Mark


----------



## brty (15 February 2010)

r22mark,



> I have 20yrs of data for XAO,




I think that TH has been a little soft in the way he has communicated to you in this thread. I don't think you understand what he has been saying. 

Allow me to elaborate...

This is a totally pointless waste of time.



> 52% chance that tomorrow will be up also ?
> Likelihood of a third day up = 27%.




Good grief....; no wonder academics think the markets are random and can't be beat if they use this type of logical approach to something that is not logical.

Have you bothered to check the variations in probabilities during different periods within the original 10 years of data you used??? TH already told you that the bull and bear phases acted differently, a quick look at any chart over the 10 years will display it.

How would you trade it?? If taking XJO data (cash) the results will be different to the SPI data, as sometimes the SPI trades at a premium, sometimes at a discount, it is not a straight one for one, especially when you are trying to eek out a slight perceived discrepancy in probability over 10 years.

Enough said

brty


----------



## professor_frink (15 February 2010)

r22mark said:


> Trembling Hand: staggered open, makes sense. Will revise code used for index results, with attention to gaps. Thanks for the heads up.
> 
> professor_frink: haven't encountered that blog, looks like good info.
> 
> ...




Honestly, I've completely lost track of what you are trying to do here, I get confused easily seeing random equations all over the thread

Here's a simple excel file with some number crunching on it for you. Just dumped some data from yahoo into excel and did a little of my own playing around. Actually scratch the excel file, the forum won't let me upload it

If today was an up day(based on yesterday's close, the only way to work it for index data), then 50.91% of the time we will see another up day, though trading this is a losing proposition.

If there are 2 consecutive up days, then 51.07% of the time we will see a 3rd up day, though the average trade result is so small you wouldn't even consider  following it.


----------



## professor_frink (16 February 2010)

ok, now that we have the excel attachment limit on the forum fixed, here's the excel file I was trying to attach last night. Not sure if it'll help you on your quest or not Mark.


----------



## r22mark (16 February 2010)

brty: First post stated this was a project - I never implied it was the holy grail. I posted asking for help to determine the correct way to calculate the various probabilities, something I still don't have an answer to. I have not implied this is a trading system for stocks or indices. As for "variations in probabilities during different periods", refer to the chart (bottom indicator) below - the prob of a random up day has changed little in the last 10 years.  Middle indicator shows consecutive up / down days. Yes you can make some observations in different market states. Yes I do try to look at all the factors. But as you previously commented "This is a totally pointless waste of time." Check out http://www.investmentwarrior.com/Archived%20Articles/111402.htm
That is what I am trying to prove / disprove.

Mark


----------



## Trembling Hand (16 February 2010)

Mark from investmentwarrior.com,



> Such inefficiencies make empirical and statistical  timing strategies (such as those used in this report) to  take advantage of such and produce systems that help to both *reduce risk and enhance return*




Having a look at the risk to reward of each move, which is about 1:1 not counting brokerage + slippage, I would say the above quote is "totally pointless waste of time" ie BS.

There is no edge there.


----------



## motorway (16 February 2010)

http://clayallen.com/MD_NEWS_V9_I3.pdf

It is Not Days up vs DAYs down that makes the difference
That is a coin toss

It is how much up vs how much down
when ever price moves
that is the engine of TRENDS
and where persistence resides 




> Another interesting fact about
> the JCG data is that 51% of the days
> were up and 49% were down. That is
> very close to flipping a coin. The trend
> ...






Yes you could almost say this chart is driven by a  RANDOM PROCESS ( days up<=> days down )

BUT *no *this chart is NOT RANDOM ( The step sizes are NOT RANDOM )
There are persistent TRENDS

Motorway


----------



## Trembling Hand (16 February 2010)

motorway said:


> There are persistent TRENDS




And this is where Mark is not looking and will BLOW UP in using 10 year averages as an idea. You only have to be wrong more than right for a few months to be completely blown out of the water.


----------



## brty (16 February 2010)

r22mark,

The graph you have presented that shows "prob of up day /30 days" is not accurate. Between the 19/5/08 and 15/7/08 there were 41 trading days. On 26 of those days the closing price was lower than the previous close. At no point does your graph fall below a probability of .53 of there being an up day during this period, even though it is per 30 days.

Using my calculator I find that by the 15/7/08 there would be a 63% probability of a down day, if the time was averaged over 41 days, hence I return to my assumption that it is a pointless exercise and I am stupid for wasting my time on it.

brty


----------



## r22mark (16 February 2010)

brty: Apologies. Coding error. You are correct - prob of a single day being up ranges from 30% to 68%, over 30 bars / 10 yrs. That did surprise me.

This exercise has become disappointing. If anyone cares to comment on the initial mathematical question, please do. Otherwise, I won't be continuing this.

Mark


----------



## doctorj (16 February 2010)

r22mark said:


> brty: Apologies. Coding error. You are correct - prob of a single day being up ranges from 30% to 68%, over 30 bars / 10 yrs. That did surprise me.
> 
> This exercise has become disappointing. If anyone cares to comment on the initial mathematical question, please do. Otherwise, I won't be continuing this.
> 
> Mark



I don't have a lot to add, aside from a thanks.  It was a useful exercise in that it's something most traders ponder at some point.  What's next?


----------

