# Data, lots of it...



## tothemax6 (4 February 2011)

Hi All,

I have a question: suppose I wanted to have, on my hard-drive, the price and financials of every asx listed company (including both new and no longer existing ones) over the last 10 years, and also commodity prices & interest rates etc - how would I make this happen? 
If anyone has any insight into how I would gain/gather this data, it would be gratefully received .


----------



## robusta (4 February 2011)

OK a couple of dumb questions.

Why do you want this data on your hard drive when you can look it up anytime?

How do you determine what information is useful in this mountain of data?

We are living in the information age - there is no shortage of data just finding the useful bits is the trick.


----------



## tothemax6 (5 February 2011)

robusta said:


> OK a couple of dumb questions.
> 
> Why do you want this data on your hard drive when you can look it up anytime?
> 
> ...



Firstly, I wish to calculate correlations between various things. Why is a long story. So yes data is abundant, however having it readable at harddrive access speeds is required (time is precious).
Regarding the useful bits, yes in a roundabout way this is actually my goal, but clicking through broker data and typing in stock codes in web pages just isn't going to suffice for my needs.


----------



## TulipFX (5 February 2011)

robusta said:


> OK a couple of dumb questions.
> 
> Why do you want this data on your hard drive when you can look it up anytime?
> 
> ...




Quantitative analysis. I have a 1TB HDD just dedicated to all the information. It is almost full with price data, analysis data, testing data and various combinations of those.

It is all forex though, so I don't know if that's useful for you.


----------



## cudderbean (5 February 2011)

tothemax6 said:


> Hi All,
> 
> I have a question: suppose I wanted to have, on my hard-drive, the price and financials of every asx listed company (including both new and no longer existing ones) over the last 10 years, and also commodity prices & interest rates etc - how would I make this happen?
> If anyone has any insight into how I would gain/gather this data, it would be gratefully received .




I have used Aussie home grown Weblink who market Bullcharts for years now, with which I am delighted... beats Metastock hands down IMHO. They have always been very helpful and responsive to my inquiries. I'm pretty sure they would have loads of backdata, which you could purchase from them. Well worth contacting them anyway. Good luck, mate.

Telephone
    Australia:	(02) 9884 7622
    International:	61 2 9884 7622
Fax
    Australia:	(02) 9884 7633
    International:	61 2 9884 7633

Email:	info@bullsystems.com 
Website: www.bullsystems.com
and 
http://www.weblink.com.au/ 


Address:		BullSystems Pty Ltd 
		Level 11, Tower B 
		The Zenith Centre 
		821 Pacific Highway 
		Chatswood  2067
		Australia


----------



## tothemax6 (6 February 2011)

TulipFX said:


> Quantitative analysis. I have a 1TB HDD just dedicated to all the information. It is almost full with price data, analysis data, testing data and various combinations of those.
> 
> It is all forex though, so I don't know if that's useful for you.



Cool, but no not what I am looking for to begin with. Perhaps I will extend my research to exchange rate correlations with commodities and stocks at a later date. Could I ask though, how did you go about accumulating this 1TB of data?
Cheers


----------



## TulipFX (6 February 2011)

tothemax6 said:


> Cool, but no not what I am looking for to begin with. Perhaps I will extend my research to exchange rate correlations with commodities and stocks at a later date. Could I ask though, how did you go about accumulating this 1TB of data?
> Cheers




From various different brokers servers. Dukascopy is my favourite though because you can pull off true tick data from their server. That is inter-second prices with the spread data included. Very helpful when testing systems which require good accuracy.

The raw data is probably about 100gb worth. Then I have another 200 of it in a couple of different formats. Add in about 100gb of analysis data. Then I have a test neuronet bot running which creates about 350mb of learning data a day and it starts to add up to quite a lot.


----------



## tothemax6 (6 February 2011)

TulipFX said:


> From various different brokers servers. Dukascopy is my favourite though because you can pull off true tick data from their server. That is inter-second prices with the spread data included. Very helpful when testing systems which require good accuracy.
> 
> The raw data is probably about 100gb worth. Then I have another 200 of it in a couple of different formats. Add in about 100gb of analysis data. Then I have a test neuronet bot running which creates about 350mb of learning data a day and it starts to add up to quite a lot.



Has your neural network programming yielded any positive results?


----------



## TulipFX (6 February 2011)

tothemax6 said:


> Has your neural network programming yielded any positive results?




At the moment we are just experimenting with it, getting it to work. We are using only one indicator (RSI) just to get it all going. It seems to be working.







You can see that as it was learning at the start it was doing a pretty poor job of deciding how to trade, since then it seems to be doing a half decent job given it has only been given one default variable and constant SL/TP to work with.


----------

