Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Perl

[ajcode]

Another project I've been working on is downloading End of Day ASX stock quotes from yahoo finance and loading these up on Metatrader 4 charts. What is required here is the installation of perl on your machine in order for things to work. I think from memory you can download and install Active Perl or Strawberry Perl on your Windows Machine. Anyway I will start this first post with the downloading of quotes (Stage 1).

I have attached 2 files called "yhquotes.pl" and "symlist.csv". So basically "yhquotes.pl" is a perl script that reads the stock symbols/tickers from symlist.csv file and downloads all quotes into a single file called "quotes.csv".

"quotes.csv" follows the format of symbol, date, open, high, low, close and volume.

If you have installed perl correctly, you should be able to double click on the file "yhquotes.pl" to execute the script. For illustrative purposes I have uploaded a screen shot of what to do if you were to use the command line:



So if your were to execute script yhquotes.pl, it should create a file called quotes.csv.

Important Note: I had to add a .txt extension to upload the files as .pl and .csv is not allowed. Please remove the .txt extension for this to work

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Ok, now for Stage 2. Here you need to load up your favourite Metatrader 4, create a demo account and open up a live chart. Once you have done this you need to check where abouts in you MT4 history folder, this chart data is being stored. This is where all the download files are to be placed.

Here is an example (Note: I already know where chart data is stored so I didn't open up any live charts).



Once you have done that just execute yhquotes.pl to create the quotes.csv file like so



The next step is to execute the script mt4dw.pl which is short for mt4 data writer. This perl script calls my dll (MT4DWDLL.dll) which then creates the MT4 history files from the quotes.csv file. So when you execute the script you will see the following:



and this is the result:



Next open up Metatrader 4, select File/Open Offline and you should be able to load up your ASX stock charts like so:

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

I have attached the final 3 files "mt4dw.pl", MT4DWDLL.dll and "MT4DWRemove.bat".
The file "MT4DWRemove.bat" just executes the command "del _*.hst" which only removes the _.hst files created from my scripts. It's a nice way to tidy up.

Note: Please remove .txt extension at the end for these to work.

Cheers,

Andrew.

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

Once you have your Metatrader 4 EOD history files, you can then use the period converter in MT4 to create a weekly chart (for each stock you are interested in, this one being BHP) like so:



and then you can view the offline weekly chart like so and add a few indicators



According to Yahoo Quotes, BHP is really loving that 61.8% Fibonacci Retracement

Cheers,

Andrew

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

I have attached an updated version of the file symlist.csv, which contains pretty much the full list of ASX stocks that yahoo finance has to offer, there are about 2100 symbols in this file. The method of obtaining the list was brute force

Note: Please remove .txt extension.

It would be good to know if anyone could tell me if they are able to follow my instructions and get this to work. Maybe someone could post a few screenshots of ASX stocks on their Metatrader platform

Cheers,

Andrew.

 

[ozhans]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

I have attached an updated version of the file symlist.csv, which contains pretty much the full list of ASX stocks that yahoo finance has to offer, there are about 2100 symbols in this file. The method of obtaining the list was brute force

Note: Please remove .txt extension.

It would be good to know if anyone could tell me if they are able to follow my instructions and get this to work. Maybe someone could post a few screenshots of ASX stocks on their Metatrader platform

Cheers,

Andrew.

Great job Andrew, you might want to add that it only works with the 32-bit version of Perl and not with the 64-bit version.

So if I understand it correctly you need to run the perl script everytime you want to update the information?

Hans.

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

Great job Andrew, you might want to add that it only works with the 32-bit version of Perl and not with the 64-bit version.

So if I understand it correctly you need to run the perl script everytime you want to update the information?

Hans.

Thanks.

Yes that is correct, all my programming projects so far are 32bit. I've done all these projects running Windows XP on a Pentium 4 IBM thinkpad R52 with 512MB ram. I'm an 80's/90's programmer as you can tell from the use of perl and I see C++ as a better C.

Yes again, you must re-run the perl script everytime you want to update the information. Might take less time if you change the start year $sy from 2000 to say 2010 as you don't need that much information for charting.

As we are dealing with in-accurate data, I don't see much point in putting too much effort into merging latest data with old data. Easier just to download the whole thing again.

 

[Tito006]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Andrew,

I have no words to thank you for this.
I really appreciate your creation.

However I was not able to download data from other markets.
I think the reason is this line:

$ex = "AX"; # Exchange AX for Australia

I tried to download data for French stocks, adding this other line:

$ex = "PA";

And I got it.
This is the example of Renault:




But when I do this, data from Australian shares are not downloaded.
And when I add German stocks ( $ex = "DE"; ), for example, the French stocks are not downloaded.

Moreover, Yahoo does not assign suffixes to American stocks (NYSE, NASDAQ and S&P 500).

Do you have a solution for this?

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Andrew,

I have no words to thank you for this.
I really appreciate your creation.

However I was not able to download data from other markets.
I think the reason is this line:
$ex = "AX"; # Exchange AX for Australia
I tried to download data for French stocks, adding this other line:
$ex = "PA";
And I got it.
But when I do this, data from Australian shares are not downloaded.
And when I add German stocks ( $ex = "DE"; ), for example, the French stocks are not downloaded.
Moreover, Yahoo does not assign suffixes to American stocks (NYSE, NASDAQ and S&P 500).

Do you have a solution for this?

Hi Tito, Great to see someone posting a screen shot of something working

Not sure what the best way to do this is. One method is to install metatrader in a different directory for each market you want to look at. One for Aussie Stocks, One for French Stocks, One for US stocks, etc. Then have this many copies of the files I uploaded above. Now the file symlist.csv only contains symbols for Aussie Stocks, so if you wanted French, German or US stocks, then you will have to fill that with the list of stocks that are available for their respective markets. Secondly the perl script as it stands now only downloads for 1 market and because the resultant stock name does not include the suffix/exchange, you can't have 2 market versions of the same stock name.

To download US stocks, you will need to change the following line in yhquotes.pl from:

$url .= ".$ex&a=$sm&b=$sd&c=$sy&d=$em&e=$ed&f=$ey&g=d&ignore=.csv";

to

$url .= "&a=$sm&b=$sd&c=$sy&d=$em&e=$ed&f=$ey&g=d&ignore=.csv";

We are removing the .$ex because as you say "Yahoo does not assign suffixes to American stocks".
You might also want to change start year from $sy = 2000 to 2010 as you might not want that much data for charting and I think it's a little quicker to run.

The second solution is for me to modify the perl script to download for any market and to do this you would have to change the stock names (except US market) in symlist.csv to something like:

BHP
BHP.AX
RNO.AX
RNO.PA

and the resultant file names after running mt4dw.pl would hopefully look like this:

_BHP1440.hst
_BHP.AX1440.hst
_RNO.AX1440.hst
_RNO.PA1440.hst

Is that something you'd like to see?

Cheers,

Andrew.

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Pe

I have attached files symlistEX.csv & yhquotesEX.pl. The change in the symbol file now is that I have added the suffix/exchange that the stock resides in (except if it is a US stock) according to yahoo finance. So for ASX stocks you need to put the .AX suffix.

When you run the yhquotesEX.pl script now, the following symbols from the file symlistEX.csv will be processed.

AAPL
GOOG
MSFT
BHP
BHP.AX
RY.TO
RNO.PA
0005.HK
TEL.NZ
CBK.DE
TCS.BO
TCS.NS
AHAP.JK
O23.SI
BP.L
BBAS3.SA
600000.SS
DANSKE.CO
BP.MI
ASC.OL
005930.KS
2891.TW

and when you run mt4dw.pl, you should get the following mt4 history files:

_AAPL1440.hst
_GOOG1440.hst
_MSFT1440.hst
_BHP1440.hst
_BHP.AX1440.hst
_RY.TO1440.hst
_RNO.PA1440.hst
_0005.HK1440.hst
_TEL.NZ1440.hst
_CBK.DE1440.hst
_TCS.BO1440.hst
_TCS.NS1440.hst
_AHAP.JK1440.hst
_O23.SI1440.hst
_BP.L1440.hst
_BBAS3.SA1440.hst
_600000.SS1440.hst
_DANSKE.CO1440.hst
_BP.MI1440.hst
_ASC.OL1440.hst
_005930.KS1440.hst
_2891.TW1440.hst

If you're only looking at the ASX market, then you don't need to worry about this post. But to generate charts in other markets, you still need to manually update the symlist file yourself as this is an ongoing process.

Note: Please remove the .txt extension from uploaded files below for this to work

 

[Tito006]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

It worked perfectly.
Thank you so much!

You're rock!

 

[doctorspok]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

THANK YOU! For this....
I really appreciate it!

 

[dead trader]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Wow, this is amazing!! Thank you so much ajcode!

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Good to hear that people have got it working and are pleased with it

Wow I have registered on the speedo with 32 (technically 31) downloads of my 32 bit MT4DWDLL.dll. This could go Platinum in 5 years

It would really make my day if in 10 years time MT4 is still around and someone on ASF posts a comment like "I'm still using that dll you posted on Aussie Stock Forums 10 years ago back in 2014"

At Aussie We'll Save You :aus:

 

[Bernmeister]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Very nice work ajcode...but a warning:

In the last few months I wrote myself a Python script to download Yahoo EOD data for ASX. Works well enough...but then I noticed not only is some of the data not being updated by Yahoo (such as Wesfarmers since December), but some of the data is just plain wrong.

Specifically I noticed that APA.AX is historically wrong - the price in the Yahoo history does not match with what actually traded. I'm note referring to the adjusted close - rather the OHLC should not be changed from what traded on the day.

In short, the Yahoo data is unreliable and at worst, WRONG!

So now I'm on the hunt for a proper ASX EOD provider...

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Hi Bernmeister,

Yeah yahoo data is not for serious traders and has many problems, but it's still ok for free and mess around with. Having said that, I think you can trial free metastock data from Data HQ or Premium Data for 1 month that contains accurate stock data. If you do that, you might want to have a read of my other thread which is:

https://www.aussiestockforums.com/forums/showthread.php?t=28692

One member (in the above thread) so far has been able to chart this Metastock data in MetaTrader 4 okay provided everything is 32 bit.

Cheers,

Andrew.

 

[dazrul]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

interesting & very useful info... thanks ajcode.

got this info from forexfactory finally.

i normally download eod data using jstock.. then manually upload it to history centre..

I use any currency symbol chart for charting. very tedious and time consuming.


will use your method to import data feed for bursa malaysia (xxxx.kl).

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

i normally download eod data using jstock.. then manually upload it to history centre..

Hi dazrul,

You could probably still use jstock, but you have to get the format right (symbol,date,open,high,low,close,volume) and store the quotes in the one big file called "quotes.csv" for my perl script "mt4dw.pl" to process it.

This could probably be done by exploring my (less looked at ) thread "Reformatting CSV (Comma Separated Value) files using Perl"

https://www.aussiestockforums.com/forums/showthread.php?t=29270

and making a few minor adjustments to the perl script "csvref.pl".

Cheers,

Andrew.

 

[WillyWonka]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

I have attached an updated version of the file symlist.csv, which contains pretty much the full list of ASX stocks that yahoo finance has to offer, there are about 2100 symbols in this file. The method of obtaining the list was brute force

Note: Please remove .txt extension.

It would be good to know if anyone could tell me if they are able to follow my instructions and get this to work. Maybe someone could post a few screenshots of ASX stocks on their Metatrader platform

Cheers,

Andrew.

Great post! I often wonderend if this was possible. I did however notice you mentioned obtaining the full asx list via brute force. I made an asx excel sheet last year which downloads data from yahoo finance as well. When I was building this I found the the asx web page has the current company list available for download here
http://www.asx.com.au/asx/research/ASXListedCompanies.csv

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

Great post! I often wonderend if this was possible. I did however notice you mentioned obtaining the full asx list via brute force. I made an asx excel sheet last year which downloads data from yahoo finance as well. When I was building this I found that the asx web page has the current company list available for download here
http://www.asx.com.au/asx/research/ASXListedCompanies.csv

Yeh ASXListedCompanies.csv is a great find. I missed that. All you need to do is keep the code column and delete the other columns and you have got yourself the official up to date list. My brute force way was to go through all the combinations from AAA to ZZZ and if the corresponding yahoo finance url existed then I'd add it to the list. This took over 10 hours