Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Perl

[WillyWonka]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

Yeh ASXListedCompanies.csv is a great find. I missed that. All you need to do is keep the code column and delete the other columns and you have got yourself the official up to date list. My brute force way was to go through all the combinations from AAA to ZZZ and if the corresponding yahoo finance url existed then I'd add it to the list. This took over 10 hours

Wow 26x26x26 combinations, Im spupprised it didnt take longer. Also there are asx codes that have numbers in them like 88E.ax. So really to get them all there would be 36x36x36 combinations

 

[WillyWonka]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Works Like a treat Only thing I had to do was change the date in the "yhquotes.pl" script as it was only downloading data up to the end of 2014. Apart from that.... freaking awesome!

$ed = 31; $em = 12; $ey = 2014 to $ed = 31; $em = 12; $ey = 2015

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Works Like a treat Only thing I had to do was change the date in the "yhquotes.pl" script as it was only downloading data up to the end of 2014. Apart from that.... freaking awesome!

$ed = 31; $em = 12; $ey = 2014 to $ed = 31; $em = 12; $ey = 2015

Good to see you're on the ball with the end date . I think from memory if you set it too far ahead yahoo cracks it. Great to see a chart. That's probably the best looking chart I've seen on ASF's

 

[DaveDaGr8]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

This gets codes in 10-20 Minutes ... Fast enough to do a scan while i'm having a coffee. It's c++ and built for my program, but it should be easy enough to modify. I can help if you tell me what you want. It will also get everything in the world.

I have another shell that creates all the codes, then i split that into 20 threads and run 20 versions of this simultaneously, just enough to saturate my bandwidth. CPU is ticking at 5%, so more internet would be good.

The biggest hurdle is that PageDownload() is blocking, so to download a page off the internet takes seconds. Any downloading off the internet will effectively pause that thread while it waits for data.

I use the same method for downloading data. This takes quite a bit longer.

 

[Atiq]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Hello and well done mate ,
but i am having problem while running MT4DW.PL , it gives me error , though dll is in the same folder all i can think of i am running 64bit system.

any solution for me mate ajcode ?

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Hello and well done mate ,
but i am having problem while running MT4DW.PL , it gives me error , though dll is in the same folder all i can think of i am running 64bit system.

any solution for me mate ajcode ?

Sounds like a 64 bit DLL is needed. I have attached zip file "MT4DW64_20150509.zip" which contains the 64 bit DLL file "MT4DWDLL.dll". Just replace the 32 bit DLL with this one and hopefully it will work.

Let me know if it works and maybe post a screen shot of an MT4 chart to show you got it working

Cheers,

Andrew.

 

[Ash8man]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

This is fantastic. What would need to be changed in the yhquotes.pl script to download index data?
For example I want to get ^AXJO, ^GSPC and ^DJI and so on.
Is this possible?

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

This is fantastic. What would need to be changed in the yhquotes.pl script to download index data?
For example I want to get ^AXJO, ^GSPC and ^DJI and so on.
Is this possible?

Thanks Ash8man,

If you scroll down to about my 10th post about symlistEX.csv & yhquotesEX.pl, where there is greater flexibility, you should hopefully be able to modify the file "symlistEX.csv" to your liking. Let me know if this works. I don't know about the "^". If it doesn't work then after downloading the quotes, you simply need to change how the symbol name is stored in the quotes.csv file. This can be easily modified in the perl script, but only if it is possible to download index data from yahoo finance in the first place.

Cheers,

Andrew.

 

[Ash8man]

Re: Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Pe

Thanks Ash8man,

If you scroll down to about my 10th post about symlistEX.csv & yhquotesEX.pl, where there is greater flexibility, you should hopefully be able to modify the file "symlistEX.csv" to your liking. Let me know if this works. I don't know about the "^". If it doesn't work then after downloading the quotes, you simply need to change how the symbol name is stored in the quotes.csv file. This can be easily modified in the perl script, but only if it is possible to download index data from yahoo finance in the first place.

Cheers,

Andrew.

Thanks for the quick reply Andrew. It worked.

To Download INDEX Stock Quotes from Yahoo Finance and Chart them in MT4 you need to change the following line in yhquotes.pl from (as Andrew mentions in post #9):

$url .= ".$ex&a=$sm&b=$sd&c=$sy&d=$em&e=$ed&f=$ey&g=d&igno re=.csv";

to

$url .= "&a=$sm&b=$sd&c=$sy&d=$em&e=$ed&f=$ey&g=d&ignore=. csv";

You then need to just add the symbols into the symlistEX.csv file.
For index quotes you put ^ in front of the code. EG ^AORD for ASX ALL ORDINARIES
For ASX quotes you put .ax after the code. EG RIO.AX for Rio Tinto
and so on for other countries/markets

Anyway thanks again for your work and time!

 

[davekalpesh]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

I have attached an updated version of the file symlist.csv, which contains pretty much the full list of ASX stocks that yahoo finance has to offer, there are about 2100 symbols in this file. The method of obtaining the list was brute force

Note: Please remove .txt extension.

It would be good to know if anyone could tell me if they are able to follow my instructions and get this to work. Maybe someone could post a few screenshots of ASX stocks on their Metatrader platform

Cheers,

Andrew.

Hi Andrew

Thank you for you sharing your work.

I have downloaded Active Perl. Also placed the files in to my script folder. However MT4 is not reading the files. Also you have not mention what to do with . MT4dw.pl and MT4DWRemove.Bat files?. I meant where to place them in MT4?

Also the Symlist.csv and yhquotes.pl.csv which are not changing from text files to other format. I have deleted .txt from the file but file is still a text file with .pl


your help will be appreciated

Thanks
Dave

 

[davekalpesh]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

I have attached an updated version of the file symlist.csv, which contains pretty much the full list of ASX stocks that yahoo finance has to offer, there are about 2100 symbols in this file. The method of obtaining the list was brute force

Note: Please remove .txt extension.

It would be good to know if anyone could tell me if they are able to follow my instructions and get this to work. Maybe someone could post a few screenshots of ASX stocks on their Metatrader platform

Cheers,

Andrew.

Hi Andrew

It was a great work. I was looking for stocks charts to access on MT4. Couden't find anywhere in the world. The answer was in my backyard and I was looking elsewhere. Anyway I have managed to use above method to access charts on MT4 now. Thanks a lot for sharing your work.

Happy Trading

 

[ajcode]

Re: Downloading ASX Stock Quotes from Yahoo Finance & Charting them in MT4 using Perl

Hi Andrew

Thank you for you sharing your work.

I have downloaded Active Perl. Also placed the files in to my script folder. However MT4 is not reading the files. Also you have not mention what to do with . MT4dw.pl and MT4DWRemove.Bat files?. I meant where to place them in MT4?

Also the Symlist.csv and yhquotes.pl.csv which are not changing from text files to other format. I have deleted .txt from the file but file is still a text file with .pl

your help will be appreciated

Thanks
Dave

Hi Dave,

In the diagram, in post 2, I show that the downloaded files (in red outlined circle) need to be put inside a subfolder in the history directory (not scripts folder). This subfolder is where the usual *.hst files are stored (For example EURUSD1440.hst). And you need to get the subfolder right (by using MT4 to work out where *.hst files are stored, i.e. open up a currency chart that you haven't used before and working out where its corresponding *.hst file is stored). Then you run the perl script yhquotes.pl (in command line type "perl yhquotes.pl") to create file "quotes.csv". After that you run perl script mt4dw.pl (in command line type "perl mt4dw.pl"). This will create the _*.hst files from created "quotes.csv" file. Then you can use MT4 to open up these offline charts.

At no stage do you use MT4 to run the scripts. You use the perl command line interpreter. None of these scripts have anything to do with MT4, except for the fact that the resultant *.hst files are used by MT4 to display the charts. And yes the *.pl scripts are text files, but are used by the perl command line interpreter to perform the required actions. The MT4DWRemove.bat file is just an executable batch script run in the command line to remove the _*.hst files created by my perl script. It's a clean way to ensure you have a clean set of _*.hst files for every day you run the perl scripts.

Hope this helps,

Edit Note: After typing this post, just realised that you worked it out . Could you please post a chart Dave?

Cheers,

Andrew.

 

[dazrul]

Hi Andrew,

1) I am here again in this thread just to say thank you for your efforts which make our routine on data update easier & faster, as well as to wish you a happy new year.

May the new year that follows be the best you have ever had.
Have a blissful new year!
Happy 2017 to you and your family!

2) do you have any smarter way extracting the historical prices from the ft.com besides copy and paste to the quotes.csv.
http://markets.ft.com/data/equities/tearsheet/historical?s=FRONTKN:KLS

Thank you.

Dazrul

 

[ajcode]

Hi Andrew,

1) I am here again in this thread just to say thank you for your efforts which make our routine on data update easier & faster, as well as to wish you a happy new year.

May the new year that follows be the best you have ever had.
Have a blissful new year!
Happy 2017 to you and your family!

2) do you have any smarter way extracting the historical prices from the ft.com besides copy and paste to the quotes.csv.
http://markets.ft.com/data/equities/tearsheet/historical?s=FRONTKN:KLS

Thank you.

Dazrul View attachment 69334

Hi Dazrul,

Thanks. And a happy 2017 to you and your family also.

I do have a smarter way to extract historical prices from mft.com and have attached file mftquotesEX_20170118.zip which contains the following files:

mftquotesEX.pl and
mftsymlistEX.csv

So just like at the start of this thread, the Perl script reads symbols in from mftsymlistEX.csv, visits the mft url and stores the quotes in file "quotes.csv".

As the Perl script is using simple html functionality, it only downloads the first page of data for each symbol. That is, we don't simulate clicking on "show more" button.

Modify file "mftsymlistEX.csv" to get the symbols you need.

Is that what you're after? Hope this helps.

Cheers,

Andrew.

 

[ajcode]

Hi Dazrul,

I have now attached file "mftquotesEX_20170121.zip", which contains the following files:

mftsymlistEX.csv - Sample symbol file (same as last post). Change to suit your needs.
mftquotesEX.pl - Does the same as described in previous post, but done a bit of a code tidy.
mftquotesEX2.pl - Enhanced script, allows you to download more than 1st page of historical data.

The last file "mftquotesEX2.pl" has a variable called "$extrapages". This allows you to download as much historical data as you like. I have set its value to 3. Change value to suit your needs.

Please let me know if this works.

Cheers,

Andrew.

 

[Tradep]

i must say i really admire your coding @ajcode

i was wondering if it is posible to do the same thing but to get history from https://www.investing.com stocks ?

 

[ajcode]

i must say i really admire your coding @ajcode

i was wondering if it is posible to do the same thing but to get history from https://www.investing.com stocks ?

Thanks Tradep.

Interesting problem. I noticed now that the website above your post have followed suit with ASF and now use https instead of http. Hence rendering my scripts useless for that site. I wonder if that was coincidence or they don't want people downloading free data so easily? How quickly did they change that? That's unreal. Could this mean that the data is extremely good??? I also noticed that they removed the show more button.

investing.com also uses https. Is their data up to scratch? They have a disclaimer that says "Fusion Media would like to remind you that the data contained in this website is not necessarily real-time nor accurate. All CFDs (stocks, indexes, futures) and Forex prices are not provided by exchanges but rather by market makers, and so prices may not be accurate and may differ from the actual market price, meaning prices are indicative and not appropriate for trading purposes."

https is a complex issue in the world of scripting in python and perl. Probably not worth the effort for lone scripts. However a messy alternative could be to use a brower plugin called imacros, but not in the usual sense because imacros is very good at doing repetitive tasks, but not if you have to do a lot of editing. So what I'm thinking is writing a perl/python script to generate the imacros script to download the web page data and then have some more perl/python scripts rip the data out and store it in a nicely formated quotes.csv file.

So to answer your question, Yes it might be possible. Confused? LOL!

Cheers,

Andrew

 

[ajcode]

Thought I'd add the equivalent python scripts for downloading from yahoo quotes.
The attached file "yhquotespy_20170426.zip" contains the following files:

mt4dw.py
yhquotes.py
yhquotesEX.py

Will need to edit the files and change the start/end dates to suit your needs.

Cheers,

Andrew.

 

[dazrul]

Hi Andrew,

Sorry for late reply, as today I got chance to visit this thread again. Thank you so much for "ex" script - it works well. However for "ex2" script, I think it is no longer valid, as mft has already changed the search format - now mft introduces Date Range with condition date range must equal or less than 1 year. Perhaps the new search format would not take so much of your busy time, in amending the script.

Regards,
Dazrul

 

[ajcode]

Seems like the yahoo finance website api "ichart.finance.yahoo.com" has been dropped forever . Anyone have any concrete news/facts on this? So now all my yahoo scripts no longer work, hence can't download historical data from yahoo finance.

I had a look on their website "au.finance.yahoo.com" and it appears you have to use an url similar to the following https link for downloading historical data (after rolling mouse cursor over Download data link):

"query1.finance.yahoo.com/v7/finance/download/BHP.AX?period1=1493380645&period2=1495972645&interval=1d&events=history&crumb=Wd4GhrEFt67"

The crumb has obviously got something to do with the cookie because if you get it wrong the page displays unauthorized and invalid cookie. What an absolute pain. So to solve the problem I need to get the crumb from first visiting "au.finance.yahoo.com" and then somehow pass that into my request in the script. Or I could hopefully write a perl/python script to generate an imacros script which can then be used to automate the process in a browser (slow and painful).

Another interesting thought has come to mind. What if I visit the url in a browser and get the crumb value, then just update my re-factored scripts with the new crumb value and run them as normal. That would be a nice hack. Probably wouldn't work, but worth a try.

Interesting yet very annoying and challenging problem.

Cheers,

Andrew.