Downloading ASX Stock Quotes from Yahoo Finance and Charting them in MT4 using Perl

[ajcode]

Hi Andrew,

Sorry for late reply, as today I got chance to visit this thread again. Thank you so much for "ex" script - it works well. However for "ex2" script, I think it is no longer valid, as mft has already changed the search format - now mft introduces Date Range with condition date range must equal or less than 1 year. Perhaps the new search format would not take so much of your busy time, in amending the script.

Regards,
Dazrul

Hi Dazrul,

I have attached file mftquotesEX_20170613.zip, which contains the following files:

mftquotesEX2.py - Python 3 script only.
mftsymlistEX.csv - Tiny List of symbols. Edit to suit your needs.

Let me know if this works? You need to edit the python 3 script to change the start and end date.

If this works for you, it will be interesting to see how long for

Cheers,

Andrew.

 

[dazrul]

Hi Andrew,

I am very happy with your Python 3 script. I did change the start date and it works perfectly. Thank you and thank you. See attached screen capture for the result.

One more request, If I want to download from the warrants' historical data from
shareinvestor dot com /fundamental /factsheet.html? counter= 5256WA.MY [[since mft does not have any historical data for warrants]] which part of the script I need to amend?

Best regards,
Dazrul

 

[ajcode]

Hi Andrew,

I am very happy with your Python 3 script. I did change the start date and it works perfectly. Thank you and thank you. See attached screen capture for the result.

One more request, If I want to download from the warrants' historical data from
shareinvestor dot com /fundamental /factsheet.html? counter= 5256WA.MY [[since mft does not have any historical data for warrants]] which part of the script I need to amend?

Best regards,
Dazrul

I would say probably the whole lot of the script needs to be changed. That's the problem with web-scraping very specific data. Maybe you could change the url and then remove most of the code after first occurrence of "page = page.splitlines()", because page will contain different HTML data. Then you need to put your thinking cap on and work out how to extract the data from the page reading one line at a time. I use line.split(). After that you need to write what you extracted to a file.

Cheers,

Andrew.

 

[Wysiwyg]

Hello. I hit a brick wall whenever enquiring about a real time data stream from a broker trading platform to my charting platform. It is possible becasue they have an API so it is coding knowledge only. Are you interested?

 

[dazrul]

I would say probably the whole lot of the script needs to be changed. That's the problem with web-scraping very specific data. Maybe you could change the url and then remove most of the code after first occurrence of "page = page.splitlines()", because page will contain different HTML data. Then you need to put your thinking cap on and work out how to extract the data from the page reading one line at a time. I use line.split(). After that you need to write what you extracted to a file.

Cheers,

Andrew.

A bit tough for a beginner indeed. But making some efforts to learn python 3 would help a lot.

Any progress on your mt4 scanner?

Regards
Dazrul

 

[ajcode]

Attached is zip file yhquotespy20170806.zip which contains the following 2 files:

symlist.csv - sample input file containing a few symbols
yhquotes.py - updated python script for downloading yahoo quotes.

Apparently yahoo finance now has cookies and leaves crumbs to pick up as well.

Hopefully this works for now .

Cheers,

Andrew.

 

[ajcode]

When running the script "yhquotes.py" (in previous post) on a bigger list of symbols in symlist.csv, you might come across the following error:

File "yhquotes.py", line 135, in <module>
p_open = float(p_openstr); p_high = float(p_highstr)
ValueError: could not convert string to float: 'null'

This is because some lines in the downloaded page contain entries like this:

2016-09-07,null,null,null,null,null,null

So to fix the error in script "yhquotes.py", add the following 2 lines:

if line == "" or "null" in line:
continue

after the code that says:

for line in page:

and be precise in your indenting .

Cheers,

Andrew.

 

[909513758]

Hello Andrew and ajcode

Thank you so much for your sharing and hard work behind it. Finding the right data and import them into platforms like MT and NT has been real pain for me for years. Poor small Australia (though we are really big in area, really!)
I am totally a naive on programming, so I am thinking to buy low cost EOD data, like which is provided by www.cooltrader.com.au, then upload those data by myself into MT 4 or 5. It will be much easier task for me to do. But is it possible? If yes, could you please tell me what should I do? Assume the EOD file is in excel format (can be csv too).

Many thanks.

Jason

 

[ajcode]

Hello Andrew and ajcode

Thank you so much for your sharing and hard work behind it. Finding the right data and import them into platforms like MT and NT has been real pain for me for years. Poor small Australia (though we are really big in area, really!)
I am totally a naive on programming, so I am thinking to buy low cost EOD data, like which is provided by www.cooltrader.com.au, then upload those data by myself into MT 4 or 5. It will be much easier task for me to do. But is it possible? If yes, could you please tell me what should I do? Assume the EOD file is in excel format (can be csv too).

Many thanks.

Jason

No worries Jason, "quotes.csv" follows the format of symbol, date, open, high, low, close and volume.

If you can download the EOD data in that exact csv format and rename the file to quotes.csv then yes it is possible. If format similar, but in multiple files then a little bit of programming is needed to merge into one big csv file.

Also you need to be able to learn how to run scripts. So being able to download data from basic yahoo website and converting to mt4 using my scripts should be learned first. Probably good to search what other useful websites members use to download financial data in csv format. A good data provider will adjust according to share splits.

Somewhere along the line you'll probably need to learn how to program in python .

Cheers,

Andrew.

 

[ajcode]

Attached is zip file yhquotespy20180802.zip, which contains the following 2 files:

symlist.csv - input file containing symbols from https://www.asx.com.au/asx/research/ASXListedCompanies.csv
yhquotes.py - updated python script for downloading asx yahoo finance data.

Only changes here are that I added those 2 lines discussed in above post to yhquotes.py

It amazes me that the script still works! Although it does take awhile to download the data. Anyone have any faster methods to download 2000+ free asx historical stock prices?

 

[ajcode]

I've noticed that when I run the script in my previous post, sometimes there are duplicate consecutive rows in the data downloaded. For example, today I found the following entry:

SFL,2018-08-06,0.075000,0.075000,0.075000,0.075000,286
SFL,2018-08-06,0.075000,0.075000,0.075000,0.075000,286

I have now amended the script to discard rows that are the same as the previous row.
Hopefully there are no more silly entries like this.

Attached is zip file yhquotespy20180807.zip containing the updated files as seen in previous post.