Data mining from websites

[Gringotts Bank]

Hi!
Does anyone know how to get data from a website (into excel format)? There's a very specific bit of data that I want to mine from stockness monster, that I can't get with my current data provider. Could be approx. 1000 bits of data so can't be done manually.

Thanks.

 

[Country Lad]

Never been to stockness monster, so I have no idea. What data are you after, I may have it in my system.

Cheers
CL

 

[Gringotts Bank]

Thanks for the offer CL but I will want to view it every day. Number of trades in first hour for each stock.

 

[captain black]

iMacros is a Firefox extension that's able to scrape data from websites. (There's Internet Explorer and Google Chrome versions as well, search for "iMacros")

https://addons.mozilla.org/en-US/firefox/addon/imacros-for-firefox/

(3) Data Extraction, Web Scraping/Mining & Enterprise Data Mash-Ups

The iMacros EXTRACT command automatically reads data from a website and exports it to CSV files – the exact opposite of filling out forms. iMacros includes full Unicode support and works with every language, including multi-byte languages such as Chinese. You can use this feature to download stock quotes, gather and compare web store prices, and more.

Some sites have legal issues with web scraping, best to check first with the particular site.

Excel can also scrape data from website tables via a web query, again, check with the particular website regarding it's policies on using their data.

 

[Gringotts Bank]

Thanks a lot captain.

 

[Gringotts Bank]

SO I have the firefox imacro, have read the small help sections, and I'm not sure how to get that data from that website. Captain, can you help or is it likely to be a difficult task?

I want number of trdaes in the first hour for each stock. Just had an idea i want to test.

"Demo extract table" is the sort of thing I need, I assume.

 

[captain black]

You would need some basic programming knowledge to use iMacros effectively.

Just having a quick look at Stock Ness, I was able to download the course of sales table for BHP into excel using a simple web query, is that the data you're after? Might be easier to use an excel sheet for each stock you require data for. It looks like Stockness data is 20 minute delayed too, using intraday data from a broker such as Interactive Brokers or use the excel plugin for web iress may be a simpler solution for live data?

 

[Gringotts Bank]

I don't really need live data, just one small bit of info at the end of each day, so delayed is quite ok. It's something I would test over the course of a week or month or so.

But yes, the info would come from the course of trades table.

Is excel query easier to do, given that i want this info for all stocks?

 

[captain black]

Is excel query easier to do, given that i want this info for all stocks?

There are macros you can use in excel but once again it comes down to programming knowledge.

Otherwise you could set up a course of sales web query in excel for each stock and set it to update daily etc. A lot of work if you're after data for 2000+ stocks initially, but once it's set-up then it will update automatically.

http://www.techrepublic.com/article/pull-data-into-microsoft-excel-with-web-queries/6115870

 

[Gringotts Bank]

Thanks again cap.

 

[Gringotts Bank]

Long shot...

Could someone write me an imacro to download the bottom 10 lines of the course of trades for each of say 1000 small cap stocks? I can provide the list of stock codes.

The website is stocknessmonster.com. Just type BHP in the code box, then 'trades' for course of trades page for it to show.

Thanks in advance.

 

[jaystar86]

Hi GB

If you haven't been able to get this done and are still after someone there is a wonderful site (I'm not affiliated with) that you will be able to find someone to create the script for you... I'd estimate it'd only cost you 20 bucks to do it...

www.odesk.com --> They are a freelance site. Just beware of who you choose and ensure you are satisified with your contractor etc... before signing them up/paying them.

Regards,

jayvan

 

[sinner]

Long shot...

Could someone write me an imacro to download the bottom 10 lines of the course of trades for each of say 1000 small cap stocks? I can provide the list of stock codes.

The website is stocknessmonster.com. Just type BHP in the code box, then 'trades' for course of trades page for it to show.

Thanks in advance.

Hey GB,

I don't know anything about iMacro, but I'm a compsci geek reckon I could write the functionality you are requesting (incl dump to spreadsheet) into a python script in no time at all.

I'll give it a shot tonight.

 

[pixel]

Hey GB,

I don't know anything about iMacro, but I'm a compsci geek reckon I could write the functionality you are requesting (incl dump to spreadsheet) into a python script in no time at all.

I'll give it a shot tonight.

Why not run it in a straight Excel macro: loop over all codes to query the URL and extract the last 10 lines in question. Provided it's a website that's unencrypted and open to all readers, there shouldn't be any legal issues either.
What's the URL - say, for BHP? Shouldn't take more than a few minutes to write the code. (I already have a framework for an ex-div spreadsheet.)

 

[Gringotts Bank]

Sorry I sort of forgot about this thread after I got it working. I posted some results under "MD stats" thread.

So yeh I'm fine. Thanks for the offers.