VB Web Data Mining

[ChaoticMetroid]

Hey, I'm wondering how exactly I'd go about extracting info (weather, stocks, dictionary searches) from online. I've never done anything involving web extraction so if you guys have any good resource or tutorial sites, or could explain what exactly is required it would help alot. Thanks.

[rdrake]

Many sites offer access to their data via an API. This gives you a controlled way to access the data from their databases.

For an example, check out Flickr.

[jacklarry]

Hey Justin,
I know you are the one that posted this message. Guess who I am?

[PaulButler]

The easiest way would be to find a webpage that has the information you want, view it's source, find the code around the data you want. Then write a program that loads site and searches for the text before and after that data, and the data between this text is what you are looking for. This assumes you know at least some basic HTML, but if not it is easy to learn.

Here is an example of the method I (attempted to) explained:

Say the site is http://somesite.com/stockinfo/INTC/. Load it up in FireFox / Internet Explorer and view the source. Say for example the text on the page says that the stock is at $53.82, so you would search the html for this and find that it appears in this context:

HTML:

<div id="stockprice">$53.82</div>

So you could take the <div...> and </div> that surround the value and search for them in the HTML of the page after loading it in VB to find the value.

This is just basic page scraping, it works for anything, not just stock info. If you want to go more advanced, look up "regular expressions".

Keep in mind that there may be legal issues with page scraping, if you intend to distribute your software.

[ChaoticMetroid]

Thanks for all your help I've got it working using something along the lines of Paul Butler's suggestion in a weather program.