A friend suggested to me following the basis outlined below, we can code the vb prog to download millions of results to a listbox.
If someone could look over and suggest coding (as i'm a little lost..), that would be great
"For me the API was useless, So i used my own method.
I knew that Google displays 10 results by default and to retrieve the results of each page i can use something like this.
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=0
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=10
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=20
-----
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=90
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=100
You need to use to HTTP/NET.WEBREQUEST to navigate these links
Code:
Start a loop
For i = 0 to 100 and using step 10 you can accomplish it
Naviage to http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=& i
Read the html source
Using regex or any similar function to parse out all the HTML links/Title/Description
next loop
For faster results, You can use about 10 BackGroundWorkers and each one to be navigating to the first 10 pages of the google search, That way you can be more quick then running a loop within a single BackGroundWorker
Hopefully you get an idea, I don't have the visual studio right now to provide you an example but i think someone would figure this out for you if you couldn't understand. "
If someone could look over and suggest coding (as i'm a little lost..), that would be great
"For me the API was useless, So i used my own method.
I knew that Google displays 10 results by default and to retrieve the results of each page i can use something like this.
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=0
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=10
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=20
-----
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=90
http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=100
You need to use to HTTP/NET.WEBREQUEST to navigate these links
Code:
Start a loop
For i = 0 to 100 and using step 10 you can accomplish it
Naviage to http://www.google.com/search?&q=" & KeywordSearch.Text & "&start=& i
Read the html source
Using regex or any similar function to parse out all the HTML links/Title/Description
next loop
For faster results, You can use about 10 BackGroundWorkers and each one to be navigating to the first 10 pages of the google search, That way you can be more quick then running a loop within a single BackGroundWorker
Hopefully you get an idea, I don't have the visual studio right now to provide you an example but i think someone would figure this out for you if you couldn't understand. "
