Read HTML Page Source to a String

[asn1981]

Hi
this might be a fairly basic question but how would i read a .html pages (that is on the users computer) source code into some kind of string variable?
thanks in advance

 

[sevenhalo]

is the html file on the local computer, or are you trying to scrape a site.

 

[asn1981]

is the html file on the local computer, or are you trying to scrape a site.

hi
no the html files are all on my computer but i need to compile certain statistics about the code in question

 

[sevenhalo]

If you just want to open it and read it into a string, this'll do it for you (assuming that the html file is on the computer running the app):

Dim strPath As String = "Full Path To File"
Dim strFile As System.IO.File
Dim strmRead As New System.IO.StreamReader(strFile.OpenRead(strPath))
Dim htmldata As String
htmldata = strmRead.ReadToEnd

But it's a sloppy way of doing it. Not to mention, since you're not flushing the buffer; you'll deffinitely see a performance hit on larger files.

 

[asn1981]

If you just want to open it and read it into a string, this'll do it for you (assuming that the html file is on the computer running the app):

Dim strPath As String = "Full Path To File"
Dim strFile As System.IO.File
Dim strmRead As New System.IO.StreamReader(strFile.OpenRead(strPath))
Dim htmldata As String
htmldata = strmRead.ReadToEnd

But it's a sloppy way of doing it. Not to mention, since you're not flushing the buffer; you'll deffinitely see a performance hit on larger files.

hi
that is the problem that i have
some of the files are very large
n the way the program will be working is like a spider therefore efficiency is key
would you have any further suggestions?

 

[sevenhalo]

Dim strPath As String = "Path to file even, so much for being anonymous :)"
Dim strFile As System.IO.File
Dim strmRead As New System.IO.StreamReader(strFile.OpenRead(strPath))
Dim arr() As String
Dim cnt As Long
While strmRead.Peek > 0
ReDim Preserve arr(cnt)
arr(cnt) = strmRead.ReadLine
cnt += 1
If cnt Mod 50 = 0 Then
strmRead.BaseStream.Flush()
End If
End While

 

[asn1981]

sevenhalo thanks alot for the quick responses i will try this now
cheers!

 

[sevenhalo]

How many lines are we looking at? I should probably ask that first before I try anything

 

[sevenhalo]

My code slowed it down!

On second thought, ignore flushing, use this:

Dim strPath As String = "FilePath"
Dim strFile As System.IO.File
Dim strmRead As New System.IO.StreamReader(strFile.OpenRead(strPath))
Dim cnt As Long = 0
Dim arr(100) As String
While strmRead.Peek > 0
If (cnt Mod 100 = 0) And cnt > 0 Then
ReDim Preserve arr(cnt + 100)
End If
arr(cnt) = strmRead.ReadLine
cnt += 1
End While
ReDim preserve arr(cnt)

I tested this against an html file with 400000 lines, took <30 seconds.

 

[asn1981]

ok great that sounds good, should be sufficient for the htmls that i am using

 

[kulrom]

http://www.vbdotnetforums.com/showthread.php?t=3657

posts #17-#20

Regards