Question Multi Threading httpwebrequest with Queue

rexcarnation

Member
Joined
Feb 12, 2014
Messages
8
Location
Indonesia
Programming Experience
Beginner
Hello,
I search all day and I cannot find the answer how to multi threading with queue. So like this.

I have 100 URL in datagridview on col 1, an then I want to scrape all url on the datagrid by 10 thread, so the process only 1 -10 on the datagridview list. And when one of all process (scrape) finish it will do next scrape on the 11 and next until all of 100 URLs processed. It like queue but running 10 queue at once.

Here is my code.

    Private Sub BtGrab_Click(sender As Object, e As EventArgs) Handles BtGrab.Click
        ToolProgress.Maximum = DataURL.Rows.Count
        ToolProgress.Value = 0

        Dim maxthread As Integer = 10
        Dim upperBound = maxthread
        Dim waitHandles(upperBound) As ManualResetEvent

        For i = 0 To upperBound
            waitHandles(i) = New ManualResetEvent(False)
            ThreadPool.QueueUserWorkItem(AddressOf ProcessItem, i)
        Next
    End Sub

    Private Sub ProcessItem(ItemIndex As Object)

        DataURL(4, ItemIndex).Value = "Process"

        Try
            DataURL(4, ItemIndex).Value = "Scrapping"
            Dim request As HttpWebRequest = HttpWebRequest.Create(DataURL(1, ItemIndex).Value)
            request.Timeout = 10000

            Dim response As HttpWebResponse
            response = CType(request.GetResponse, HttpWebResponse)
            Dim webstream As Stream = response.GetResponseStream
            Dim streamreader As New StreamReader(webstream, Encoding.UTF8)
            Dim html As String = streamreader.ReadToEnd

            response.Close()
            DataURL(4, ItemIndex).Value = "Great"

        Catch ex As Exception
            DataURL(4, ItemIndex).Value = "Fail"
        End Try
       
    End Sub


my code just run only 1 -10 not the other.
plz help me :(

i found some article here Create Multiple Background workers with WebClient-VBForums
but any thread which process the same url.
 

rexcarnation

Member
Joined
Feb 12, 2014
Messages
8
Location
Indonesia
Programming Experience
Beginner
Why do you want to limit it to 10 at a time?
because, I think it maybe consume high resource, and the internet connection must be fast. Using 100 httpwebrequest at once is like open 100 webpages at once (am I true) ?.
I try to using SemaphoreSlim. it works, but it's not queue. The item on the datagridview is processed randomly.

how the easy way to limit it ?
 

jmcilhinney

VB.NET Forum Moderator
Staff member
Joined
Aug 17, 2004
Messages
14,321
Location
Sydney, Australia
Programming Experience
10+
I think it maybe consume high resource
Perhaps you should test whether that's actually the case before you try to write code to avoid it.

If you really do want to go that way then you should use the asynchronous methods already built into the WebRequest class. You populate your own queue, dequeue the initial set of items using a loop and then, when each item completes, you dequeue the next, continuing like that until there are no more items left.
 

jmcilhinney

VB.NET Forum Moderator
Staff member
Joined
Aug 17, 2004
Messages
14,321
Location
Sydney, Australia
Programming Experience
10+

jmcilhinney

VB.NET Forum Moderator
Staff member
Joined
Aug 17, 2004
Messages
14,321
Location
Sydney, Australia
Programming Experience
10+
Ahh, Im sorry. I miss it.
Can you show me how to use it (the code example), or you have recommended links to learn it (on VB .NET language) :D

Thank you
You should start by reading the documentation for the HttpWebRequest class to see what those asynchronous methods are. As is the case throughout the documentation, you'll probably find code examples there. If you need more, you then know what keywords to use to search for more explanation and examples on the web for yourself. If you still aren't sure how to proceed, you can then post back here with a more specific question.
 
Top Bottom