Question Completing an action faster

Lend

New member
Joined
Oct 4, 2014
Messages
1
Programming Experience
1-3
I am attempting to scrap text from multiple URL's but in quick succession. I currently have it set up in a queue system but it isn't fairing so well as it only gains a small amount of values.

Start Button
VB.NET:
Try            For i As UInt64 = startID To endID
                intQueue.Enqueue(i)
            Next
            For i As Integer = startID To endID
                Dim t As New Thread(AddressOf Scrap)
                t.IsBackground = True
                t.Start()
            Next
        Catch ex As Exception
            MessageBox.Show(ex.ToString)
        End Try

Scrap Method
VB.NET:
Private Sub Scrap()        Try


            If lCount < endID Then
                Do While intQueue.Count > 0




                    Dim source As String = WebReuqest(URL & intQueue.Dequeue)
                    If source <> Nothing Then
                        Dim r As New Regex("About (.*)")
                        Dim m As Match = r.Match(source)
                        If m.Success Then
                            UpdateListbox(m.Value.Split(" ")(1))
                            testLbl.Text = endID & ":" & lCount
                            Call rCount()
                            RichTextBox1.Lines = (From o In ListBox1.Items
                              Let ostr = o.ToString
                              Select ostr).ToArray
                        
                        End If
                    End If
                Loop
            ElseIf lCount = endID Then
                MessageBox.Show("Finished!")


            End If
        Catch ex As Exception
            Dim sb As New System.Text.StringBuilder()
            Dim Str = sb.Append("Error:").Append(" ").Append(ex.ToString).ToString()
            MessageBox.Show(Str, _
        "Critical Warning", MessageBoxButtons.OKCancel, MessageBoxIcon.Warning, MessageBoxDefaultButton.Button1, MessageBoxOptions.RightAlign, True)


        End Try
Any ideas how I would make this faster?

Thanks so much!
 
Firstly, the term is "scrape", not "scrap".

As for the question, your code is poorly structure at the moment. If you're going to invoke that Scrap method (which you will be renaming to Scrape) once for each item then that method should know only about that one item. It should not be looking at how many items are in the queue or anything like that. It should not be touching any controls or displaying any completion messages either.

I would suggest a couple of options:

1. Put all your items into a list, start a BackgroundWorker and then run a Parallel.ForEach call over that list.
2. Use the Task class.
 
Back
Top