I query a database and Post() individually each result that meets the criteria of my select statement, and write the results on screen. If my query returns 5 results, I have to individually send all 5 requests. What is the best way for me to send all 5 at the same time (or sequentially) and associate the appropriate response to the appropriate Post()?
W/O code - my current process is
1) Query SQL Server
2) Store Returned result in a dictionary
3) Post() dictionary
4) Get() xml result
5) Repeat
By leveraging the .NET Framework's parallel libraries, you can speed up your process by having multiple similar tasks execute in parallel.There aren't any special actions that need to be performed, but there are some considerations:
You may get blocked if you make too many requests too frequently to the destination (introduce some rate-limiting).
Whichever machine is running this code will need to be able to support multiple threads.
You may want a re-try mechanism.
You are limited to a specific number of concurrent network requests in .NET see( http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit(v=vs.110).aspx) and (System.Net.ServicePointManager.DefaultConnectionLimit == 24 --> BUG?)
Here is a brief implementation:
Public Class Program
Shared Sub Main()
Dim urls As New ConcurrentQueue(Of String)
urls.Enqueue("www.google.com")
urls.Enqueue("www.yahoo.com")
Dim myMethod As Action = Sub()
Dim localRequest As String
Dim localResponse As String
While urls.TryDequeue(localRequest)
System.Threading.Thread.Sleep(100) 'Rate limiting, adjust to suit your needs
localResponse = WebWorker.MakeRequest(localRequest)
Console.WriteLine(localResponse.ToString())
'Do something with localResponse
End While
End Sub
Parallel.Invoke(New ParallelOptions() With {.MaxDegreeOfParallelism = 10 }, myMethod, myMethod, myMethod)
End Sub
'Do something with the responses
End Class
Public NotInheritable Class WebWorker
Private Sub New()
End Sub
Public Shared Function MakeRequest(request As String) As String
Dim response As New String()
Dim status As Boolean
Dim req As HttpWebRequest = Nothing
Dim resp As HttpWebResponse = Nothing
Dim maxIterations As Integer = 2
Dim currentAttempt As Integer = 0
While currentAttempt < maxIterations AndAlso status = False
Try
req = DirectCast(HttpWebRequest.Create(New Uri(request)), HttpWebRequest)
Using resp = DirectCast(req.GetResponse(), HttpWebResponse)
If resp.StatusCode = HttpStatusCode.OK Then
status = True
Else
currentAttempt += 1
End If
' end using
End Using
Catch ex As Exception
currentAttempt += 1
End Try
' end try/catch
End While
' end while
Return response
End Function
End Class
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.