Send Multiple POST() Requests

[chalupabatman]

I query a database and Post() individually each result that meets the criteria of my select statement, and write the results on screen. If my query returns 5 results, I have to individually send all 5 requests. What is the best way for me to send all 5 at the same time (or sequentially) and associate the appropriate response to the appropriate Post()?

W/O code - my current process is
1) Query SQL Server
2) Store Returned result in a dictionary
3) Post() dictionary
4) Get() xml result
5) Repeat

 

[nayanpatel]

By leveraging the .NET Framework's parallel libraries, you can speed up your process by having multiple similar tasks execute in parallel. There aren't any special actions that need to be performed, but there are some considerations:

• You may get blocked if you make too many requests too frequently to the destination (introduce some rate-limiting).
• Whichever machine is running this code will need to be able to support multiple threads.
• You may want a re-try mechanism.
• You are limited to a specific number of concurrent network requests in .NET see( http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit(v=vs.110).aspx) and (System.Net.ServicePointManager.DefaultConnectionLimit == 24 --> BUG?)

Here is a brief implementation:

Public Class Program

    Shared Sub Main()

        Dim urls As New ConcurrentQueue(Of String)
        urls.Enqueue("www.google.com")
        urls.Enqueue("www.yahoo.com")
        Dim myMethod As Action = Sub() 

            Dim localRequest As String
            Dim localResponse As String
            While urls.TryDequeue(localRequest)
                System.Threading.Thread.Sleep(100) 'Rate limiting, adjust to suit your needs
                localResponse = WebWorker.MakeRequest(localRequest)
                Console.WriteLine(localResponse.ToString())
                'Do something with localResponse
            End While

        End Sub
        Parallel.Invoke(New ParallelOptions() With {.MaxDegreeOfParallelism = 10 }, myMethod, myMethod, myMethod)
    End Sub

    'Do something with the responses
End Class

Public NotInheritable Class WebWorker
    Private Sub New()
    End Sub

    Public Shared Function MakeRequest(request As String) As String
        Dim response As New String()
        Dim status As Boolean
        Dim req As HttpWebRequest = Nothing
        Dim resp As HttpWebResponse = Nothing
        Dim maxIterations As Integer = 2
        Dim currentAttempt As Integer = 0

        While currentAttempt < maxIterations AndAlso status = False
            Try
                req = DirectCast(HttpWebRequest.Create(New Uri(request)), HttpWebRequest)

                Using resp = DirectCast(req.GetResponse(), HttpWebResponse)
                    If resp.StatusCode = HttpStatusCode.OK Then
                        status = True
                    Else
                        currentAttempt += 1

                    End If
                    ' end using
                End Using
            Catch ex As Exception
                currentAttempt += 1
            End Try
            ' end try/catch
        End While
        ' end while
        Return response
    End Function

End Class